Opened 2 years ago

Closed 13 months ago

#14336 closed bug (fixed)

ghci leaks memory

Reported by: NeilMitchell Owned by:
Priority: normal Milestone: 8.8.1
Component: GHCi Version: 8.2.1
Keywords: Cc: ndmitchell@…, alanz, saurabhnanda, mnislaih, pggiarrusso, cocreature
Operating System: Windows Architecture: x86_64 (amd64)
Type of failure: Runtime performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

The following script spawns ghci, and that spawned ghci then goes on to leak memory:

import Control.Concurrent
import Control.Monad
import System.IO
import System.Process

main = do
    (Just hin, Nothing, Nothing, pid) <- createProcess (proc "ghci" ["+RTS","-S"]){std_in=CreatePipe}
    forever $ do
        threadDelay 100000 -- 0.1s
        hPutStrLn hin "\"this is a test of outputting stuff\""
        hFlush hin

This script just writes a string to GHCi, which then echos it back. The +RTS -S is useful to watch the live memory tick up in realtime, but it leaks without it, and the leak can be seen in process explorer (87Mb to 700Mb over about 30 minutes).

While repeatedly writing commands may not be a standard usage of ghci, it is when driven by tools such as ghcid (https://hackage.haskell.org/package/ghcid) and other IDE-like uses.

Change History (28)

comment:1 Changed 2 years ago by NeilMitchell

Cc: ndmitchell@… added

comment:2 Changed 2 years ago by Phyx-

Hmm I believe this is not Windows specific, peaking into the heap it seems to be all PAPs and closures so I believe it's a generic GHCi bug, where a reference is being held to the previous repl results and so preventing GC.

I'll find some time to take a look at it, thanks for the report!

comment:3 Changed 22 months ago by alanz

Cc: alanz added

comment:4 Changed 22 months ago by saurabhnanda

Cc: saurabhnanda added

comment:5 Changed 22 months ago by saurabhnanda

I can confirm that this occurs in GHCi, Intero, and HIE on Mac OSX. Someone suggested :set +r could solve this, but it doesn't seem to help. However, :set -fobject-code *seems* to either slow-down the memory leak or stop it altogether.

comment:6 Changed 22 months ago by mnislaih

Cc: mnislaih added

comment:7 Changed 21 months ago by pggiarrusso

Other reports of GHCi leaks include https://ghc.haskell.org/trac/ghc/ticket/12848 https://github.com/commercialhaskell/intero/issues/84. The first suggests that loading code leaks memory, which might relate to comment:5.

comment:8 Changed 21 months ago by pggiarrusso

Cc: pggiarrusso added

comment:9 Changed 21 months ago by saurabhnanda

I have proposed this as an idea for GSoC / HSoC at https://github.com/haskell-org/summer-of-haskell/pull/15/files

If anyone is interesting in mentoring students working on fixing this space leak, please participate in the discussion at https://github.com/haskell-org/summer-of-haskell/issues/29

comment:10 Changed 21 months ago by mpickering

Just to be clear, I should be looking at the third column increasing? It starts out for me at around 8,569,448 and gradually increases up to 13,360,528 after running 1000 iterations.

Other things I tried.

  1. Not creating it bindings as one was created for each line. This didn't seem to make a big difference for small examples but does for big examples. (Max residency 12mb (unmod) vs 13mb (mod))
  2. Not running any Haskell code, so changing the string line to `:set -XTypeApplications", the live column still increased but more slowly. (Up to 11219144 after 1000 iterations).
  3. Running with :set -fobject-code, seemed to make no difference or make things slightly worse (14051160) (Just once at the start of the loop).
  4. Increasing the size of the string (by 100x) makes things much worse 62235312 but not 100x worse, just 6x worse, maximum residency 55mb. BUT, running with my modified compiler which doesn't generate it bindings, only 13mb max residency.

Anyway I don't really know what I am looking for so if someone could point out which number I should be looking at and what it means for my hardware averse brain it would be useful.

Here is the patch I used for no_it. https://gist.github.com/78366267566cfe8cf85101fc11b169ed

Last edited 21 months ago by mpickering (previous) (diff)

comment:11 Changed 21 months ago by mpickering

30000 iterations with a short string is about the same max residency with/without no_it. (155mb vs 149mb)

comment:12 Changed 21 months ago by mpickering

30000 iterations of :set -XTypeApplications only leads to 16mb of maximum residency so perhaps there is no serious leak there. (6mb with 1000 iterations)

comment:13 Changed 21 months ago by NeilMitchell

The end thing that matters is process memory usage, and given enough time (half an hour or so) that clearly increases, but not linearly (thanks to effects like GC). The use of the RTS columns just gives you an easy way to spot it faster. The columns for the RTS stats are:

    Alloc    Copied     Live     GC     GC      TOT      TOT  Page Flts
    bytes     bytes     bytes   user   elap     user     elap

Of those, Live bytes is the one I was watching (3rd one along)

comment:14 Changed 21 months ago by mpickering

Doing 1000 iterations with a very big string (approx 20000 chars) results in 47mb max residency with no_it. Doing 1000 iterations with a vanilla compiler causes my computer to oom, 100 iterations requires 488mb

  • 100 - no_it 42mb
  • 100 - vanilla 488mb
  • 1000 - no_it 47mb
  • 1000 - vanilla oom

comment:15 Changed 21 months ago by cocreature

Cc: cocreature added

comment:16 Changed 21 months ago by mpickering

I put the patch up at https://phabricator.haskell.org/D4299 if anyone wants to try it.

comment:17 Changed 21 months ago by Ben Gamari <ben@…>

In 41afbb3/ghc:

Add flag -fno-it

This flag stops ghci creating the special variable `it`
after evaluating an expression. This stops ghci leaking
as much memory when evaluating expressions. See #14336

Reviewers: bgamari

Reviewed By: bgamari

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14336

Differential Revision: https://phabricator.haskell.org/D4299

comment:18 Changed 21 months ago by bgamari

Milestone: 8.6.1

Can someone verify whether there are any leaks remaining after the above patch?

comment:19 Changed 21 months ago by mpickering

I think there still are leaks but at a much slower rate. I also suspect module reloading still causes big leaks.

comment:20 Changed 20 months ago by NeilMitchell

Passing -fno-it causes () to be printed to stdout vastly more often. Did this also break that bit? Looking at the section on GHCi Plans I note that the no-it branch is missing the it::() check in step A.

comment:21 Changed 20 months ago by mpickering

I reasoned that it didn't matter at all if ghci was being invoked programatically. In fact, it is more consistent to print out () rather than omit it.

This was because it wasn't very convenient to insert this check in the no-it version as it worked before by checking if the type of it was ().

Is it breaking your program?

comment:22 Changed 20 months ago by NeilMitchell

Yes, it breaks my program - but I can work around it - although the workaround is a bit grim.

comment:23 Changed 18 months ago by NeilMitchell

Anything that wraps ghci into a program that users are still expected to interact with is going to start having weird special cases to undo these changes. For my specific case of ghcid I can remove the () when ghcid itself sends a command in (no real problem), but I also allow the user to interact with the underlying ghci (via the --test flag), and there the surprising change in behaviour is going to be unpleasant and hard to avoid without just filtering out () lines, some of which might be deliberate.

Any chance of getting the behaviour to omit () added back?

comment:24 Changed 17 months ago by simonpj

Any chance of getting the behaviour to omit () added back?

This doesn't seem to have much to do with the Description!

I have no idea what the old behaviour is, what the new behaviour is, or which patch made the change that you object to.

Would you like give a short ab-initio specification of the change you want? It does not sound controversial or hard to implement. Maybe you can offer a patch too?

comment:25 Changed 17 months ago by NeilMitchell

Old behaviour: ghci omitted printing out () for things of type ().

New behaviour: ghci, when passed -no-it (which is the solution to memory leaks and introduced in this patch) always prints out ().

Desired behaviour: Have -no-it not print out ().

Last edited 17 months ago by NeilMitchell (previous) (diff)

comment:26 Changed 17 months ago by simonpj

OK. Sounds plausible. Doesn't sound hard. Can anyone offer a patch?

comment:27 Changed 15 months ago by bgamari

Milestone: 8.6.18.8.1

These won't be addressed in GHC 8.6.

comment:28 Changed 13 months ago by NeilMitchell

Resolution: fixed
Status: newclosed

If this ever makes it into a released GHC (which it will with GHC 8.6) then I need to adjust ghcid to cope with it. Once I've done that, changing the behaviour for a future release has very little benefit to me. As a result, closing this ticket with Fixed as the initial ticket was fixed.

Note: See TracTickets for help on using tickets.