Opened 9 years ago

Closed 4 years ago

#4029 closed bug (fixed)

ghci leaks memory when loading a file

Reported by: blarsen Owned by: jme
Priority: high Milestone: 8.0.1
Component: GHCi Version:
Keywords: memory leak Cc: judah.jacobson@…, blarsen, jme
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Compile-time performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D1950
Wiki Page:

Description

In the 6.12 series, ghci leaks memory when loading definitions. This appears to be a regression from previous releases. The problem appears in both 6.12.1 and 6.12.2, but not in 6.8.3 or 6.10.4.

As a test, take the attached Hello World file, fire up ghci, and :load Hello.hs. Watch the memory usage of ghci---each time you repeat the load command, the memory usage of the process grows. (It grows slowly on this simple example, but on a real project, the growth can be several megabytes per :load.)

I discovered this due to my workflow. When programming Haskell, I typically use emacs with haskell-mode, and frequently reload the definitions of the code I am working on. After upgrading to 6.12.1, after coding for some time, I noticed my 8GB machine was swapping. (It is faster to :reload the file rather than :load, and reloading doesn't appear to suffer from the memory leak, but sometimes I switch between modules and :reload is not applicable.)

My workaround for now is to regularly restart my ghci processes to prevent their memory usage from growing so much that my system starts swapping.

I have observed this bug on 32- and 64-bit GHCi on various versions of Ubuntu, on both Intel and AMD processors.

Attachments (6)

Hello.hs (72 bytes) - added by blarsen 9 years ago.
Hello world
4029.patch (2.0 KB) - added by simonmar 9 years ago.
patch for Haskeline
ghc-stage2.hp (150.7 KB) - added by simonmar 9 years ago.
Leak.hs (879 bytes) - added by jme 4 years ago.
leak.script (220 bytes) - added by jme 4 years ago.
workaround.script (360 bytes) - added by jme 4 years ago.

Download all attachments as: .zip

Change History (27)

Changed 9 years ago by blarsen

Attachment: Hello.hs added

Hello world

comment:1 Changed 9 years ago by igloo

Milestone: 6.12.3
Priority: normalhigh

Thanks for the report.

comment:2 Changed 9 years ago by simonmar

Verified, and it's still happening in HEAD. This is a bad regression, we should definitely fix it in 6.12.3.

Changed 9 years ago by simonmar

Attachment: 4029.patch added

patch for Haskeline

comment:3 Changed 9 years ago by simonmar

Cc: judah.jacobson@… added

Making Haskeline's InputT monad more strict fixes this, though I only have a vague idea of what's going on (the heap profiler doesn't work with GHCi). Poking around in the heap I found that data from previous compilations appears to be hung onto by chains of thunks originating in the InputT monad.

Judah: could this lead to any problems? (see attached patch for Haskeline)

comment:4 Changed 9 years ago by judahj

That looks like it would be fine; InputT is always run on top of a MonadIO instance, so using a lazy StateT monad wouldn't make much sense anyway.

The latest release of haskeline (0.6.2.2) uses a continuation-based implementation of StateT; can you verify whether that also fixes the problem?

Note that there were several other nontrivial internal changes between haskeline-0.6.2.1 (which was released with 6.12.[1,2]) and 0.6.2.2; so I'm not sure how suitable the latter is for the ghc-6.12.3 release.

comment:5 Changed 9 years ago by simonmar

Owner: set to simonmar

comment:6 Changed 9 years ago by simonmar

Status: newmerge

Thanks Judah - 0.6.2.2 fixes the leak, so I've imported that. It depends on utf8-string 0.3.6 (we had 0.3.4), so I've imported that too. The version bump won't be a problem for 6.12.3 because we don't ship utf8-string.

comment:7 Changed 9 years ago by igloo

Resolution: fixed
Status: mergeclosed

Fixed in 6.12 by:

Sat May  8 15:41:50 PDT 2010  Ian Lynagh <igloo@earth.li>
  * Update haskeline and utf8-string; fixes #4029 in 6.12 branch

comment:8 Changed 9 years ago by blarsen

Owner: simonmar deleted
Resolution: fixed
Status: closednew

Is this fix supposed to be included in 6.12.3? I still observe the ghci memory leaks with 6.12.3.

comment:9 Changed 9 years ago by igloo

Milestone: 6.12.36.14.1

comment:10 Changed 9 years ago by simonmar

Resolution: fixed
Status: newclosed

This is fixed in HEAD, I've no idea why the fix didn't work in the 6.12 branch, there must be something else going on.

Heap profile from yesterday's GHC attached created by repeating :load hello.hs 500 times.

Changed 9 years ago by simonmar

Attachment: ghc-stage2.hp added

comment:12 Changed 9 years ago by blarsen

Cc: blarsen added
Resolution: fixed
Status: closednew

I tested just now with ghc-6.13.20100629-x86_64-unknown-linux.tar.bz2.

Watching the ghci process memory usage using top, I see that every time I :load Hello.hs, the resident memory usage increases by 12-20kB, and virtual memory usage increases by >100kB.

I would take a heap profile from repeatedly loading as Simon did but I am not sure how.

Let me know what more I can do to diagnose what's going on.

comment:13 Changed 9 years ago by simonmar

Owner: set to simonmar

You're right - there was another leak. I'm just testing the fix now.

comment:14 Changed 9 years ago by simonmar

Milestone: 6.14.1_|_
Priority: highnormal

I fixed a leak:

Fri Jul  2 06:02:10 PDT 2010  Simon Marlow <marlowsd@gmail.com>
  * Fix a few places where we forgot to close the text codecs (#4029)
  Each time you invoke :load in GHCi it resets the CAFs, including
  stdin/stdout/stderr, and each of these was allocating a new iconv_t.

The process still grows when repeatedly loading a file, but much more slowly than before. The heap profile shows that there is a small leak somewhere (THUNK_2_0 is growing, everything else is constant). You can get a heap profile just by adding +RTS -h to ghci.

So I guess we should leave the ticket open as there's still a small leak, but I don't think it's a show stopper, so I'll remove the ticket from the 6.14.1 milestone.

comment:15 Changed 9 years ago by simonmar

Owner: simonmar deleted

Changed 4 years ago by jme

Attachment: Leak.hs added

Changed 4 years ago by jme

Attachment: leak.script added

Changed 4 years ago by jme

Attachment: workaround.script added

comment:16 Changed 4 years ago by jme

There is a new leak on the 8.0 branch. For example, running attachment:leak.script, which simply loads attachment:Leak.hs 20 times, results in a max residency of 115 MB:

$ inplace/bin/ghc-stage2 --version
The Glorious Glasgow Haskell Compilation System, version 8.0.0.20160206
$ inplace/bin/ghc-stage2 --interactive +RTS -t < leak.script > /dev/null
<<ghc: 2104620200 bytes, 311 GCs, 26012178/114880456 avg/max bytes residency (11 samples), 236M in use, 0.002 INIT (0.002 elapsed), 1.869 MUT (1.882 elapsed), 1.425 GC (
1.424 elapsed) :ghc>>

This is an obvious regression from 7.10.3, for which the max residency is 18.3 MB:

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3
$ ghc --interactive +RTS -t < leak.script > /dev/null
<<ghc: 1757780832 bytes, 461 GCs, 11400611/18318760 avg/max bytes residency (16 samples), 48M in use, 0.002 INIT (0.002 elapsed), 1.362 MUT (1.367 elapsed), 1.150 GC (1.
149 elapsed) :ghc>>

There is a quick fix for the bulk of the leak (just make the hsc_IC field in HscTypes.HscEnv strict), but I'd like to track down what caused the regression before I submit a patch. In the meantime, anyone who needs a workaround (without recompiling) should periodically run a command which references the contents of the loaded module. For example, by querying the type of a constructor after each load, attachment:workaround.script is able to reduce the max residency to 29.1 MB:

$ inplace/bin/ghc-stage2 --interactive +RTS -t < workaround.script > /dev/null
<<ghc: 2106120848 bytes, 422 GCs, 16580354/29127088 avg/max bytes residency (14 samples), 80M in use, 0.002 INIT (0.002 elapsed), 1.862 MUT (1.875 elapsed), 1.363 GC (1.
363 elapsed) :ghc>>

comment:17 Changed 4 years ago by thomie

Milestone: 8.0.1
Priority: normalhigh

comment:18 Changed 4 years ago by jme

Cc: jme added

comment:19 Changed 4 years ago by jme

Owner: set to jme

comment:20 Changed 4 years ago by jme

Differential Rev(s): Phab:D1950
Status: newpatch

comment:21 Changed 4 years ago by Ben Gamari <ben@…>

In 6ca9b15/ghc:

GHCi: Fix load/reload space leaks (#4029)

This patch addresses GHCi load/reload space leaks which could be
fixed without adversely affecting performance.

Test Plan: make test "TEST=T4029"

Reviewers: austin, bgamari

Reviewed By: bgamari

Subscribers: mpickering, thomie

Differential Revision: https://phabricator.haskell.org/D1950

GHC Trac Issues: #4029

comment:22 Changed 4 years ago by bgamari

Resolution: fixed
Status: patchclosed
Note: See TracTickets for help on using tickets.