Opened 5 years ago

Last modified 4 years ago

#10228 new bug

Increased memory usage with GHC 7.10.1

Reported by: jstolarek Owned by:
Priority: normal Milestone:
Component: Compiler Version: 7.10.1
Keywords: Cc:
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Compile-time performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

(Repost from ghc-devs)

I just uprgaded both of my machines to use GHC 7.10.1. I keep sandboxed installations of GHC and this means I had to rebuild Agda and Idris because the binaries built with GHC 7.8.4 were stored inside deactivated 7.8.4 sandbox. Sadly, I had problems building both Agda and Idris due to GHC taking up all of available memory.

With Idris the problematic module was Idris.ElabTerm (~2900LOC). The interesting part of the story is that when I do a clean build of Idris GHC consumes all of memory when compiling that module and I have to kill the build. But when I restart the build after killing GHC the module is compiled using a reasonable amount of memory and within reasonable time.

With Agda the problematic module is Agda.TypeChecking.Serialise (~2000LOC). The trick with killing the build and restarting it didn't work in this case. I had to compile Agda with GHC 7.8.4 (which works without problems though the mentioned module still requires a lot of memory) and alter my setup so that Agda binary is not stored inside GHC sandbox.

I wonder if any of you came across similar issues with GHC 7.10.1? Do we have any performance data that allows to compare memory usage and performance of GHC 7.10.1 with previous stable releases?

All of the above happened on 64bit Debian Wheezy with 2GB of RAM.

On my second machine, this time with 4GB of RAM, compiling Agda ran out of memory (again Agda.TypeChecking.Serialise module) and I had to kill the build. But once I restarted the build the module was compiled succesfully in a matter of minutes and using around 50% of memory.

This looks like some kind of memory leak in GHC.

Change History (6)

comment:1 Changed 5 years ago by simonpj

It would be extremely helpful if you, or someone else, could boil out a test case that demonstrates the problem. And, better still, use a profiled compiler to isolate where the problem is.

Simon

comment:2 Changed 5 years ago by simonpj

See also [this email thread](https://mail.haskell.org/pipermail/ghc-devs/2015-April/008751.html), which has further responses.

Simon

comment:3 Changed 5 years ago by rwbarton

The type of behavior you describe is not totally unexpected since when GHC compiles multiple modules in a single run, it retains the information read from all the interface files it has encountered in case it needs that information later. So it's a time/space trade-off, not really a memory leak. Of course, it's possible that that interface file information is consuming much more space than it needs to. (Or the cause could be something else entirely.)

comment:4 Changed 4 years ago by rwbarton

I tried to investigate this, but Agda has a bunch of upper bounds that rule out building with 7.10, and with --allow-newer, I got an error about No instance for (GHC.Generics.Generic (Ptr a)) and gave up. How did you manage to build Agda with 7.10 in the first place?

comment:5 Changed 4 years ago by rwbarton

Oh, the unreleased version on github seems to be building with 7.10.

Maybe GHC failing to build Agda on machines with low memory is a feature, since Agda itself needs so much memory to typecheck anything anyways :)

comment:6 Changed 4 years ago by rwbarton

The memory usage is up only 10% over 7.8, so this doesn't seem to be a significant regression.

7.8

 327,943,402,392 bytes allocated in the heap
  18,580,963,960 bytes copied during GC
     870,373,552 bytes maximum residency (24 sample(s))
      13,857,232 bytes maximum slop
            2446 MB total memory in use (0 MB lost due to fragmentation)

7.10

 506,171,221,880 bytes allocated in the heap
  34,277,256,496 bytes copied during GC
     945,447,464 bytes maximum residency (35 sample(s))
      18,326,864 bytes maximum slop
            2681 MB total memory in use (0 MB lost due to fragmentation)
Note: See TracTickets for help on using tickets.