Opened 10 years ago

Closed 10 years ago

#3664 closed merge (fixed)

Ghc eats tremendous heaps of RAM in -prof build (highlighting-kate)

Reported by: slyfox Owned by: igloo
Priority: normal Milestone: 6.12 branch
Component: Compiler Version: 6.12.1 RC1
Keywords: Cc:
Operating System: Linux Architecture: x86_64 (amd64)
Type of failure: Compile-time performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

Tried to build highlighting-kate-0.2.5 from hackage with ghc-6.12rc1 and could not wait build result. Ghc ate 2.5BG of ram in 6 minutes, then I stopped it as machine swapped horribly.

$ ghc --info
 [("Project name","The Glorious Glasgow Haskell Compilation System")
 ,("Project version","6.12.0.20091010")
 ,("Booter version","6.10.4")
 ,("Stage","2")
 ,("Have interpreter","YES")
 ,("Object splitting","YES")
 ,("Have native code generator","YES")
 ,("Support SMP","YES")
 ,("Unregisterised","NO")
 ,("Tables next to code","YES")
 ,("Win32 DLLs","")
 ,("RTS ways","l debug  thr thr_debug thr_l thr_p  dyn debug_dyn thr_dyn thr_debug_dyn")
 ,("Leading underscore","NO")
 ,("Debug on","False")
 ,("LibDir","/usr/lib64/ghc-6.12.0.20091010")
 * Using cabal-1.8.0.
[1 of 1] Compiling Main             ( /tmp/paludis/dev-haskell-highlighting-kate-0.2.5/work/highlighting-kate-0.2.5/Setup.lhs, /tmp/paludis/dev-haskell-highlighting-kate-0.2.5/work/highlighting-kate-0.2.5/Setup.o )
Linking setup ...
Configuring highlighting-kate-0.2.5...
Flags chosen: executable=True, splitbase=True
Dependency base >=3 && <5: using base-4.2.0.0
Dependency containers -any: using containers-0.3.0.0
Dependency filepath -any: using filepath-1.1.0.3
Dependency parsec <3: using parsec-2.1.0.1
Dependency pcre-light -any: using pcre-light-0.3.1
Dependency xhtml -any: using xhtml-3000.2.0.1
Using Cabal-1.8.0 compiled by ghc-6.12
Using compiler: ghc-6.12.0.20091010

...

/usr/bin/gcc /tmp/paludis/dev-haskell-highlighting-kate-0.2.5/temp/14064.c -o /tmp/paludis/dev-haskell-highlighting-kate-0.2.5/temp/14064 -D__GLASGOW_HASKELL__=612 -I. -O0 -O0 -I/usr/lib64/ghc-6.12.0.20091010/bytestring-0.9.1.5/include -I/usr/lib64/ghc-6.12.0.20091010/base-4.2.0.0/include -I/usr/lib64/ghc-6.12.0.20091010/include -I/usr/lib64/ghc-6.12.0.20091010/include -L/usr/lib64/xhtml-3000.2.0.1/ghc-6.12.0.20091010 -L/usr/lib64/pcre-light-0.3.1/ghc-6.12.0.20091010 -L/usr/lib64/parsec-2.1.0.1/ghc-6.12.0.20091010 -L/usr/lib64/ghc-6.12.0.20091010/filepath-1.1.0.3 -L/usr/lib64/ghc-6.12.0.20091010/containers-0.3.0.0 -L/usr/lib64/ghc-6.12.0.20091010/bytestring-0.9.1.5 -L/usr/lib64/ghc-6.12.0.20091010/array-0.3.0.0 -L/usr/lib64/ghc-6.12.0.20091010/base-4.2.0.0 -L/usr/lib64/ghc-6.12.0.20091010/integer-gmp-0.2.0.0 -L/usr/lib64/ghc-6.12.0.20091010/ghc-prim-0.2.0.0 -L/usr/lib64/ghc-6.12.0.20091010 -L/usr/lib64/ghc-6.12.0.20091010
Preprocessing library highlighting-kate-0.2.5...
Preprocessing executables for highlighting-kate-0.2.5...
Building highlighting-kate-0.2.5...

...

[41 of 61] Compiling Text.Highlighting.Kate.Syntax.Php ( Text/Highlighting/Kate/Syntax/Php.hs, dist/build/Text/Highlighting/Kate/Syntax/Php.p_o )
 
Ctrl+C
VIRT: 2.5G RAM, RSS 1.2G RAM, swap activity is large.

Attachments (1)

highlighting-kate-0.2.5.1-mangled.tar.gz (43.2 KB) - added by slyfox 10 years ago.
highlighting-kate-0.2.5.1-mangled.tar.gz - does not require external packages or cabal installed. just run ./build.sh

Download all attachments as: .zip

Change History (11)

comment:1 Changed 10 years ago by slyfox

Note: during nonprofiled build of this package memory consumtion holds at the level of 400 Megabytes

comment:2 Changed 10 years ago by igloo

Milestone: 6.12 branch
Type of failure: Compile-time performance bug

Thanks for the report. Is this a regression since 6.10?

That PHP module includes a couple of 50000 character lines, mostly composed of a list of strings. I suspect that's the problem, but haven't profiled at all.

It would be useful if someone could make a standalone, minimal testcase.

comment:3 in reply to:  2 ; Changed 10 years ago by slyfox

Replying to igloo:

Thanks for the report. Is this a regression since 6.10?

Yes, it is.

That PHP module includes a couple of 50000 character lines, mostly composed of a list of strings. I suspect that's the problem, but haven't profiled at all.

It would be useful if someone could make a standalone, minimal testcase.

What is minimal? Will be enough to strip this package to 3 almost unmodified .hs files but leave all external deps, like parsec and pcre-light, or you would like it fully buildable with ghc --make ?

Changed 10 years ago by slyfox

highlighting-kate-0.2.5.1-mangled.tar.gz - does not require external packages or cabal installed. just run ./build.sh

comment:4 Changed 10 years ago by slyfox

Results:

 [("Project name","The Glorious Glasgow Haskell Compilation System")
 ,("Project version","6.10.4")
 ,("Booter version","6.10.4")
...
$ /usr/bin/time ./build.sh
...
<<ghc: 16394164352 bytes, 31188 GCs, 32045920/173159800 avg/max bytes residency (44 samples), 358M in use, 0.00 INIT (0.00 elapsed), 29.55 MUT (35.98 elapsed), 22.76 GC (23.75 elapsed) :ghc>>
56.29user 1.31system 1:00.03elapsed 95%CPU (0avgtext+0avgdata 1530144maxresident)k
17376inputs+0outputs (64major+134666minor)pagefaults 0swaps

Compiled successfully. Ate ~400MB

 [("Project name","The Glorious Glasgow Haskell Compilation System")
 ,("Project version","6.12.0.20091121")
 ,("Booter version","6.10.4")
...
$ /usr/bin/time ./build.sh
...
Heap exhausted;
Current maximum heap size is 768000000 bytes (732 MB);
use `+RTS -M<size>' to increase it.
<<ghc: 32113310800 bytes, 61096 GCs, 303443071/754207120 avg/max bytes residency (45 samples), 761M in use, 0.00 INIT (0.00 elapsed), 39.25 MUT (41.92 elapsed), 148.00 GC (150.19 elapsed) :ghc>>
Command exited with non-zero status 251
187.83user 1.69system 3:12.25elapsed 98%CPU (0avgtext+0avgdata 3192560maxresident)k
9488inputs+0outputs (39major+216475minor)pagefaults 0swaps

Failed: Heap exhausted (I limited heap in build.sh by 768M)

comment:5 in reply to:  3 ; Changed 10 years ago by igloo

Replying to slyfox:

Replying to igloo:

That PHP module includes a couple of 50000 character lines, mostly composed of a list of strings. I suspect that's the problem, but haven't profiled at all.

It would be useful if someone could make a standalone, minimal testcase.

What is minimal? Will be enough to strip this package to 3 almost unmodified .hs files but leave all external deps, like parsec and pcre-light, or you would like it fully buildable with ghc --make ?

Eliminating external deps is the most important thing, as that makes it much easier to try in a development compiler, or to compare 2 compilers.

Apart from that, just making it as small as possible means that it is quicker to build, easier to see what is going on (in terms of both the source code, and the compiler debugging output), and makes it easy to add a regression test to the testsuite.

comment:6 in reply to:  5 Changed 10 years ago by slyfox

Eliminating external deps is the most important thing, as that makes it much easier to try in a development compiler, or to compare 2 compilers.

Apart from that, just making it as small as possible means that it is quicker to build, easier to see what is going on (in terms of both the source code, and the compiler debugging output), and makes it easy to add a regression test to the testsuite.

OK, attached sample does not depend on external stuff (12 .hs files, 9 of them are parsec-2 ones). Seems to need only compiler. I'm afraid I can't trim it down to 1 small file, it's very fragile and seems to depend on external inlining stuff. If I would know the nature of memory consumption - I might try to make syntetic test by hands.

comment:7 Changed 10 years ago by simonpj

Thanks for the nice test case.

Happily the performance hole is fixed in the HEAD, I believe by my recent patch to FloatOut:

Mon Dec  7 08:32:46 GMT 2009  simonpj@microsoft.com
  * Fix a nasty (and long-standing) FloatOut performance bug
  
  The effect was that, in deeply-nested applications, FloatOut would
  take quadratic time.  A good example was compiling 
      programs/barton-mangler-bug/Expected.hs
  in which FloatOut had a visible pause of a couple of seconds!
  Profiling showed that 40% of the entire compile time was being
  consumbed by the single function partitionByMajorLevel.
  
  The bug was that the floating bindings (type FloatBinds) was kept
  as a list, which was partitioned at each binding site.  In programs
  with deeply nested lists, such as
         e1 : e2 : e3 : .... : e5000 : []
  this led to quadratic behaviour.
  
  The solution is to use a proper finite-map representation;
  see the new definition of FloatBinds near the bottom of FloatOut.

    M ./compiler/simplCore/FloatOut.lhs -76 +126

Here are the stats for your run:

  15,066,832,088 bytes allocated in the heap
   2,907,321,496 bytes copied during GC
     188,073,960 bytes maximum residency (28 sample(s))
       6,128,376 bytes maximum slop
             412 MB total memory in use (7 MB lost due to fragmentation)

Note the residency is 188M, still large but more manageable.

Because 6.12.1 is otherwise ready to go we were planning to put this patch 6.12.2, rather than risk destabilising 6.12.1 (the patch has not been tested nearly as thoroughly as the rest of 6.12.1). Can you use the HEAD meanwhile?

I'm leaving the bug open just for this question to be resolved.

Simon

comment:8 in reply to:  7 Changed 10 years ago by slyfox

Replying to simonpj:

Note the residency is 188M, still large but more manageable.

Because 6.12.1 is otherwise ready to go we were planning to put this patch 6.12.2, rather than risk destabilising 6.12.1 (the patch has not been tested nearly as thoroughly as the rest of 6.12.1). Can you use the HEAD meanwhile?

I'm leaving the bug open just for this question to be resolved.

Sure. I'll wait 6.12.2.

Thanks!

comment:10 Changed 10 years ago by simonpj

Owner: set to igloo
Type: bugmerge

OK so I'll assign the ticket to Ian, to merge into 6.12.2, when 6.12.1 is out.

Simon

comment:11 Changed 10 years ago by igloo

Resolution: fixed
Status: newclosed

Merged

Note: See TracTickets for help on using tickets.