Opened 4 years ago

Closed 4 years ago

#10877 closed bug (fixed)

x86_64: dll-split: out of memory (requested 1099512676352 bytes)

Reported by: RyanGlScott Owned by:
Priority: highest Milestone: 8.0.1
Component: Compiler Version: 7.11
Keywords: Cc: gcampax, simonmar
Operating System: Linux Architecture: x86_64 (amd64)
Type of failure: Building GHC failed Test Case:
Blocked By: Blocking:
Related Tickets: #10682 Differential Rev(s): Phab:D1405
Wiki Page:

Description

Attempting to build GHC on an x86_64 Linux machine fails for me:

$ make
===--- building phase 0
make --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for `phase_0_builds'.
===--- building phase 1
make --no-print-directory -f ghc.mk phase=1 phase_1_builds
make[1]: Nothing to be done for `phase_1_builds'.
===--- building final phase
make --no-print-directory -f ghc.mk phase=final all
inplace/bin/dll-split compiler/stage2/build/.depend-v-dyn.haskell "DynFlags" "Annotations ApiAnnotation Avail Bag BasicTypes Binary BooleanFormula BreakArray BufWrite Class CmdLineParser CmmType CoAxiom ConLike Coercion Config Constants CoreArity CoreFVs CoreSubst CoreSyn CoreTidy CoreUnfold CoreUtils CoreSeq CoreStats CostCentre Ctype DataCon Demand Digraph DriverPhases DynFlags Encoding ErrUtils Exception FamInstEnv FastFunctions FastMutInt FastString Fingerprint FiniteMap ForeignCall Hooks HsBinds HsDecls HsDoc HsExpr HsImpExp HsLit PlaceHolder HsPat HsSyn HsTypes HsUtils HscTypes IOEnv Id IdInfo IfaceSyn IfaceType InstEnv Kind Lexeme Lexer ListSetOps Literal Maybes MkCore MkId Module MonadUtils Name NameEnv NameSet OccName OccurAnal OptCoercion OrdList Outputable PackageConfig Packages Pair Panic PatSyn PipelineMonad Platform PlatformConstants PprCore PrelNames PrelRules Pretty PrimOp RdrName Rules Serialized SrcLoc StaticFlags StringBuffer TcEvidence TcRnTypes TcType TrieMap TyCon Type TypeRep TysPrim TysWiredIn Unify UniqFM UniqSet UniqSupply Unique Util Var VarEnv VarSet Bitmap BlockId ByteCodeAsm ByteCodeInstr ByteCodeItbls CLabel Cmm CmmCallConv CmmExpr CmmInfo CmmMachOp CmmNode CmmSwitch CmmUtils CodeGen.Platform CodeGen.Platform.ARM CodeGen.Platform.ARM64 CodeGen.Platform.NoRegs CodeGen.Platform.PPC CodeGen.Platform.PPC_Darwin CodeGen.Platform.SPARC CodeGen.Platform.X86 CodeGen.Platform.X86_64 Hoopl Hoopl.Dataflow InteractiveEvalTypes MkGraph PprCmm PprCmmDecl PprCmmExpr Reg RegClass SMRep StgCmmArgRep StgCmmClosure StgCmmEnv StgCmmLayout StgCmmMonad StgCmmProf StgCmmTicky StgCmmUtils StgSyn Stream"
dll-split: out of memory (requested 1099512676352 bytes)
make[1]: *** [compiler/stage2/dll-split.stamp] Error 1
make: *** [all] Error 2

Change History (17)

comment:1 Changed 4 years ago by thomie

That's odd. Works fine for me, and Phabricator is also x86_64.

  • Did you try 'make maintainer-clean && ./boot && ./configure && make'?
  • Maybe it is something specific to dll-split. You could try to apply the patch from Phab:D677. I don't think the patch will apply cleanly, but it's a small patch, so just try to fix the merge conflict. Only applying the change in DynFlags might also be enough.

comment:2 Changed 4 years ago by RyanGlScott

I agree that this is odd, since the build does work for me on other x86_64 machines. I'm not sure what's different about this one.

make maintainer-clean && ./boot && ./configure && make doesn't work. I tried manually applying the changes from Phab:D677 and running that, but unfortunately, a similar issue creeps up:

[rgscott@tank ghc]$ make
===--- building phase 0
make --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for `phase_0_builds'.
===--- building phase 1
make --no-print-directory -f ghc.mk phase=1 phase_1_builds
make[1]: Nothing to be done for `phase_1_builds'.
===--- building final phase
make --no-print-directory -f ghc.mk phase=final all
"inplace/bin/ghc-stage2" -hisuf dyn_hi -osuf  dyn_o -hcsuf dyn_hc -fPIC -dynamic  -O0 -H64m     -hide-all-packages -i -iutils/ghctags/. -iutils/ghctags/dist-install/build -iutils/ghctags/dist-install/build/autogen -Iutils/ghctags/dist-install/build -Iutils/ghctags/dist-install/build/autogen     -optP-include -optPutils/ghctags/dist-install/build/autogen/cabal_macros.h -package-id Cabal-1.23.0.0-inplace -package-id base-4.8.2.0-inplace -package-id containers-0.5.6.2-inplace -package-id ghc-7.11.20150916-inplace -XHaskell2010  -no-user-package-db -rtsopts      -odir utils/ghctags/dist-install/build -hidir utils/ghctags/dist-install/build -stubdir utils/ghctags/dist-install/build   -c utils/ghctags/./Main.hs -o utils/ghctags/dist-install/build/Main.dyn_o 
ghc-stage2: out of memory (requested 1099512676352 bytes)
make[1]: *** [utils/ghctags/dist-install/build/Main.dyn_o] Error 1
make: *** [all] Error 2

comment:3 Changed 4 years ago by thomie

Well it made it to ghc-stage2 at least. Could you try running the testsuite (run 'make test'). Do all tests fail, or just a few?

comment:4 Changed 4 years ago by RyanGlScott

All tests fail—I get the same out of memory error from ghc-stage2:

$ make
===--- building phase 0
make --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for `phase_0_builds'.
===--- building phase 1
make --no-print-directory -f ghc.mk phase=1 phase_1_builds
make[1]: Nothing to be done for `phase_1_builds'.
===--- building final phase
make --no-print-directory -f ghc.mk phase=final all
"inplace/bin/ghc-stage2" -hisuf dyn_hi -osuf  dyn_o -hcsuf dyn_hc -fPIC -dynamic  -O0 -H64m     -hide-all-packages -i -iutils/ghctags/. -iutils/ghctags/dist-install/build -iutils/ghctags/dist-install/build/autogen -Iutils/ghctags/dist-install/build -Iutils/ghctags/dist-install/build/autogen     -optP-include -optPutils/ghctags/dist-install/build/autogen/cabal_macros.h -package-id Cabal-1.23.0.0-inplace -package-id base-4.8.2.0-inplace -package-id containers-0.5.6.2-inplace -package-id ghc-7.11.20150916-inplace -XHaskell2010  -no-user-package-db -rtsopts      -odir utils/ghctags/dist-install/build -hidir utils/ghctags/dist-install/build -stubdir utils/ghctags/dist-install/build   -c utils/ghctags/./Main.hs -o utils/ghctags/dist-install/build/Main.dyn_o 
ghc-stage2: out of memory (requested 1099512676352 bytes)
make[1]: *** [utils/ghctags/dist-install/build/Main.dyn_o] Error 1
make: *** [all] Error 2
Last edited 4 years ago by RyanGlScott (previous) (diff)

comment:5 Changed 4 years ago by thomie

Cc: gcampax added
Milestone: 8.0.1
Priority: normalhighest

Well I have no idea, but that's a release blocker. Maybe @gcampax can debug this, as the co-author of 0d1a8d09f452977aadef7897aa12a8d41c7a4af0 (#9706).

You can use ./configure --disable-large-address-space in the meantime.

Last edited 4 years ago by thomie (previous) (diff)

comment:6 Changed 4 years ago by rwbarton

Presumably it is something special about the reporter's system. Something like overcommit being disabled (though in my preliminary tests this did not cause reserving large memory areas to fail). What distro is it?

comment:7 Changed 4 years ago by RyanGlScott

Tell me if you want more specific distro info than this:

$ uname -a
Linux tank.soic.indiana.edu 3.10.0-229.11.1.el7.x86_64 #1 SMP Wed Jul 22 12:06:11 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.1 (Maipo)

$ cat /proc/meminfo 
MemTotal:       263928412 kB
MemFree:        52989964 kB
MemAvailable:   167374740 kB
Buffers:            3540 kB
Cached:         109364396 kB
SwapCached:        52700 kB
Active:         143228364 kB
Inactive:       59588436 kB
Active(anon):   87890144 kB
Inactive(anon):  5694784 kB
Active(file):   55338220 kB
Inactive(file): 53893652 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:      134217724 kB
SwapFree:       134030536 kB
Dirty:                28 kB
Writeback:             0 kB
AnonPages:      93435332 kB
Mapped:           203232 kB
Shmem:            136064 kB
Slab:            5871708 kB
SReclaimable:    5490788 kB
SUnreclaim:       380920 kB
KernelStack:       18336 kB
PageTables:       230584 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    266181928 kB
Committed_AS:   101990088 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      720588 kB
VmallocChunk:   34224804536 kB
HardwareCorrupted:     0 kB
AnonHugePages:  92712960 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      310208 kB
DirectMap2M:    59412480 kB
DirectMap1G:    210763776 kB

comment:8 Changed 4 years ago by gcampax

Is it possible that you have an address space limit configured (ulimit -v or /etc/security/limits.conf )? If so, is it ok to disable it to run any Haskell code, or is it a policy to have it? Unfortunately the large address space support fails if the address space limit is low.

comment:9 Changed 4 years ago by RyanGlScott

Yes, it does appear that I have an address space limit:

$ ulimit -v
67108864

This isn't a machine I have root access on, so I don't believe this can be changed easily.

Is 67.1 GB of address space not enough?

comment:10 Changed 4 years ago by bgamari

Cc: simonmar added

simonmar, perhaps we should be looking at the ulimit and adjust our allocation request downwards if necessary?

comment:11 Changed 4 years ago by simonmar

Urgh, the problem is that making the size of the area dynamic imposes a runtime cost - we have to read that value every time the GC visits an object. But I can't think of an alternative.

comment:12 Changed 4 years ago by bgamari

Presumably it will sit in cache in this case, no?

comment:13 Changed 4 years ago by bgamari

Differential Rev(s): Phab:D1354
Status: newpatch

comment:14 Changed 4 years ago by bgamari

Differential Rev(s): Phab:D1354Phab:D1405

getrlimit and friends seem to be a bit finicky. The new approach (Phab:D1405) is just to successively reduce the allocation size until we find a size that the kernel will give us.

comment:15 Changed 4 years ago by Ben Gamari <ben@…>

In f5974c88/ghc:

rts: Make MBLOCK_SPACE_SIZE dynamic

Previously this was introduced in D524 as a compile-time constant.
Sadly, this isn't flexible enough to allow for environments where
ulimits restrict the maximum address space size (see, for instance,

Consequently, we are forced to make this dynamic. In principle this
shouldn't be so terrible as we can place both the beginning and end
addresses within the same cache line, likely incurring only one or so
additional instruction in HEAP_ALLOCED.

Test Plan: validate

Reviewers: austin, simonmar

Reviewed By: simonmar

Subscribers: thomie

Differential Revision: https://phabricator.haskell.org/D1353

GHC Trac Issues: #10877

comment:16 Changed 4 years ago by Ben Gamari <ben@…>

In 4ad2a8f/ghc:

rts/posix: Reduce heap allocation amount on mmap failure

Since the two-step allocator the RTS asks the kernel for a large upfront
mmap'd region of memory (on the order of terabytes). While we have no
expectation that this entire region will be backed by physical memory,
this scheme nevertheless fails on some systems with resource limits.
Here we use a back-off scheme to reduce our allocation request until we
find a size agreeable to the kernel. Fixes #10877.

This also fixes a latent bug wherein the heap reservation retry logic
would fail to free the previously reserved address space, which would
likely result in a heap allocation failure.

Test Plan:
set address space limit with `ulimit -v 67108864` and try running
a compiled program

Reviewers: simonmar, austin

Reviewed By: simonmar

Subscribers: thomie, RyanGlScott

Differential Revision: https://phabricator.haskell.org/D1405

GHC Trac Issues: #10877

comment:17 Changed 4 years ago by bgamari

Resolution: fixed
Status: patchclosed
Note: See TracTickets for help on using tickets.