Opened 5 years ago

Last modified 3 years ago

#9421 new bug

Problems and workarounds when installing and using a 32bit GHC on 64bit Linux machine

Reported by: MikolajKonarski Owned by: MikolajKonarski
Priority: normal Milestone:
Component: Compiler Version: 7.8.3
Keywords: Cc: rwbarton, nh2, mikolaj.konarski@…
Operating System: Linux Architecture: x86_64 (amd64)
Type of failure: Installing GHC failed Test Case:
Blocked By: #9614 Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description (last modified by MikolajKonarski)

I'm compiling i386 binaries on amd64 linux and reporting problems and workarounds in this meta ticket.

The very first problem I encounter is a magic number mismatch message. This is basically the same problem as reported here:

https://www.mail-archive.com/ghc-devs@haskell.org/msg04439.html

I have empty .cabal and .ghc, no ghc in paths and I have old Ubuntu LTS (12.04.4) so I was able to install ia32-libs and a few other libs

I've downloaded

http://www.haskell.org/ghc/dist/7.8.3/ghc-7.8.3-i386-unknown-linux-deb7.tar.xz

configured, and make install says:

Installing library in
/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/haskell98-2.0.0.3
"utils/ghc-cabal/dist-install/build/tmp/ghc-cabal-bindist" copy libraries/haskell2010 dist-install "strip" '' '/mikolaj/env/7.8.3i386/local/src/ghc' '/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3' '/mikolaj/env/7.8.3i386/local/src/ghc/share/doc/ghc/html/libraries' 'v p dyn'
Installing library in
/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/haskell2010-1.1.2.0
"/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/bin/ghc-pkg" --force --global-package-db "/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/package.conf.d" update rts/dist/package.conf.install
Reading package info from "rts/dist/package.conf.install" ... done.
"utils/ghc-cabal/dist-install/build/tmp/ghc-cabal-bindist" register libraries/ghc-prim dist-install "/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/bin/ghc" "/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/bin/ghc-pkg" "/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3" '' '/mikolaj/env/7.8.3i386/local/src/ghc' '/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3' '/mikolaj/env/7.8.3i386/local/src/ghc/share/doc/ghc/html/libraries' NO  
ghc-cabal: Bad interface file: dist-install/build/GHC/CString.hi
magic number mismatch: old/corrupt interface file? (wanted 33214052, got
129742)
make[1]: *** [install_packages] Error 1
make: *** [install] Error 2

Change History (21)

comment:1 Changed 5 years ago by rwbarton

Cc: rwbarton added

comment:2 Changed 5 years ago by MikolajKonarski

Here is the trivial workaround (now also described on the wiki the wiki ):

CFLAGS=-m32 ./configure --prefix=.....

Now I can compile many packages and the 32bit executables work (e.g., alex, happy). I still struggle with gtk2hs and I haven't tried beasts like yesod, etc, though. The command I use is

cabal install --ghc-option="-optc-m32" --ghc-option="-opta-m32" --ghc-option="-optl-m32" --ld-option="-melf_i386"

Long story

The workaround is inspired by https://ghc.haskell.org/trac/ghc/ticket/8910 and http://www.haskell.org/pipermail/glasgow-haskell-users/2012-February/022014.html

I had to install many -dev:i386 versions of libraries on my Ubuntu to make it work. The most problematic was libgmp-dev, since my Ubuntu only tolerates amd64 or i386 version of this lib at once (zlib1g-dev:i386 didn't have that problem). I ended up installing just libgmp10:i386 and adding a hacky symlink instead of the libgmp-dev:i386 version:

sudo ln -s /usr/lib/i386-linux-gnu/libgmp.so.10.0.2 /usr/lib/i386-linux-gnu/libgmp.so

The remaining gtk2hs problem

This is probably a bug in gtk2hs-buildtools (I reinstalled them for 32bits with many extra flags, but no improvement), but I won't report it on gtk2hs issue tracker until I research a bit more. I try monstrosities like

CFLAGS=-m32 cabal install glib -j1 --ghc-option="-optc-m32" --ghc-option="-opta-m32" --ghc-option="-optl-m32" --ld-option="-melf_i386" -v3 --gcc-option="-m32" --configure-option="CFLAGS=-m32" --ghc-option="-optl-Wl,-melf_i386"

but I'm still getting

/tmp/glib-0.13.0.1-1099/glib-0.13.0.1/dist/setup/setup configure --verbose=3
--ghc --prefix=/mikolaj/.cabal --bindir=/mikolaj/.cabal/bin
--libdir=/mikolaj/.cabal/lib --libsubdir=i386-linux-ghc-7.8.3/glib-0.13.0.1
--libexecdir=/mikolaj/.cabal/libexec --datadir=/mikolaj/.cabal/share
--datasubdir=i386-linux-ghc-7.8.3/glib-0.13.0.1
--docdir=/mikolaj/.cabal/share/doc/i386-linux-ghc-7.8.3/glib-0.13.0.1
--htmldir=/mikolaj/.cabal/share/doc/i386-linux-ghc-7.8.3/glib-0.13.0.1/html
--haddockdir=/mikolaj/.cabal/share/doc/i386-linux-ghc-7.8.3/glib-0.13.0.1/html
--sysconfdir=/mikolaj/.cabal/etc --configure-option=CFLAGS=-m32 --user
--flags=closure_signals --extra-prog-path=/mikolaj/.cabal/bin
--constraint=utf8-string ==0.3.8 --constraint=text ==1.1.1.3
--constraint=containers ==0.5.5.1 --constraint=bytestring ==0.10.4.0
--constraint=base ==4.7.0.1 --disable-tests --disable-benchmarks
--gcc-option=-m32 --ghc-option=-optc-m32 --ghc-option=-opta-m32
--ghc-option=-optl-m32 --ghc-option=-optl-Wl,-melf_i386 --ld-option=-melf_i386
[1 of 2] Compiling Gtk2HsSetup      ( Gtk2HsSetup.hs, dist/setup-wrapper/Gtk2HsSetup.o )
/tmp/ghc1232_0/ghc1232_4.s: Assembler messages:

/tmp/ghc1232_0/ghc1232_4.s:43:0:
     Error: invalid instruction suffix for `push'

I haven't looked inside the generated files nor inside the gtk2hs-buildtools yet.

Last edited 4 years ago by MikolajKonarski (previous) (diff)

comment:3 Changed 5 years ago by MikolajKonarski

This is now documented on the wiki.

The problem with GTK is indeed in the gtk2hs setup scripts: https://github.com/gtk2hs/gtk2hs/issues/49

This suggests that on multi-arch OSes it would be helpful if GHC called gcc and linker (and itself) with the flags implied by the target arch. I guess we don't support gcc and ld old enough to reject such options, so the only problem is verbosity of the calls when they are shown during debugging. But then, having options spelled out explicitly tends to be good for debugging.

comment:4 Changed 5 years ago by MikolajKonarski

I've worked around the gtk problems by replacing ghc with

ghc-7.8.3 -optc-m32 -opta-m32 -optl-m32 $@

and by setting up the following symlinks (installing both i386 and amd64 versions of gtk dev libs is not supported on my Ubuntu):

sudo ln -s /lib/i386-linux-gnu/libglib-2.0.so.0 /usr/lib/i386-linux-gnu/libglib-2.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libgobject-2.0.so.0 /usr/lib/i386-linux-gnu/libgobject-2.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libgio-2.0.so.0 /usr/lib/i386-linux-gnu/libgio-2.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libcairo.so.2 /usr/lib/i386-linux-gnu/libcairo.so
sudo ln -s  /usr/lib/i386-linux-gnu/libpango-1.0.so.0 /usr/lib/i386-linux-gnu/libpango-1.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libpangocairo-1.0.so.0 /usr/lib/i386-linux-gnu/libpangocairo-1.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libgthread-2.0.so.0 /usr/lib/i386-linux-gnu/libgthread-2.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libgtk-x11-2.0.so.0 /usr/lib/i386-linux-gnu/libgtk-x11-2.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libgdk-x11-2.0.so.0 /usr/lib/i386-linux-gnu/libgdk-x11-2.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libatk-1.0.so.0 /usr/lib/i386-linux-gnu/libatk-1.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libpangoft2-1.0.so.0 /usr/lib/i386-linux-gnu/libpangoft2-1.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libgdk_pixbuf-2.0.so.0 /usr/lib/i386-linux-gnu/libgdk_pixbuf-2.0.so
sudo ln -s  /usr/lib/i386-linux-gnu/libfreetype.so.6 /usr/lib/i386-linux-gnu/libfreetype.so
sudo ln -s  /usr/lib/i386-linux-gnu/libfontconfig.so.1 /usr/lib/i386-linux-gnu/libfontconfig.so
sudo ln -s  /usr/lib/i386-linux-gnu/libX11.so.6 /usr/lib/i386-linux-gnu/libX11.so

The GTK calls in the generated binary work great and the binary is reported as

ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x46cf78948f9bad6f6beed815d6c0f71679830324, stripped

However, there is an unrelated problem: whenever compressions is required the binary crashes with

user error (Codec.Compression.Zlib: incompatible zlib version)

The ldd utility reports

	libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xf7025000)

recompiling zlib package with zlib1g-dev:i386 or without and with various hacky symlinks doesn't help.

comment:5 Changed 5 years ago by MikolajKonarski

To work around zlib problems I had to add hsc2hs options:

cabal install zlib --ghc-option="-optc-m32" --ghc-option="-opta-m32" --ghc-option="-optl-m32" --ld-option="-melf_i386" --hsc2hs-options="--cflag=-m32 --lflag=-m32"

This was inspired by

http://www.haskell.org/pipermail/glasgow-haskell-users/2009-November/018070.html

and was needed despite

https://ghc.haskell.org/trac/ghc/ticket/3400#comment:16

Now everything works correctly for me. Next up for testing would be a TH heavy package or just a very large application. Yesod comes to mind. Please report here if you try that.

Last edited 5 years ago by MikolajKonarski (previous) (diff)

comment:6 Changed 5 years ago by MikolajKonarski

Description: modified (diff)
Owner: set to MikolajKonarski
Summary: magic number mismatch when installing 32bit on 64bit machineProblems and workarounds when installing and using a 32bit GHC on 64bit Linux machine

comment:7 Changed 5 years ago by MikolajKonarski

Regarding the zlib problem (wrong hsc2hs options when comping zlib for 32bits on 64 bits), https://ghc.haskell.org/trac/ghc/ticket/3400#comment:16 (further changed in #7760) could not possibly help, because it only passes options to hsc2hs when compiling ghc

https://github.com/ghc/hsc2hs/blob/4a0f67704d89712f8493a0c7eccffa9243d6ef09/ghc.mk#L30

and passes the options not from ghc but from CONF_* variables.

I guess hsc2hs links to the correct version of libz after all, but the name it uses is a bit different, even though equivalent, to the name the internal checks in the haskell package zlib expect and so zlib panics. If that's the case I could file a bug report agains zlib. But if we go the route of automatically adding m32 to all invocations whenever we detect the 32bits on 64 bits scenario, fixes to zlib (nor to gtk) won't be needed and possibly some other problems, with yet different packages, will be avoided.

For the record, the faulty call to hsc2hs looks rather reasonable, except for the mention of Dx86_64_BUILD_ARCH (are our i386 GHCs releases cross-compiled?), but even that should not cause any problems normally, I suppose.

("/home/mikolaj/local/bin/hsc2hs",["--cc=/usr/bin/gcc","--ld=/usr/bin/gcc",
"--cflag=-D__GLASGOW_HASKELL__=708","--cflag=-Dlinux_BUILD_OS=1",
"--cflag=-Dx86_64_BUILD_ARCH=1","--cflag=-Dlinux_HOST_OS=1",
"--cflag=-Di386_HOST_ARCH=1","--cflag=-Idist/build/autogen",
"--cflag=-include","--cflag=dist/build/autogen/cabal_macros.h",
"--lflag=-lz", [...], "--lflag=-L/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/rts-1.0",
"--lflag=-Wl,-R,/mikolaj/env/7.8.3i386/local/src/ghc/lib/ghc-7.8.3/rts-1.0","--lflag=-lm",
"--lflag=-lrt","--lflag=-ldl","-o","dist/build/Codec/Compression/Zlib/Stream.hs",
"Codec/Compression/Zlib/Stream.hsc"])

comment:8 Changed 5 years ago by MikolajKonarski

Blocked By: 6086 added

comment:9 Changed 5 years ago by rwbarton

Hi Mikolaj, thanks so much for your work on this!

Could you clarify in the very nice wiki page you have created that these instructions pertain to installing the x86 binary distribution on x86_64 systems? Another possible approach is to build one's own x86_64-to-x86 cross-compiler and cross-compiled x86 compiler from source, following the general directions at Building/CrossCompiling. This currently also works fairly well, though I haven't tested it as thoroughly as you have tested running the x86 binary distribution on x86_64.

comment:10 Changed 5 years ago by rwbarton

I fear it might be difficult, when building the Linux/x86 GHC on an Linux/x86 system, to anticipate all the possible systems that might be capable of building and running Linux/x86 executables (at least Linux/x86_64, but possibly others now or in the future?), and what command-line options the C toolchain needs to build such executables.

But here is a slightly different suggestion. Inside $PREFIX/lib/ghc-7.8.3 you will find a file settings, which contains (among other things) the options that GHC passes to the C compiler, linker, etc. GHC reads this file at startup, so you can change any of these settings just by editing the file. Can you see if it is possible to effect all of your needed options like -optc=-m32 via changes to this settings file? Those options should be respected by other Haskell tools like cabal and hsc2hs, so this may fix your other issues automatically. (If those tools do not respect these settings, it is probably a bug in those tools that should be fixed.)

If it's not possible because you need (for example) a new setting "assembler flags", it should be fairly easy to add such settings to this file.

If the needed changes can be made through the settings file, then we won't need to modify GHC itself at all, and we can have the binary distribution configure script automatically detect that you are trying to install an x86 GHC on an x86_64 system, and modify the settings file accordingly.

(By the way I'm a bit confused by your option -optl-m32, because my linker doesn't have a -m32 option, only -melf_i386. Does that -optl-m32 really do anything?)

comment:11 Changed 5 years ago by rwbarton

Blocked By: 6086 removed

(The problem here isn't that we aren't passing ld flags to cabal, rather that we don't know what the right ld flags are at all.)

comment:12 in reply to:  9 Changed 5 years ago by MikolajKonarski

Replying to rwbarton:

Thank you for your feedback. I've updated the wiki page.

Re anticipating setting for all possible systems, how would it hurt, if i386 GHC, whenever it detected that TARGET /= HOST, always kept setting something like

--ghc-option="-optc-m32" --ghc-option="-opta-m32" --ghc-option="-optl-m32" --ld-option="-melf_i386"

It's not x86_64 specific, at least, isn't it? As a default, it's less clean but more effective than the optimistic empty list of options.

Thank you for the lib/settings suggestion. I will try that and report here.

Last edited 5 years ago by MikolajKonarski (previous) (diff)

comment:13 Changed 5 years ago by rwbarton

Well, ld -melf_i386 is surely OS-dependent, as neither OS X nor Windows uses ELF binaries. So the options you listed are not always the correct ones. (Also, that is actually an option needed for Cabal, not for GHC.)

But speaking more generally, I don't see any advantage to GHC (or Cabal) trying to determine the proper way to invoke the C compiler at runtime. We already have a configure script that's run during installation of the binary distribution whose exact purpose is to determine things like what flags need to be passed to the C toolchain. So that is the logical place to put this 32-bit-on-64-bit detection.

comment:14 in reply to:  13 Changed 5 years ago by MikolajKonarski

Replying to rwbarton:

OK, I think you are right the settings file is the better place to set options according to the ACTUAL_HOST/TARGET pair. I tried modifying the file manually and, at least initially, "-m32" for "C compiler flags" is necessary and sufficient for GHC alone. It apparently gets passed through to gcc when it does -x assembler, so we don't need "assembler flags" nor "-opta-m32". Also, "-optl-m32" is not needed at all, at least for simple packages. This is all very encouraging. However, I need to manually pass --ld-option="-melf_i386" to cabal, because the "ld flags" are ignored by cabal. I will investigate.

comment:15 Changed 5 years ago by MikolajKonarski

I tried investigating why cabal ignores "ld flags" from the $PREFIX/lib/ghc-7.8.3/settings file. Cabal doesn't now about "ld flags", but it does know about "Ld Linker flags". However, just setting "Ld Linker flags" in the file doesn't help. I code-dived in GHC and it mentions each of the keys exactly once: in the template for the settings file

settings.in: ("ld flags", "@SettingsLdFlags@"),

and when dealing with commandline arguments

ghc/Main.hs-mode_flags =
[...]
ghc/Main.hs-  [ Flag k'                      (PassFlag (setMode (printSetting k)))
ghc/Main.hs-  | k <- ["Project version",
[...]
ghc/Main.hs:          "Ld Linker flags"],

It seems the two keys are different things, from different namespaces --- almost all of the keys in ghc/Main.h differ from their settings.in wording.

BTW, there is also a single possibily relevant comment in the cabal source code, but I hope the part about ld is outdated and, anyway, I don't understand it:

Cabal/Distribution/Simple.hs-  -- The C compiler's compilation and linker flags (e.g.
Cabal/Distribution/Simple.hs:  -- "C compiler flags" and "Gcc Linker flags" from GHC) have already
Cabal/Distribution/Simple.hs-  -- been merged into ccFlags, so we set both CFLAGS and LDFLAGS
Cabal/Distribution/Simple.hs-  -- to ccFlags
Cabal/Distribution/Simple.hs-  -- We don't try and tell configure which ld to use, as we don't have
Cabal/Distribution/Simple.hs-  -- a way to pass its flags too

Looking at GHC code for other settings, I'd guess that GHC may not read and merge "ld flags" with other ld options (perhaps because it doesn't call ld, but cabal does?). And so cabal doesn't learn this part of ld setting from GHC. But the GHC source code naming of the gcc linker flags and the /usr/bin/ld flags is not easy to follow and sometimes the gcc linker flags seem to be used for the system linker, etc., so it's not obvious to me if GHC or cabal is to blame or both.

comment:16 Changed 5 years ago by MikolajKonarski

Duncan helped me with that one, so I have some hypotheses now. The old GHC 7.6.3 has no "ld flags" nor "Ld Linker flags" in the settings file, but it somehow produces "Ld Linker flags" in the ghc --info output and this is what cabal expects. However, GHC 7.8.3 has "ld flags" in the settings file and doesn't produce "Ld Linker flags" at all in ghc --info. This may be a bug or a half-done transition to another convention, but right now it fools the newest cabal. BTW, I can't probably specify "Ld Linker flags" manually in the settings file, because it's supposed to be a list of options, not a string, as the other options. Perhaps "ld flags" was intended to let people extend "Ld Linker flags" manually via the settings file? Anyway, it doesn't work right now.

comment:17 Changed 5 years ago by rwbarton

Yes, I looked into the cabal linker flags issue too, but forgot to mention it, sorry! Indeed there were several changes between 7.6.3 and 7.8 but cabal was not updated accordingly. (There was some minor fallout on the GHC side also, see #9614.) I think best is to update cabal to parse the new (7.8) fields if present and if not fall back on the old (7.6.3) fields.

comment:18 Changed 5 years ago by duncan

I don't mind if we have to update Cabal here. 7.8 is out and we can't undo that. But I'd rather that we had some test in place to look at the ghc --info output (and might as well cover the things mentioned in #9614 while we're at it), since we do have tools that rely on them and ghc should not change them unintentionally.

comment:19 Changed 5 years ago by MikolajKonarski

Blocked By: 9614 added

comment:20 Changed 5 years ago by nh2

Cc: nh2 added

comment:21 Changed 3 years ago by MikolajKonarski

Cc: mikolaj.konarski@… added
Note: See TracTickets for help on using tickets.