Opened 13 months ago

Last modified 13 months ago

#15576 new bug

Hadrian puts its build tree in the wrong place

Reported by: simonpj Owned by:
Priority: normal Milestone: 8.6.1
Component: Compiler Version: 8.4.3
Keywords: Cc: andrey.mokhov@…
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

I have a source tree in my slow, network-mounted backed-up filestore /home/simonpj/ghc, and a separate link-tree in my fast, physically-connected, non-backed up filestore /playpen/ghc. Every file in /playpen/ghc is a symlink to the corresponding source file in /home/simonpj/ghc.

But when I do

$ cd /playpen/ghc
$ hadrian/build.sh -c -j4 --flavour=quick  

I find a new _build director in /home/simonpj/ghc. This is bad! It should be in /playpen/ghc. Otherwise (a) it's slower, (b) it creates backup churn and (c) as it happens, I ran out of space on the backed-up filestore.

Can this be fixed? It's a bad Hadrian shortcoming.

Change History (12)

comment:1 Changed 13 months ago by snowleopard

You can change the build directory via the command line using --build-root=PATH or -oPATH.

Can you try this?

hadrian/build.sh -c -j4 --flavour=quick -o/playpen/ghc/_build

This is documented here: https://github.com/snowleopard/hadrian#command-line-flags.

We can also add support for specifying the build directory in UserSettings so that you don't need to specify it each time via the command line.

Last edited 13 months ago by snowleopard (previous) (diff)

comment:2 Changed 13 months ago by simonpj

I don't particularly want to change the build directory; I just want it to be right here at the root of my build tree, where it has always been. (That is -o$PWD, I suppose.) Is that not a good default place for it to be?

I suppose I can get used to doing something in UserSettings in every build tree; but I'd prefer the old behaviour. Is it difficult to achieve for some reason?

comment:3 Changed 13 months ago by snowleopard

Ah, now I'm confused.

If you run Hadrian from /playpen/ghc I don't understand why build results appear in /home/simonpj/ghc.

Is it difficult to achieve for some reason?

Actually, I would say achieving the current behaviour seems difficult!

I think the magic symlink-traversal code in build.sh and build.cabal.sh has something to do with it. I believe it was added by Herbert, but I don't recall why.

Can you try building and running Hadrian directly using Cabal?

cd hadrian
cabal new-build --disable-profiling --disable-documentation -j exe:hadrian
cabal new-run hadrian -- -c -j4 --flavour=quick --directory=".."

(These commands are copied from build.cabal.sh, which you are calling from build.sh.)

Last edited 13 months ago by snowleopard (previous) (diff)

comment:4 Changed 13 months ago by simonpj

I double-checked.

  • I removed /home/simonpj/ghc/_build entirely
  • I removed /playpen/ghc/_build entirely
  • I did cd /playpen/ghc
  • And then hadrian/build.sh -c -j20 --flavour=quick

Result:

  • a _build directory immediately appeared in /home/simonpj/ghc, populated with generated, hadrian, stage0, stage1.
  • The hadrian/ directory contains .shake.database.
  • No _build directory appeared in /playpen/ghc

comment:5 Changed 13 months ago by simonpj

OK I did what you ask in comment:3. The first cabal new-build completed fine; no _build directories anywhere.

The second command cabal new-run command is still running. The start of its output is below. (NB: ~/5builds/HEAD-3 is the build tree that I was previously calling /playpen/ghc; but it really IS the build tree.)

Results so far: _build is created in /playpen/ghc (or in fact ~/5builds/HEAD-3); there is no _build in /users/simonpj/ghc.

Note the messages from configure about .git...

simonpj@cam-05-unx:~/5builds/HEAD-3/hadrian$ cabal new-run hadrian -- -c -j4 --flavour=quick --directory=".."
Up to date
| Running boot...
| Run Configure ".": hadrian/cfg/system.config.in (and 4 more) => hadrian/cfg/system.config (and 4 more)
configure: WARNING: cannot determine snapshot version: no .git directory and no VERSION file
configure: WARNING: cannot determine snapshot revision: no .git directory and no 'GIT_COMMIT_ID' file
Warning: libraries/text/text.cabal:4:1: The field "bug-reports" is specified
more than once at positions 4:1, 42:1
| Copy file: settings => _build/stage1/lib/settings
| Copy file: utils/hsc2hs/template-hsc.h => _build/stage1/lib/template-hsc.h
| Copy file: driver/ghci-usage.txt => _build/stage1/lib/ghci-usage.txt
| Copy file: llvm-passes => _build/stage1/lib/llvm-passes
| Copy file: llvm-targets => _build/stage1/lib/llvm-targets
Warning: utils/hpc/hpc-bin.cabal:11:2: Tabs used as indentation at 11:2
| Copy file: driver/ghc-usage.txt => _build/stage1/lib/ghc-usage.txt
| Copy file: driver/ghc-usage.txt => _build/stage0/lib/ghc-usage.txt
| Copy file: llvm-passes => _build/stage0/lib/llvm-passes
| Copy file: llvm-targets => _build/stage0/lib/llvm-targets
| Copy file: driver/ghci-usage.txt => _build/stage0/lib/ghci-usage.txt
| Copy file: settings => _build/stage0/lib/settings
| Successfully generated _build/stage1/bin/ghc-split.
| Make '_build/stage1/bin/ghc-split' executable.
...more

It appears that my build-tree link script does not add a symlink to .git. I have no idea why; there is a symlink to other dot-files like .gitignore. Here's the script

#!/usr/bin/perl

if (-l "Makefile") {
    $source_tree = `ls -l Makefile`;
    $source_tree =~ s/^.*->\s*(.*)\/Makefile\s*$/$1/;
    print "Linking to source dir: $source_tree\n";
    system("lndir $source_tree");
    if ($? == 0) {
	system("kill-dangling-symlinks");
    } else {
	print "errors from lndir; NOT removing dangling symlinks\n";
    }
} else {
    print "Linked Makefile not found.";
    exit(1);
}

comment:6 Changed 13 months ago by snowleopard

Aha, it looks like it works. Presumably missing .git files are only important for making releases etc. where we want to record the build commit.

Did the build eventually complete with GHC in /playpen/ghc/_build/stage1/bin?

I'll investigate the issue with build.sh logic that negates your symlink-based setup.

comment:7 Changed 13 months ago by simonpj

Did the build eventually complete with GHC in /playpen/ghc/_build/stage1/bin?

Yes, it did.

comment:8 Changed 13 months ago by simonpj

But how can I build stage2?

comment:9 Changed 13 months ago by snowleopard

It is Stage2 GHC.

Here is how Hadrian build tree is organised:

  • stage0 contains everything built by the Stage0 (bootstrapping) GHC, including the Stage1 GHC, which lives in _build/stage0/bin/ghc.exe.
  • stage1 contains everything built by the Stage1 GHC, including the Stage2 GHC in _build/stage1/bin/ghc.exe.
  • stage2 contains everything built by the Stage2 GHC, for example Haddock. Stage3 GHC will also be here.

comment:10 Changed 13 months ago by snowleopard

We have now removed the symlink traversal logic from Hadrian build scripts, so if you run hadrian/build.sh from /playpen/ghc the results will appear in /playpen/ghc/_build.

Could you please check?

comment:11 Changed 13 months ago by simonpj

Could you please check?

How do I check? I assume I have to update Hadrian somehow?

comment:12 Changed 13 months ago by snowleopard

Here is how to update Hadrian: https://ghc.haskell.org/trac/ghc/wiki/Building/Hadrian/QuickStart#UpdatingHadrian

Hopefully, we will soon merge Hadrian into the GHC tree and there will be no need for fiddling with submodules.

Note: See TracTickets for help on using tickets.