Opened 20 months ago

Closed 19 months ago

Last modified 16 months ago

#14697 closed bug (fixed)

Redundant computation in fingerprintDynFlags when compiling many modules

Reported by: niteria Owned by:
Priority: normal Milestone:
Component: Compiler Version:
Keywords: Cc: simonmar
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: Compile-time performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): phab:D4445
Wiki Page:

Description (last modified by niteria)

I profiled a build of a production code base with thousands of modules and computing fingerprintDynFlags is 7% of time and 14% of allocations.

Here's a synthetic test case inspired by what I observed:

SIZE=1000                                             

for i in $(seq -w 1 $SIZE); do                      
  echo "module A$i where" > A$i.hs                  
  echo "data A$i = A$i" >> A$i.hs                   
done

This generates a 1000 modules each with one datatype. Compiling them with:

inplace/bin/ghc-stage2 A*.hs -optP-D__F{1..10000}__

results in fingerprintDynFlags being the top cost centre in the profile. AFAICT there's only one module dependent piece that goes into computing fingerprintDynFlags and the rest is the same between those 1000 modules.

Now, why would I have so many preprocessor flags? This is how the Buck build system currently works. If a Haskell library depends on a C++ library then the GHC invocation gets the C++ library's directory as include path (-optP -I -optP some/library/path). This can grow quite big.

Change History (5)

comment:1 Changed 20 months ago by niteria

Cc: simonmar added
Description: modified (diff)

comment:2 Changed 19 months ago by niteria

Differential Rev(s): phab:D4445
Status: newpatch

comment:3 Changed 19 months ago by Bartosz Nitka <niteria@…>

In b8f03bbe/ghc:

Cache the fingerprint of sOpt_P

Before this change we would compute a hash of
all the command line -optP flags once per file.
With a lot of files and many -optP flags, that's a lot
of repeated work.

I added a new Note that explains the approach and rationale.

Test Plan: new test

Reviewers: simonmar, simonpj, bgamari

Reviewed By: simonpj

Subscribers: rwbarton, thomie, carter

GHC Trac Issues: #14697

Differential Revision: https://phabricator.haskell.org/D4445

comment:4 Changed 19 months ago by niteria

Resolution: fixed
Status: patchclosed

comment:5 Changed 16 months ago by Ben Gamari <ben@…>

In af986f9d/ghc:

testsuite: Disable T14697 on Windows

Test Plan: Validate on Windows

Subscribers: thomie, carter

GHC Trac Issues: #14697, #15072

Differential Revision: https://phabricator.haskell.org/D4619
Note: See TracTickets for help on using tickets.