Opened 4 years ago

Closed 4 years ago

#11725 closed bug (fixed)

Performance Regression from 7.8.3 to 7.10.3

Reported by: dominic Owned by: dominic
Priority: normal Milestone: 8.0.1
Component: Compiler Version: 7.10.3
Keywords: Cc:
Operating System: Linux Architecture: Unknown/Multiple
Type of failure: Runtime performance bug Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

Some time ago I wrote a little test program for random number generation. Under 7.8.3 I get

Total   time    9.03s  (  9.15s elapsed)

Under 7.10.3 I get

Total   time   24.773s  ( 25.288s elapsed)

Now of course it could be the libraries that I am using rather than GHC itself so I have tried to make them as similar as possible. For 7.8.3 I have

  build-depends:      base ==4.7.0.1,
                      mtl ==2.1.3.1,
                      primitive == 0.6,
                      mwc-random == 0.13.3.2,
                      vector == 0.10.12.3,
                      random ==1.1,
                      random-fu == 0.2.6.2,
                      random-source == 0.3.0.6

For 7.10.3 I have

  build-depends:      base ==4.8.2.0,
                      mtl ==2.2,
                      primitive == 0.6,
                      mwc-random == 0.13.3.2,
                      vector == 0.10.12.3,
                      random ==1.1,
                      random-fu == 0.2.6.2,
                      random-source == 0.3.0.6

So the only differences are in mtl and base. I don’t seem to be able to coax cabal into using mtl-2.2 for 7.8.3 with all the other required libraries.

dominic@ghcPerformance:~$ cabal install 'random-source ==0.3.0.6' 'random-fu ==0.2.6.2' 'random ==1.1' 'primitive ==0.6' 'mwc-random ==0.13.3.2' 'mtl ==2.2' 'vector ==0.10.12.3' --with-ghc=ghc-7.10.3
Resolving dependencies...
All the requested packages are already installed:
mtl-2.2
mwc-random-0.13.3.2
primitive-0.6
random-1.1
random-fu-0.2.6.2
random-source-0.3.0.6
vector-0.10.12.3
Use --reinstall if you want to reinstall anyway.
dominic@ghcPerformance:~$ cabal install 'random-source ==0.3.0.6' 'random-fu ==0.2.6.2' 'random ==1.1' 'primitive ==0.6' 'mwc-random ==0.13.3.2' 'mtl ==2.2' 'vector ==0.10.12.3' --with-ghc=ghc-7.8.3
Resolving dependencies...
cabal: Could not resolve dependencies:
trying: random-source-0.3.0.6/installed-70e... (user goal)
next goal: mtl (user goal)
rejecting: mtl-2.2.1, 2.2.0.1 (global constraint requires ==2.2)
rejecting: mtl-2.2/installed-cc5..., 2.2 (conflict: random-source =>
mtl==2.1.3.1/installed-8bc...)
rejecting: mtl-2.1.3.1/installed-8bc..., 2.1.3.1, 2.1.2, 2.1.1, 2.1, 2.0.1.1,
2.0.1.0, 2.0.0.0, 1.1.1.1, 1.1.1.0, 1.1.0.2, 1.1.0.1, 1.1.0.0, 1.0 (global
constraint requires ==2.2)
Backjump limit reached (change with --max-backjumps).

Cabal seems to be telling me that random-source-0.3.0.6 is the problem but if I look at the constraints for that package here https://hackage.haskell.org/package/random-source then I see

 mtl (>=1 && <3)

I am not sure how to proceed from here. I’d like to solve this myself but I don’t want to start building versions of ghc to see which change caused the regression without first eliminating mtl.

Any ideas would be gratefully received.

{-# LANGUAGE TemplateHaskell   #-}
{-# LANGUAGE GADTs             #-}
{-# LANGUAGE FlexibleInstances #-}

import Data.Random
import Data.Random.Source
import qualified System.Random.MWC as MWC
import Control.Monad.Reader
import Control.Monad.Primitive

$(monadRandom [d|
  instance (PrimMonad m, s ~ PrimState m) => MonadRandom (ReaderT (MWC.Gen s) m) where
    getRandomWord16 = ask >>= lift . MWC.uniform
    getRandomWord32 = ask >>= lift . MWC.uniform
    getRandomWord64 = ask >>= lift . MWC.uniform
  |])

testUniform :: MonadRandom m => Int -> m [Double]
testUniform n = replicateM (fromIntegral n) (sample stdUniform)

n :: Int
n = 10^7

main :: IO ()
main = do
    seed <- MWC.create
    xs <- runReaderT (testUniform n) seed
    print (sum xs / fromIntegral n)

This cabal file will build this on 7.8.3

name:                PerfTest8
version:             0.1.0.0
homepage:            TBD
license:             MIT
author:              Dominic Steinitz
maintainer:          idontgetoutmuch@gmail.com
category:            System
build-type:          Simple
cabal-version:       >=1.10

executable Random8
  main-is:            TestMwcViaRandomSource.hs
  build-depends:      base ==4.7.0.1,
                      mtl ==2.1.3.1,
                      primitive == 0.6,
                      mwc-random == 0.13.3.2,
                      vector == 0.10.12.3,
                      random ==1.1,
                      random-fu == 0.2.6.2,
                      random-source == 0.3.0.6
  default-language:   Haskell2010

This cabal file will build this on 7.10.3

name:                PerfTest10
version:             0.1.0.0
homepage:            TBD
license:             MIT
author:              Dominic Steinitz
maintainer:          idontgetoutmuch@gmail.com
category:            System
build-type:          Simple
cabal-version:       >=1.10

executable Random10
  main-is:            TestMwcViaRandomSource.hs
  build-depends:      base ==4.8.2.0,
                      mtl ==2.2,
                      primitive == 0.6,
                      mwc-random == 0.13.3.2,
                      vector == 0.10.12.3,
                      random ==1.1,
                      random-fu == 0.2.6.2,
                      random-source == 0.3.0.6
  default-language:   Haskell2010

Change History (11)

comment:1 Changed 4 years ago by dominic

It's entirely possible that this has been solved in 8.0.1 so I have built this compiler on OS X 10.11.3. At the moment I can't build one of the packages on which my example depends

~/Dropbox/Private/Random8 $ ~/Library/Haskell/ghc-7.10.3/lib/cabal-install-1.25.0.0/bin/cabal install flexible-defaults --constraint="template-haskell==2.11.0.0" --with-ghc=/Users/dom/ghc/inplace/bin/ghc-stage2
~/Library/Haskell/ghc-7.10.3/lib/cabal-install-1.25.0.0/bin/cabal install flexible-defaults --constraint="template-haskell==2.11.0.0" --with-ghc=/Users/dom/ghc/inplace/bin/ghc-stage2
Resolving dependencies...
cabal: Entering directory '/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T/cabal-tmp-99610/flexible-defaults-0.0.0.2'
Configuring flexible-defaults-0.0.0.2...
Building flexible-defaults-0.0.0.2...
Preprocessing library flexible-defaults-0.0.0.2...
[1 of 3] Compiling Language.Haskell.TH.FlexibleDefaults.Solve ( src/Language/Haskell/TH/FlexibleDefaults/Solve.hs, dist/dist-sandbox-59d6d2c7/build/Language/Haskell/TH/FlexibleDefaults/Solve.o )

src/Language/Haskell/TH/FlexibleDefaults/Solve.hs:13:1: warning:
    The import of ‘Data.Monoid’ is redundant
      except perhaps to import instances from ‘Data.Monoid’
    To import instances alone, use: import Data.Monoid()
[2 of 3] Compiling Language.Haskell.TH.FlexibleDefaults.DSL ( src/Language/Haskell/TH/FlexibleDefaults/DSL.hs, dist/dist-sandbox-59d6d2c7/build/Language/Haskell/TH/FlexibleDefaults/DSL.o )

src/Language/Haskell/TH/FlexibleDefaults/DSL.hs:79:82: error:
    Not in scope: type constructor or class ‘InlineSpec’

src/Language/Haskell/TH/FlexibleDefaults/DSL.hs:90:32: error:
    Not in scope: type constructor or class ‘InlineSpec’

src/Language/Haskell/TH/FlexibleDefaults/DSL.hs:121:14: error:
    Not in scope: type constructor or class ‘InlineSpec’
cabal: Leaving directory '/var/folders/3p/m593dprn5snbjz6sc45c77f80000gn/T/cabal-tmp-99610/flexible-defaults-0.0.0.2'
Failed to install flexible-defaults-0.0.0.2
cabal: Error: some packages failed to install:
flexible-defaults-0.0.0.2 failed during the building phase. The exception was:
ExitFailure 1

Here's the offending package: https://github.com/mokus0/flexible-defaults and here is the bit of code which seems to cause the problem but which afaict should *not* be generated given the CPP directive.

applyInline :: String -> Maybe Inline -> Q [Dec] -> Q [Dec]
#if MIN_VERSION_template_haskell(2,8,0)
applyInline n (Just inl) = fmap (PragmaD (InlineP (mkName n) inl FunLike AllPhases) :)
#elif MIN_VERSION_template_haskell(2,4,0)
applyInline n (Just inl)
    | inl /= Inlinable  = fmap (PragmaD (InlineP (mkName n) (InlineSpec (inl == Inline) False Nothing)) :)
#endif
applyInline _ _ = id
Last edited 4 years ago by dominic (previous) (diff)

comment:2 Changed 4 years ago by dominic

Owner: set to dominic

comment:4 Changed 4 years ago by simonpj

A 3x performance regression looks bad. If you can pin it down more precisely it would be a big help. Thanks!

comment:5 Changed 4 years ago by dominic

Here's my proposed approach to pinning it down.

  1. Build ghc-8.0.1-rc2 and run my program under it. If the regression is gone then I propose to do no further investigation.
  2. git bisect the sources from 7.8.3 to 7.10.3 until I find the change that causes the regression.

Does that sound reasonable? Is there any guidance on how to do (2) efficiently and effectively?

At the moment I cannot get my sources to compile under ghc-8.0.1-rc2 so my first mini-step is to solve this.

I should point out that this is somewhat tangential to my day job although we use RNGs extremely heavily in our application.

comment:6 Changed 4 years ago by dominic

Good news! After a heroic battle fighting cabal, template-haskell, unnecessarily restrictive upper bounds and the ghc build system on OS X 10.11.3, in which the aforementioned almost won, I managed to get my test program to compile under ghc-8.0.1-rc2.

Under ghc-7.10.2 (NB not comparable with the above as that was done on a different machine) I get

Total   time   12.764s  ( 13.242s elapsed)

Under ghc-8.0.1-rc2 I get

Total   time    2.896s  (  3.290s elapsed)

So better even than 7.8.3.

For info I ran my comparison of 7.8.3 against 7.10.3 on

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
stepping        : 2
microcode       : 0xffffffff
cpu MHz         : 2397.169
cache size      : 30720 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ss\
se3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm fsgsbase bmi1 avx2 smep bmi2 erms xsaveopt
bugs            :
bogomips        : 4794.33
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

and I ran my comparison of 7.10.3 againt 8.0.1-rc2 on

Intel(R) Core(TM) i7-3720QM CPU @ 2.60GHz
Last edited 4 years ago by dominic (previous) (diff)

comment:7 Changed 4 years ago by dominic

There seems little point in keeping this ticket open but how do we stop this sort of performance regression happening in the future?

comment:8 Changed 4 years ago by bgamari

It's hard saying without knowing more precisely what the performance regression was due to.

comment:9 Changed 4 years ago by dominic

I don't propose to do any archeology myself so feel free to close this. I refer you to https://mail.haskell.org/pipermail/haskell-cafe/2016-January/122660.html though. I may start a suite of my own performance tests as I would prefer not to be caught out again.

comment:10 in reply to:  9 Changed 4 years ago by bgamari

Replying to dominic:

I don't propose to do any archeology myself so feel free to close this. I refer you to https://mail.haskell.org/pipermail/haskell-cafe/2016-January/122660.html though. I may start a suite of my own performance tests as I would prefer not to be caught out again.

If you can come up with any tests that have minimal build dependencies (ideally none although boot libraries are acceptable) please contribute them to nofib. We are in bad need of better coverage.

comment:11 Changed 4 years ago by bgamari

Milestone: 8.0.1
Resolution: fixed
Status: newclosed

This seems to be one of many performance regressions in 7.10 which has mysteriously improved with 8.0. There have been a variety of performance improvements made in 8.0; it would be great to know which of these helped here. You, the reader, can help here by identifying which commit(s) are responsible for this improvement.

Note: See TracTickets for help on using tickets.