Opened 2 years ago

Last modified 7 months ago

#14030 new task

Implement the "Derive Lift instances for data types in template-haskell" proposal

Reported by: RyanGlScott Owned by: RyanGlScott
Priority: normal Milestone: 8.10.1
Component: Template Haskell Version: 8.3
Keywords: deriving Cc: harpocrates, int-index
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

Back in September 2015, I proposed using the DeriveLift extension to, well, derive Lift instance for data types in the template-haskell library. The proposal was well received, but I was unable to implement it at the time due to DeriveLift's newness (having only been introduced in GHC 8.0). Now that GHC 8.0 is the oldest version of GHC that we support bootstrapping with, this is no longer an obstacle.

Change History (12)

comment:1 Changed 2 years ago by RyanGlScott

Owner: set to RyanGlScott

comment:2 Changed 2 years ago by RyanGlScott

Keywords: deriving added

comment:3 Changed 20 months ago by bgamari

Milestone: 8.4.18.6.1

This ticket won't be resolved in 8.4; remilestoning for 8.6. Do holler if you are affected by this or would otherwise like to work on it.

comment:4 Changed 15 months ago by bgamari

Milestone: 8.6.18.8.1

These won't be addressed by GHC 8.6.

comment:5 Changed 10 months ago by harpocrates

Cc: harpocrates added

comment:6 Changed 10 months ago by RyanGlScott

At one point in time (a release of GHC or two ago), I attempted to implement this, but ran into interface file-related issues. Unfortunately, I no longer have the exact errors in front of me, but I recall that there was some difficulty due to the fact that:

  1. We were deriving Lift instances in Language.Haskell.TH.Syntax
  2. The derived Lift code references functions from Language.Haskell.TH.Lib.Internal
  3. Because Language.Haskell.TH.Lib.Internal imports Language.Haskell.TH.Syntax, there were (seemingly) some issues with import cycles

That being said, I just tried implementing this afresh with GHC 8.6.2 as my bootstrapping compiler, and I did not hit any issues at all! This is quite strange, and I'm somewhat paranoid that there is an issue lurking underneath the surface that I just haven't exposed yet.

While typing this out, I realized one thing. I think the last time I tried this, I might have been using GHC 8.2.2 as my bootstrapping compiler. There is a notable difference between 8.2.2 and future major releases: 8.2.2 does not have Language.Haskell.TH.Lib.Internal (it wouldn't be until 8.4 that that would come into existence). I think GHC might have been confused because the DeriveLift-generated code that the bootstrapping compiler (8.2.2) generated mentioned things from Language.Haskell.TH.Lib, but the stage-1 compiler generated DeriveLift code that mentioned Language.Haskell.TH.Lib.Internal.

If that's the case, then that would explain why I can't reproduce the issue now (with 8.6.2). Still, this has me a bit worried that what we're doing here is a bit fragile, and that perhaps moving functions from Language.Haskell.TH.Lib.Internal to some other, hypothetical module in the future might cause this issue to surface once more...

comment:7 Changed 10 months ago by RyanGlScott

OK, I decided that I wanted to remember exactly what the error message was, so I compiled GHC 8.4.4 using 8.2.2 as the bootstrapping compiler. Here is the error that I got:

$ PATH=/opt/ghc/8.2.2/bin:$PATH make -j1
===--- building phase 0
make --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for 'phase_0_builds'.
===--- building phase 1
make --no-print-directory -f ghc.mk phase=1 phase_1_builds
"/opt/ghc/8.2.2/bin/ghc" -hisuf hi -osuf  o -hcsuf hc -static  -O0 -H64m -Wall   -package-db libraries/bootstrapping.conf  -this-unit-id template-haskell-2.13.0.0 -hide-all-packages -i -ilibraries/template-haskell/. -ilibraries/template-haskell/dist-boot/build -Ilibraries/template-haskell/dist-boot/build -ilibraries/template-haskell/dist-boot/build/./autogen -Ilibraries/template-haskell/dist-boot/build/./autogen -Ilibraries/template-haskell/.    -optP-include -optPlibraries/template-haskell/dist-boot/build/./autogen/cabal_macros.h -package-id base-4.10.1.0 -package-id ghc-boot-th-8.4.4 -package-id pretty-1.1.3.3 -Wall -this-unit-id template-haskell -XHaskell2010   -no-user-package-db -rtsopts  -fno-warn-deprecated-flags      -odir libraries/template-haskell/dist-boot/build -hidir libraries/template-haskell/dist-boot/build -stubdir libraries/template-haskell/dist-boot/build    -c libraries/template-haskell/./Language/Haskell/TH/Syntax.hs -o libraries/template-haskell/dist-boot/build/Language/Haskell/TH/Syntax.o 

libraries/template-haskell/Language/Haskell/TH/Syntax.hs:2002:43: error:
    • Failed to load interface for ‘Language.Haskell.TH.Lib’
      Use -v to see a list of the files searched for.
    • In the expression:
        Language.Haskell.TH.Lib.conE
          (mkNameG_d
             "template-haskell" "Language.Haskell.TH.Syntax" "NominalR")
      In an equation for ‘lift’:
          lift NominalR
            = Language.Haskell.TH.Lib.conE
                (mkNameG_d
                   "template-haskell" "Language.Haskell.TH.Syntax" "NominalR")
      When typechecking the code for ‘lift’
        in a derived instance for ‘Lift Role’:
        To see the code I am typechecking, use -ddump-deriv
      In the instance declaration for ‘Lift Role’
     |
2002 |   deriving( Show, Eq, Ord, Data, Generic, Lift )
     |                                           ^^^^
libraries/template-haskell/ghc.mk:3: recipe for target 'libraries/template-haskell/dist-boot/build/Language/Haskell/TH/Syntax.o' failed
make[1]: *** [libraries/template-haskell/dist-boot/build/Language/Haskell/TH/Syntax.o] Error 1
Makefile:122: recipe for target 'all' failed
make: *** [all] Error 2

If you pass -v, you'll see what the culprit is:

$ "/opt/ghc/8.2.2/bin/ghc" -hisuf hi -osuf  o -hcsuf hc -static  -O0 -H64m -Wall   -package-db libraries/bootstrapping.conf  -this-unit-id template-haskell-2.13.0.0 -hide-all-packages -i -ilibraries/template-haskell/. -ilibraries/template-haskell/dist-boot/build -Ilibraries/template-haskell/dist-boot/build -ilibraries/template-haskell/dist-boot/build/./autogen -Ilibraries/template-haskell/dist-boot/build/./autogen -Ilibraries/template-haskell/.    -optP-include -optPlibraries/template-haskell/dist-boot/build/./autogen/cabal_macros.h -package-id base-4.10.1.0 -package-id ghc-boot-th-8.4.4 -package-id pretty-1.1.3.3 -Wall -this-unit-id template-haskell -XHaskell2010   -no-user-package-db -rtsopts  -fno-warn-deprecated-flags      -odir libraries/template-haskell/dist-boot/build -hidir libraries/template-haskell/dist-boot/build -stubdir libraries/template-haskell/dist-boot/build    -c libraries/template-haskell/./Language/Haskell/TH/Syntax.hs -o libraries/template-haskell/dist-boot/build/Language/Haskell/TH/Syntax.o -v
Glasgow Haskell Compiler, Version 8.2.2, stage 2 booted by GHC version 8.0.2
Using binary package database: /opt/ghc/8.2.2/lib/ghc-8.2.2/package.conf.d/package.cache
Using binary package database: libraries/bootstrapping.conf/package.cache
package flags [-package-id base-4.10.1.0{unit base-4.10.1.0 True ([])},
               -package-id ghc-boot-th-8.4.4{unit ghc-boot-th-8.4.4 True ([])},
               -package-id pretty-1.1.3.3{unit pretty-1.1.3.3 True ([])}]
loading package database /opt/ghc/8.2.2/lib/ghc-8.2.2/package.conf.d
loading package database libraries/bootstrapping.conf
package binary-0.8.5.1 overrides a previously defined package
package hpc-0.6.0.3 overrides a previously defined package
package Cabal-2.0.1.0 is unusable due to shadowed dependencies:
  binary-0.8.5.1
package ghc-8.2.2 is unusable due to shadowed dependencies:
  binary-0.8.5.1 hpc-0.6.0.3 ghc-boot-8.2.2 ghci-8.2.2
package ghc-boot-8.2.2 is unusable due to shadowed dependencies:
  binary-0.8.5.1
package ghci-8.2.2 is unusable due to shadowed dependencies:
  binary-0.8.5.1 ghc-boot-8.2.2
wired-in package ghc-prim mapped to ghc-prim-0.5.1.1
wired-in package integer-gmp mapped to integer-gmp-1.0.1.0
wired-in package base mapped to base-4.10.1.0
wired-in package rts mapped to rts
wired-in package template-haskell mapped to template-haskell-2.13.0.0
wired-in package ghc mapped to ghc-8.4.4
wired-in package dph-seq not found.
wired-in package dph-par not found.
*** Checking old interface for Language.Haskell.TH.Syntax (use -ddump-hi-diffs for more details):
*** Parser [Language.Haskell.TH.Syntax]:
!!! Parser [Language.Haskell.TH.Syntax]: finished in 25.95 milliseconds, allocated 36.318 megabytes
*** Renamer/typechecker [Language.Haskell.TH.Syntax]:
!!! Renamer/typechecker [Language.Haskell.TH.Syntax]: finished in 775.89 milliseconds, allocated 669.498 megabytes

libraries/template-haskell/Language/Haskell/TH/Syntax.hs:2002:43: error:
    • Failed to load interface for ‘Language.Haskell.TH.Lib’
      Locations searched:
        libraries/template-haskell/./Language/Haskell/TH/Lib.hi
        libraries/template-haskell/./Language/Haskell/TH/Lib.hi-boot
        libraries/template-haskell/dist-boot/build/Language/Haskell/TH/Lib.hi
        libraries/template-haskell/dist-boot/build/Language/Haskell/TH/Lib.hi-boot
        libraries/template-haskell/dist-boot/build/./autogen/Language/Haskell/TH/Lib.hi
        libraries/template-haskell/dist-boot/build/./autogen/Language/Haskell/TH/Lib.hi-boot
    • In the expression:
        Language.Haskell.TH.Lib.conE
          (mkNameG_d
             "template-haskell" "Language.Haskell.TH.Syntax" "NominalR")
      In an equation for ‘lift’:
          lift NominalR
            = Language.Haskell.TH.Lib.conE
                (mkNameG_d
                   "template-haskell" "Language.Haskell.TH.Syntax" "NominalR")
      When typechecking the code for ‘lift’
        in a derived instance for ‘Lift Role’:
        To see the code I am typechecking, use -ddump-deriv
      In the instance declaration for ‘Lift Role’
     |
2002 |   deriving( Show, Eq, Ord, Data, Generic, Lift )
     |                                           ^^^^
*** Deleting temp files:
Deleting: 
*** Deleting temp dirs:
Deleting:

It appears as though GHC is searching in through the interface files of the stage-1 GHC (8.4.4), but using the bootstrapping compiler's (8.2.2's) DeriveLift name information to determine where to search! This adds more evidence to my belief that this is quite fragile. I'm not sure how to do better, however.

comment:8 Changed 10 months ago by harpocrates

Thanks for figuring out what the issue was in 8.2.2! Isn't this actually great news? Once https://phabricator.haskell.org/D5220 makes it through, the code that Lift will generate won't involve conE, mkNameG_d, etc. - it'll just be generating syntax for a bracket!

I'm gonna try this out just to make sure I'm not missing anything. Another sanity check will be to see where we still use THNames.

What do you think?

comment:9 in reply to:  8 Changed 10 months ago by RyanGlScott

Replying to harpocrates:

the code that Lift will generate won't involve conE, mkNameG_d, etc. - it'll just be generating syntax for a bracket!

I'm not so sure. Nominally, yes, the generated code will only use the bracket syntax. But one must ask: what does the bracket syntax desugar down to? The answer may surprise you:

λ> :set -ddump-simpl
λ> data MyBool = F | T
λ> deriving instance Lift MyBool

==================== Tidy Core ====================
Result size of Tidy Core
  = {terms: 38, types: 12, coercions: 3, joins: 0/0}

-- RHS size: {terms: 21, types: 2, coercions: 0, joins: 0/0}
$clift_r4ul :: MyBool -> Q Exp
[GblId, Arity=1, Unf=OtherCon []]
$clift_r4ul
  = \ (ds_d4um :: MyBool) ->
      case ds_d4um of {
        F ->
          Language.Haskell.TH.Lib.Internal.conE
            (mkNameG_d
               (GHC.CString.unpackCString# "interactive"#)
               (GHC.CString.unpackCString# "Ghci4"#)
               (GHC.CString.unpackCString# "F"#));
        T ->
          Language.Haskell.TH.Lib.Internal.conE
            (mkNameG_d
               (GHC.CString.unpackCString# "interactive"#)
               (GHC.CString.unpackCString# "Ghci4"#)
               (GHC.CString.unpackCString# "T"#))
      }

-- RHS size: {terms: 1, types: 0, coercions: 3, joins: 0/0}
interactive:Ghci5.$fLiftMyBool [InlPrag=INLINE (sat-args=0)]
  :: Lift MyBool
[GblId[DFunId(nt)], Arity=1, Unf=OtherCon []]
interactive:Ghci5.$fLiftMyBool
  = $clift_r4ul
    `cast` (Sym (Language.Haskell.TH.Syntax.N:Lift[0] <MyBool>_N)
            :: (MyBool -> Q Exp) ~R# Lift MyBool)

It turns out that bracket syntax desugars down to code that, in fact, uses conE and mkNameG_d! It's a bit more indirect, but I worry that there's the same potential for fragility.

comment:10 Changed 9 months ago by osa1

Milestone: 8.8.18.10.1

Bumping milestones of low-priority tickets.

comment:11 Changed 7 months ago by int-index

Cc: int-index added

comment:12 Changed 7 months ago by RyanGlScott

To confirm my hunch that even generating code with bracket syntax would still be fragile, I tried compiling GHC 8.4.4 with 8.2.2 as the bootstrapping compiler again, but instead of attempting to derive a Lift instance for Role, I defined one manually using bracket syntax:

instance Lift Role where
  lift NominalR = [| NominalR |]
  ...

Alas, that also suffers from the same issues observed in comment:7:

libraries/template-haskell/Language/Haskell/TH/Syntax.hs:2005:28: error:
    • Failed to load interface for ‘Language.Haskell.TH.Lib’
      Use -v to see a list of the files searched for.
    • In the expression: [| NominalR |]
      In an equation for ‘lift’: lift NominalR = [| NominalR |]
      In the instance declaration for ‘Lift Role’
     |
2005 |   lift NominalR          = [| NominalR |]
     |                            ^^^^^^^^^^^^^^

This all seems horribly delicate. There has to be a way to ensure that subsequent changes to template-haskell won't break the build in similar ways, but I can't think of how to orchestrate this.

Note: See TracTickets for help on using tickets.