Opened 2 years ago

Closed 12 months ago

#14298 closed feature request (fixed)

Let Template Haskell dynamically add something with which to link

Reported by: harpocrates Owned by:
Priority: normal Milestone: 8.8.1
Component: Template Haskell Version: 8.2.1
Keywords: Cc: tmcdonell
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s): Phab:D4064
Wiki Page:

Description (last modified by harpocrates)

As of today, Template Haskell supports emitting foreign files (for C compiler languages) via addForeignFile. In doing so, GHC takes on the work of compiling these, linking them, and then cleaning up any other files.

This makes packages like inline-c (https://hackage.haskell.org/package/inline-c) possible, where you can write C snippets in quasiquotes and these can interact with your Haskell code. The user doesn't need to pass extra options to GHC (except to enable the right language extensions), and they don't have to see any of the intermediate generated artifacts.

Unfortunately, that breaks down for non C compiler languages. It would be nice for TH to also support directly adding something to pass to the linker, since then one could

  • use TH's runIO to generate libraries or object files by calling out to whatever other compilers
  • add those via TH
  • have GHC statically link against the content (from the second bullet point)

I'm not sure what the API for this could be, but maybe

  1. add a LangLinkable constructor to the ForeignSrcLang data type
  2. add a qAddForeignFilePath :: ForeignSrcLang -> FilePath -> m () method to Quasi m

Change History (17)

comment:1 Changed 2 years ago by harpocrates

Description: modified (diff)
Summary: Let Template Haskell dynamically add a library against which to linkLet Template Haskell dynamically add something with which to link

comment:2 Changed 2 years ago by harpocrates

I have a functional first-pass implementation of this complete.

Is this feature something GHC would eventually consider merging in? If so, what is the process for doing that, and where would discussion for settling on the right TH API take place?

comment:3 Changed 2 years ago by goldfire

TH is a bit of a gray area. Non-TH language features go through the ghc-proposals process, but TH features tend not to. I would recommend posting to ghc-devs. That will reach the people that care deeply about TH and would be able to evaluate this idea.

(I care deeply about TH, but I have a very weak understanding of how the FFI works, and so I don't have an informed opinion here.)

comment:4 Changed 2 years ago by bgamari

I'm a bit skeptical of this; what should happen when GHC is invoked in single-shot mode (e.g. -c)? In this case we don't perform any linking but rather simply emit an object file. It seems like in generate this feature would need support from external tools (e.g. cabal) to work properly.

Last edited 2 years ago by bgamari (previous) (diff)

comment:5 Changed 2 years ago by harpocrates

Let me reply to those two points in reverse order. :)

It seems like in generate this feature would need support from external tools (e.g. cabal) to work properly.

I'm not sure that is the case. TH already has support for adding in C source files, compiling them, and linking them. This feature request is requesting a way to hook into just the linking part of the pipeline that was built in https://phabricator.haskell.org/D3280 (instead of also running a C compiler).

I don't think cabal factors into the existing qAddForeignFile, so I don't see why it should factor into my proposed qAddForeignFilePath.

I'm a bit skeptical of this; what should happen when GHC is invoked in single-shot mode (e.g. -c)?

I'd expect to see the same thing that would happen if GHC where invoked in single-shot mode on something involving addForeignFile, that is to say: cannot find object file './Support.dyn_o' where Support is the module containing the TH.

That said, my understanding of -c is that it shouldn't work on any module that uses TH from another module.

comment:6 Changed 2 years ago by hsyl20

About the API, I would prefer:

addForeignObject :: ByteString -> Q ()

in order to directly produce objects from TH without having to generate temporary object files.

Then it should be enough to use addDependentFile and Data.ByteString.getContents to write the addForeignObjectFile :: FilePath -> Q () helper function.

(People from Tweag I/O were also interested in this: https://www.reddit.com/r/haskell/comments/6p1aqo/building_inlinec_projects_just_got_a_lot_easier/dkn2w8w/)

comment:7 Changed 2 years ago by bgamari

That said, my understanding of -c is that it shouldn't work on any module that uses TH from another module.

That isn't quite true. TH will work perfectly well in single-shot mode. However, as usual, the user must guarantee that any needed modules are compiled for whatever way GHC will want to load to evaluate splices.

For instance,

$ pwd
/opt/exp/ghc/ghc-landing/testsuite/tests/th
$ ghc -c TH_NestedSplices_Lib.hs  -dynamic-too -fforce-recomp
$ ghc -c TH_NestedSplices.hs -fforce-recomp

It's important that this behavior be preserved since Cabal is slowly moving towards using single-shot mode to improve module-level build parallelism.

About the API, I would prefer:

addForeignObject :: ByteString -> Q ()

The interface in comment:6 sounds plausible.

The worry that I have with just allowing the user to specify an object file path as in the original proposal is that it makes it seem as though they can safely addForeignObject the same object file from multiple modules. However, this is not safe as they will encounter conflicting symbol errors when they attempt to link the final object. Of course, we could in principle document our way out of this issue.

comment:8 Changed 2 years ago by harpocrates

I really appreciate all of the detailed explanations! Thanks!

addForeignObject :: ByteString -> Q ()

Seems like this would solve most of the problems brought up, and I prefer it too. It's worth noting that bytestring isn't currently a dependency of template-haskell though.

comment:9 in reply to:  8 Changed 2 years ago by hsyl20

It's worth noting that bytestring isn't currently a dependency of template-haskell though.

Then maybe we should avoid bytestring with something like:

addForeignObjectFromMem :: Word -> Ptr () -> Q ()

comment:10 Changed 2 years ago by harpocrates

Differential Rev(s): Phab:D4064

Here's a patch which implements the API from comment:6.

comment:11 Changed 2 years ago by tmcdonell

Cc: tmcdonell added

comment:12 Changed 21 months ago by Ben Gamari <ben@…>

In ceb91477/ghc:

Support adding objects from TH

The user facing TH interface changes are:

  * 'addForeignFile' is renamed to 'addForeignSource'
  * 'qAddForeignFile'/'addForeignFile' now expect 'FilePath's
  * 'RawObject' is now a constructor for 'ForeignSrcLang'
  * 'qAddTempFile'/'addTempFile' let you request a temporary file
    from the compiler.

Test Plan: unsure about this, added a TH test

Reviewers: goldfire, bgamari, angerman

Reviewed By: bgamari, angerman

Subscribers: hsyl20, mboes, carter, simonmar, bitonic, ljli, rwbarton, thomie

GHC Trac Issues: #14298

Differential Revision: https://phabricator.haskell.org/D4217

comment:13 Changed 21 months ago by bgamari

Milestone: 8.6.1

comment:14 Changed 18 months ago by bgamari

Milestone: 8.6.18.8.1

These won't be addressed by GHC 8.6.

comment:15 Changed 18 months ago by harpocrates

Isn't this fixed in ceb914771aece0aa6d87339227ce406c7179d1d1 (which is in GHC 8.6)? I think the ticket can be closed now.

comment:16 Changed 17 months ago by bgamari

Indeed it is. Thanks for noticing!

comment:17 Changed 12 months ago by harpocrates

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.