Continuous Integration

This page is to support the discussion of GHC DevOps Group on the CI solution for GHC to provide continuous testing and release artefact generation. See also #13716.

If you are looking for information about how to maintain our new CircleCI and Appveyor infrastructure see ContinuousIntegration/Usage.




  • Build GHC for Tier 1 platforms (Windows, Linux & macOS), run ./validate, and produce release artefacts (distributions and documentation).
  • Build PRs (differentials) and run ./validate on Linux/x86-64.
  • Security: PRs builds run arbitrary user code; this must not be able to compromise other builds and especially not release artefacts.
  • Infrastructure reproducibility (infrastructure can be spun up and configured automatically)
  • Infrastructure forkability (devs forking the GHC repo can run their own CI without additional work)
  • Low maintenance overhead
  • Low set up costs


  • Build PRs (differentials) and run ./validate on non-Linux/x86-64 Tier 1 platforms.
  • Build GHC for non-Tier 1 platforms & run ./validate

Possible solutions



  • We can run build nodes on any architecture and OS we choose to set up.


  • Security is low on PR builds unless we spend further effort to sandbox builds properly. Moreover, even with sandboxing, Jenkins security record is troublesome.
  • Jenkins is well known to be time consuming to set up.
  • Additional time spent setting up servers.
  • Additional time spent maintaining servers.
  • It is unclear how easy it is to make the set up reproducible.
  • The set up is not forkable (a forker would need to set up their own servers).

CircleCI & AppVeyor


  • Easy to set up
  • Excellent security
  • Low maintenance cost
  • Infrastructure reproducible
  • Infrastructure forkable
  • Easy to integrate with GitHub


  • Direct support only for Linux, Windows, macOS, iOS & Android, everything else needs to rely on cross-compiling and QEMU, or building on remote drones.
  • We are limited to CircleCI's & Appveyor's current feature set and have to rely on them for feature development.
  • We need to deal with two different CI providers.

Discussion summary

After a detailed discussion on the GHC DevOps Group mailing list, the group reached the conclusion that all factors considered, CircleCI & AppVeyor provides the best trade offs for the following reasons.


  • Both solutions incur hardware or subscription costs. In the case of Jenkins, we need servers (and we just learnt that RackSpace terminated its OSS program at the end of the year) and, in the case of CircleCI & AppVeyor, we will likely need more resource than they provide for free to open source projects. However, Google X and Tweag I/O have indicated that they would be happy to help cover such costs.
  • Both solutions incur some developer costs. However, by all accounts and our own experience, this is substantially lower for the CircleCI & AppVeyor solution. In particular, it completely eliminates server maintenance costs. Integration with GitHub is straight forward and Phabricator also seems to have some support.

Overall, CircleCI & AppVeyor are more cost-effective.


  • Jenkins is fully customisable and provides complete flexibility wrt to the architectures and operations systems used in build machines.
  • With CircleCI & AppVeyor, we are dependent on the features provided by the hosting services and limited in the architectures and OSes that are directly supported. In particular, we will have to fall back to cross-compiling and using QEMU for less commonly used platforms. Our base line, the Tier 1 platforms, as well as Android and iOS on ARM are supported by CircleCI & AppVeyor.

Overall, Jenkins is more flexible.


  • Security is crucial as we are planning to run untrusted and unreviewed code in PRs/differentials in the CI environment.
  • To provide adequate security, we need to add sandboxing to Jenkins (which is possible, but more work). However, even with that, the security track record of Jenkins is not very good.
  • In the case of CircleCI & AppVeyor, security is taken care of by the provider.

Overall, CircleCI & AppVeyor are more secure.


  • With forkability, we refer to a user's ability to fork GHC, modify it, and run their own CI instance (not using GHC HQ's infrastructure) without any significant extra work. This significantly improves scaling and takes load of the GHC HQ's infrastructure. CircleCI & AppVeyor is forkable, but Jenkins is not (as a user needs to set up their own infrastructure).
  • While Jenkins can be combined with other Docker/Kubernetes to improve scaling, this is difficult (e.g., Tweag I/O had some rather bad experiences with this). In contrast, this is handled by the provider in the CircleCI & AppVeyor case.

Overall, CircleCI & AppVeyor are more scalable.


  • With our own infrastructure, reliability is in our own hands, but we will not be able (as it would be too expensive) to provide any real reliability guarantees or support with guaranteed turn around times.
  • In the case of CircleCI & AppVeyor we depend on the providers for a reliable service. Ben talked to the Rust team who were very positive about CircleCI in this respect, but at the end of the day, it will be hard to know without trying (given we want to rely mostly on free open-source plans).

Overall, let's call this a tie.

Usage estimate

A quick back-of-the-envelope calculation suggests that to simply keep up with our current average commit rate (around 200 commits/month) for the four environments that we currently build we need a bare minimum of:

   200 commit/month
 * 4 build/commit             (Linux/i386, Linux/amd64,
                               OS X, Windows/amd64)
 * 2.5 CPU-hour/build         (approx. average across platforms
                               for a validate)
 / (2 CPU-hour/machine-hour)  (CircleCI appears to use 2 vCPU instances)
 / (30*24 machine-hour/month)
 ~ 2 machines

Note that this doesn't guarantee reasonable wait times but rather merely ensure that we can keep up on the mean. On top of this, we see around 300 differential revisions per month. This requires another 3 machines to keep up.

So, we need at least five machines but, again, this is a minimum; modelling response times is hard but I expect we would likely need to add at least two more machines to keep response times in the contributor-friendly range, especially considering that under Circle CI we will lose the ability to prioritize jobs (by contrast, with Jenkins we can prioritize pull requests as this is the response time that we really care about). Now consider that we would like to add at least three more platforms (FreeBSD, OpenBSD, Linux/aarch64, all of which may be relatively slow to build due to virtualisation overhead) as well as a few more build configurations on amd64 (LLVM, unregisterised, at least one cross-compilation target) and a periodic slow validation and we may be at over a dozen machines.


Circle CI & AppVeyor integration on the Tweag GHC fork:

  • Linux/x86_64 (CircleCI): build & store artefacts works
  • macOS/x86_64 (CircleCI): build & store artefacts works
  • Windows/x86_64 (AppVeyor): asked for increased limits

Update (Jul. 24th, 2018): We wrote a small web application meant to act as a bridge between Phabricator and Circle CI. We have successfully used it to run Circle CI builds against Phabricator differentials. The code lives here. We will soon be deploying it and trying to use it for several job types. See here for an example of using this setup.


Below, we track all work that needs to be done until we have achieved the following specific goals:

  • automatic per-commit builds of master for Linux/x86_64 on CircleCI (per each block of commits from one PR/Differential would be sufficient),
  • automatic builds of Linux/i386, macOS/x86_64 & Windows/x86_64 on CircleCI and AppVeyor at least once per day,
  • automatic Linux/x86_64 build for each PR/Differential on CircleCl,
  • automatic generation of all release artefacts on Linux/i386, Linux/x86_64, macOS/x86_64 & Windows/x86_64 on CircleCI and AppVeyor at least once per day,
  • end-to-end testing by creating a source distribution, from that a binary distribution, and use that for regression testing. (This has the advantage of also testing the distribution creation process.)

Relevant Trac tickets

The following are issues about the CI system itself,

Status: new (18 matches)

Ticket Type Summary Priority Owner
#11958 task Improved testing of cross-compiler low bgamari
#13122 task Investigate reporting build errors with harbormaster.sendmessage normal bgamari
#14416 task CI with CircleCI highest bgamari
#14475 task Upload documentation dumps normal
#14502 task Build Alpine Linux binary distributions normal bgamari
#14505 bug CircleCI only builds pushed heads normal bgamari
#14508 task Bring up Appveyor for Windows CI normal mrkkrp
#14599 bug 32-bit Windows test environment normal bgamari
#14949 bug Perform builds on non-Debian-based systems on Circle CI normal mrkkrp
#15011 task Automate update of VersionHistory table normal bgamari
#15582 bug Phabricator shows "drafts" by default normal bgamari
#15699 bug Run sanity checker in more testsuite runs normal bgamari
#15749 bug Long Harbormaster builds normal bgamari
#15972 bug Test nofib tests for correctness in CI high bgamari
#16134 bug Continuous integration on FreeBSD normal bgamari
#16200 bug Mechanical checking of submodule versions for releases normal bgamari
#16264 bug Install reqlib'd libraries during CI normal bgamari
#16355 task Save CI performance metrics on windows jobs normal davide

The following are GHC issues which are currently breaking CI builds,


  • Talk to CircleCI about increased limits for the free plan for GHC. Determine how much on top of that we need.
  • AppVeyor has a 1h limit for OSS projects by default. Determine how much on top of the OSS plan we need and whether they are willing to relax the limits for a visible OSS project. Status: wrote email, but no response yet.

Per-commit build on Linux/x86_64

Probably easiest to just trigger those builds from GitHub (as all commits are mirrored there anyway).

  • Need to get builds on every individual commit (e.g., to do easy bisection) — see #14505. Alternative: Use GitHub PRs instead of pushing to master directly.

Daily builds on Linux/i386, macOS/x86_64 & Windows/x86_64

  • Implement AppVeyor build config. Blocked: waiting for AppVeyor to increase time limit. (Apparently, Rust are on a payed plan, so maybe we have to do that, too.)
  • Linux/i386 ought to be a small change on Linux/x86_64, or is there more to it?
  • Low priority: Implement end-to-end testing. Blocked: on #14392, #14411 & #14412.
  • Low priority: We want to run ./validate --slow at some point — see #13205.
  • Low priority: We also want to run LLVM and unregisterised builds

Per-PR/Differential build on Linux/x86_64

  • This is the CircleCI 1.0 documentation on Phabricator integration: Does this work with CircleCI 2.0 as well? (The CircleCI API 1.1 supposedly can drive both.)
  • Implement CircleCI/GitHub integration for PRs.

Daily release artifacts for all Tier 1 platforms

  • Tar balls are currently being put into CircleCI artefacts store (where they will be kept for one month).
  • Low priority: Implement S3 upload for longer term storage.
  • Documentation bundle for the website needs to be generated and uploaded. Might be easier with Hadrian:
Last modified 15 months ago Last modified on Jul 24, 2018 1:13:39 PM