Opened 7 months ago

Closed 6 months ago

Last modified 5 months ago

#16199 closed bug (fixed)

GHC 8.6 bundles libraries that don't correspond to Hackage releases

Reported by: RyanGlScott Owned by:
Priority: highest Milestone: 8.6.4
Component: libraries (other) Version: 8.6.3
Keywords: Cc: angerman, bgamari
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: #14678 Differential Rev(s): https://gitlab.haskell.org/ghc/ghc/merge_requests/139
Wiki Page:

Description

GHC 8.6 claims to be bundled with transformers-0.5.5.0, but this isn't true. That's because it's using transformers commit ​https://gitlab.haskell.org/ghc/packages/transformers/tree/80557845cdc0e72bc05cec19cf7a1bf5495e9e69, which happened after the commit corresponding to the Hackage release of 0.5.5.0 (https://gitlab.haskell.org/ghc/packages/transformers/commit/33b3c8a71778ae37040088dfe022c648373777a8).

This is not the first time this has happened: see #14678. While we were able to catch #14678 before a major release was shipped, but unfortunately, that didn't happen this time.

Attachments (1)

verify-packages.sh (896 bytes) - added by angerman 7 months ago.
verify-packages.sh

Download all attachments as: .zip

Change History (15)

comment:1 Changed 7 months ago by RyanGlScott

Last edited 7 months ago by RyanGlScott (previous) (diff)

comment:2 Changed 7 months ago by RyanGlScott

Summary: GHC 8.6's bundled version of transformers doesn't correspond to a Hackage releaseGHC 8.6 bundles libraries that don't correspond to Hackage releases

comment:3 Changed 7 months ago by bgamari

Sigh, this is quite bad. We really need to rule this sort of thing out systematically from the release process. I'll work on this.

In the meantime I have bumped process to 1.6.3.0 and transformers to 0.5.5.0 on ghc-8.6.

comment:4 Changed 7 months ago by RyanGlScott

We should be careful about transformers, since there are actually API-relevant code changes that were introduced in commit https://gitlab.haskell.org/ghc/packages/transformers/commit/80557845cdc0e72bc05cec19cf7a1bf5495e9e69. In particular, I'm pretty sure that contravariant-1.5 requires these changes to compile (at least, angerman reported an error when he tried to build contravariant-1.5 with the Hackage version of transformers-0.5.5.0).

comment:5 Changed 7 months ago by angerman

So I've come up with a plan which I think should work. I'll try to whip it up today and see how it performs. I'll need the produces artifacts anyway to get this resolved.

For each of the (external) libraries, we'll run cabal new-sdist (1) Then we'll pull the same one from hackage with cabal get --pristine. (2) And finally run a diff over the unpacked one from (1) and (2).

This should yield an empty set.

I'll try using this approach to generate me the sufficient patches for the packages that are in ghc and for which hackage has different source.

comment:6 Changed 7 months ago by angerman

Alright, here we are (for ghc-8.6.3-release)

-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 Win32-2.6.1.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 array-0.5.3.0.patch
-rw-r--r--   1 angerman  staff   642B Jan 18 10:00 binary-0.8.6.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 bytestring-0.10.8.2.patch
-rw-r--r--   1 angerman  staff    70B Jan 18 10:00 containers-0.6.0.1.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 deepseq-1.4.4.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 directory-1.3.3.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 filepath-1.4.2.1.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 haskeline-0.7.4.3.patch
-rw-r--r--   1 angerman  staff   675B Jan 18 10:00 hpc-0.6.0.3.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 mtl-2.2.2.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 parallel-3.2.2.0.patch
-rw-r--r--   1 angerman  staff   1.4K Jan 18 10:00 parsec-3.1.13.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 pretty-1.1.3.6.patch
-rw-r--r--   1 angerman  staff   6.0K Jan 18 10:00 process-1.6.3.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 stm-2.5.0.0.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 terminfo-0.4.1.2.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 text-1.2.3.1.patch
-rw-r--r--   1 angerman  staff   5.4K Jan 18 10:00 time-1.8.0.2.patch
-rw-r--r--   1 angerman  staff    20K Jan 18 10:00 transformers-0.5.5.0.patch
-rw-r--r--   1 angerman  staff   4.8K Jan 18 10:00 unix-2.7.2.2.patch
-rw-r--r--   1 angerman  staff     0B Jan 18 10:00 xhtml-3000.2.2.1.patch

Changed 7 months ago by angerman

Attachment: verify-packages.sh added

verify-packages.sh

comment:7 Changed 7 months ago by angerman

The verify-packages.sh will generate the cabal sdists, pull the corresponding package from hackage, and diff them. Giving you the output above in the end. Ideally they should all be 0B in size.

comment:8 Changed 7 months ago by angerman

Differential Rev(s): https://gitlab.haskell.org/ghc/ghc/merge_requests/139
Status: newpatch

comment:9 in reply to:  5 Changed 7 months ago by hvr

Replying to angerman:

For each of the (external) libraries, we'll run cabal new-sdist (1) Then we'll pull the same one from hackage with cabal get --pristine. (2) And finally run a diff over the unpacked one from (1) and (2).

This should yield an empty set.

Fwiw, we can't guarantee this in general, nor do we need to.

There is no requirement that you can actually reinstall a boot package release bundled with GHC. There's no need for that, and there's packages that inevitably fall into this category. So tooling needs to be able to cope with that anyway; and in fact Cabal does.

That being said, the only invariant that actually matters and the only one I care about very strongly is that of consistent API versioning:

If a package foo-1.2.3 is bundled with GHC then if-and-only-if there is a valid build-plan (meeting all involved version constraints), then the foo-1.2.3 from Hackage re-synthesized via Cabal must not to be distinguishable at the API level from the GHC pkg-db pre-installed one. In other words, foo-1.2.3 shall uniquely determine the observable API exposed of that package.

A very prominent example was the mtl package that shipped with GHC 8.4.1 which broke the API versioning and caused real-world problems which (see https://github.com/haskell-hvr/HsYAML/issues/1 which required some very awkward CPP -- but in general this kind of problem is not always workaroundable by CPP! We must be able to trust the API versioning or we'd have to give up on version numbers altogether)

Anything else, like diverging testsuites/benchmark components or other deltas which have no effect on the exposed API are of very little concern to me.

Avoiding divergences would require enough time between the final release and the preceding release of a GHC last-beta or release-candidate after which boot package maintainers can be encouraged to publish their package release candidates on Hackage to the primary index (and I have to stress: "and no sooner!", as it's strongly discouraged to make releases on Hackage which target or advertise support for unreleased dependencies or tooling, or causing avoidable uploads to Hackage as is clearly specified in the Hackage upload guidelines), after we have ensured (in topological dep ordering) that the package candidates are sound and in sync with the artifacts included in the GHC release, and also fix any minor and not so minor issues detected in the process.

or spelled out differently, the protocol for finalising boot libraries would be more or less like

  1. release a last-beta/release-candidate
  2. go over all boot libraries in topological sorted order and ensure their exposed API matched the Hackage release candidate; ensure that the compatibility advertised via version constraints is accurate; if needed, coordinate fixups with the maintainer
  3. restart 2. in case there's not-so-minor changes which might cause an avalanche effect on their rev-deps
  4. if there were fixups needed, make sure to ./validate again;
  5. when it's clear that the release-candidate has no more issues (otherwise go back to 1.) and we're going to cut the final release; encourage package maintainers to publish their latest Hackage package release candidate (which we verified to match the one contained in GHC's release) to the main index (i.e. press that "publish" button in the Hackage release candidate UI)

comment:10 Changed 7 months ago by jrp

I don't know whether this is just regular cabal hell or an instance of this problem, but running:

cabal new-install parsec
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: ghc-8.6.3/installed-8.6... (user goal)
[__1] next goal: process (user goal)
[__1] rejecting: process-1.6.5.0 (conflict: ghc =>
process==1.6.3.0/installed-1.6...)
[__1] rejecting: process-1.6.4.0, process-1.6.3.0/installed-1.6...,
process-1.6.3.0, process-1.6.2.0, process-1.6.1.0, process-1.6.0.0,
process-1.5.0.0, process-1.4.3.0, process-1.4.2.0, process-1.4.1.0,
process-1.4.0.0, process-1.3.0.0, process-1.2.3.0, process-1.2.2.0,
process-1.2.1.0, process-1.2.0.0, process-1.1.0.2, process-1.1.0.1,
process-1.1.0.0, process-1.0.1.5, process-1.0.1.4, process-1.0.1.3,
process-1.0.1.2, process-1.0.1.1, process-1.0.0.0 (constraint from user target
requires ==1.6.5.0)
[__1] fail (backjumping, conflict set: ghc, process)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: process, ghc

If an instance of this problem, then I'd recommend an 8.6.4...

comment:11 Changed 6 months ago by RyanGlScott

https://gitlab.haskell.org/ghc/ghc/merge_requests/139 was merged. Should we close this ticket, or would it behoove us to keep it open until we can verify that 8.6.4 fixes this issue?

comment:12 Changed 6 months ago by bgamari

Milestone: 8.6.4
Resolution: fixed
Status: patchclosed

Let's close it.

comment:13 Changed 6 months ago by jrp

It would be great to get 8.6.4 out, if only to fix this bug, which is a real productivity killer as I seem to need to empty my .ghc and .cabal directories and reinstall the libraries needed for whatever my project using to get going. Unless someone knows of a better workaround.

Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: directory-1.3.3.2 (user goal)
[__1] next goal: ghc (user goal)
[__1] rejecting: ghc-8.6.3/installed-8.6... (conflict: directory==1.3.3.2, ghc
=> directory==1.3.3.0/installed-1.3...)
[__1] rejecting: ghc-8.6.1, ghc-8.4.3, ghc-8.4.1, ghc-8.2.2, ghc-8.2.1
(constraint from user target requires ==8.6.3)
[__1] fail (backjumping, conflict set: directory, ghc)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: directory, ghc

comment:14 Changed 5 months ago by Marge Bot <ben+marge-bot@…>

In 6e15ca54/ghc:

Bump transformers to 0.5.6.2

See #16199.
Note: See TracTickets for help on using tickets.