Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#10152 closed task (fixed)

Testsuite diffs can be misleading when normalisers are in use

Reported by: ezyang Owned by:
Priority: low Milestone: 8.0.1
Component: Test Suite Version: 7.11
Keywords: Cc:
Operating System: Unknown/Multiple Architecture: Unknown/Multiple
Type of failure: None/Unknown Test Case:
Blocked By: Blocking:
Related Tickets: Differential Rev(s):
Wiki Page:

Description

When output fails to match, GHC will output the diff based on the original outputs. However, this is misleading: some "differences" will actually have been normalised by the whitespace and more substantive normalisers we apply. So you really have to compare the normalised output to see what the /real/ difference is.

Unfortunately, the real normalised output is missing newlines, so it's not really suitable for diffing. So it's not really clear what an improved algorithm would be.

This seems to not actually be a problem in practice, since SOP for messages which need heavy normalizing is that when the new message looks good, it's just 'make accept'ed. Can be a little confusing when you're developing normalisers.

Change History (3)

comment:1 Changed 4 years ago by Thomas Miedema <thomasmiedema@…>

In 5ddd90415f307cac72d75d86da58e552b168ee30/ghc:

Testsuite: diff non-whitespace normalised output (#10152)

On a test failure, we show a diff between the expected and the actual
output. The method of how we do this has changed a couple of times:

* In 2007: 9951189ccf90b709436fa55ee49eeef031f79f4e
  "On failure, diff the normalised test outputs"

* In 2011: 3019b1e409c129ef7af63e6a7408fb36ec44444b
  "When the output files differ, present the diffs between the *actual*
  output, not the normalised output. The latter may have newlines
  removed, making the diff unreadable."

* In 2015 (now): do something in between.
  - Do apply the normalisers again, to make the diff smaller (only
    showing the actual problem).
  - But don't apply normalise_whitespace, as it indeed makes the diff
    unreadable.

Differential Revision: https://phabricator.haskell.org/D984

comment:2 Changed 4 years ago by thomie

Milestone: 7.12.1
Resolution: fixed
Status: newclosed

comment:3 Changed 4 years ago by thoughtpolice

Milestone: 7.12.18.0.1

Milestone renamed

Note: See TracTickets for help on using tickets.