Changes between Version 10 and Version 11 of Performance/Tests


Ignore:
Timestamp:
Oct 10, 2018 9:30:43 AM (14 months ago)
Author:
davide
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Performance/Tests

    v10 v11  
    131131
    132132* CircleCI must push test results (git notes) to the git.haskell.com repo.
    133 * Make clear what that plan is regarding the [#Drift drift issue].
    134 * Hear the concerns of the Haskell community.
     133
     134== Future work
     135
     136* With performance results saved as git notes, we are open to creating better tool support for analysing that data (see [#Drift drift issue]).
     137* Though a large part of the benefit of this change would be to lower tolerance percentages for tests, that will only be done later, after results have been collected.
     138
    135139
    136140==== [=#Drift Drift Issue]
     
    138142The change to comparing with the previous commit vs an expected value means that performance results may drift in one direction without tests ever failing. e.g. With a tolerance value of 10% and an initial performance result of 100, the test will pass even if the next few commits drift well past 10% of the original results: 110, 121, 121, 133, 146. Tests pass because the commit-to-commit change is never more than 10%. We need some means of detecting such drift.
    139143
    140 == Future work
    141 
    142 * With performance results saved as git notes, we are open to creating better tool support for analysing that data.
    143 * Though a large part of the benefit of this change would be to lower tolerance percentages for tests, that will only be done later, after results have been collected.
     144It appears that automatically detecting drift is a hard problem. So we leave it up to the developers to check for drift manually. At the moment via a CLI tool, though in the future it should be possible to create better tools that e.g. plot the data.
    144145
    145146== See also