Changes between Version 2 and Version 3 of Building/RunningTests/PerformanceTests


Ignore:
Timestamp:
Feb 22, 2019 12:33:02 PM (10 months ago)
Author:
davide
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Building/RunningTests/PerformanceTests

    v2 v3  
    1 = Running Performance Tests =
     1= Performance Tests =
    22
    3 The test suit contains a number of performance tests included as part of a normal run of the test suit. Performance tests measure some metric (e.g. number of bytes allocated) of ghc or generated code, then compares that metric to some baseline value. Tests will fail if the measured metric is not within some tolerance percentage of the baseline value. If you'd like to add your own test, then see [wiki:Building/RunningTests/Adding#Performancetests adding performance Tests]. To simply run the test suit see [wiki:Building/RunningTests/Running Running the Testsuite] and specifically [wiki:Building/RunningTests/Running#PerformanceTestBaselines here] for establishing performance tests baselines.
     3The test suit contains a number of performance tests included as part of a normal run of the test suit. Performance tests measure some metric (e.g. number of bytes allocated) of ghc or generated code, then compares that metric to some baseline value. Tests will fail if the measured metric is not within some tolerance percentage of the baseline value. If you'd like to add your own test, then see [wiki:Building/RunningTests/Adding#Performancetests adding performance Tests]. To simply run the test suit see [wiki:Building/RunningTests/Running Running the Testsuite] and specifically [wiki:Building/RunningTests/Running#PerformanceTestBaselines here] for establishing performance test baselines.
    44
    55== Performance Metrics are Logged ==
    66
    7 Whenever a performance test is run, the resulting metrics are stored using [https://git-scm.com/docs/git-notes git notes] under the "perf" [https://git-scm.com/docs/git-notes#git-notes---refltrefgt ref space] on the current commit. Git notes are generally stored locally, and not shared between git repositories (e.g. when pushing/pulling branches). This is desirable as performance is largely influenced by the machine on which they were measured. Each metric saved has the following data:
     7Whenever a performance test is run, the resulting metrics are stored using [https://git-scm.com/docs/git-notes git notes] under the "perf" [https://git-scm.com/docs/git-notes#git-notes---refltrefgt ref space] on the current commit. Git notes are generally stored locally, and not shared between git repositories (e.g. when pushing/pulling branches). This is desirable as performance is largely dependent on the machine on which the tests are run. Each metric saved has the following data:
     8[=#MetricDataFields]
    89* **Environment** usually just 'local' unless run on CI.
    910* **Test** the name of the test.
     
    1112* **Metric** what metric was recorded e.g. max bytes allocated.
    1213* **Value** the observed value of the metric.
    13 While you're free to delete notes manually, the test runner will never delete results, so multiple values for the same test may be recorded.
     14While you're free to delete notes manually via the [https://git-scm.com/docs/git-notes git notes --ref=perf] command, the test runner will never delete results, so multiple values for the same test may be recorded.
    1415
    1516=== CI Performance Metrics ===
    1617
    17 Gitlab CI is setup to collect performance metrics and push them (as git notes) to a separate repo: https://gitlab.haskell.org/ghc/ghc-performance-notes.git. You can fetch these results to the "ci/perf" ref space using the following command:
     18Gitlab CI is setup to collect performance metrics and push them (again as git notes) to a separate repo: https://gitlab.haskell.org/ghc/ghc-performance-notes.git. You can fetch these results to the "ci/perf" ref space using the following command:
    1819
     20[=#FetchCiGitNotes]
    1921{{{
    2022$ git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf
     
    2325This allows the test runner to use CI results to derive baselines where local results are not available.
    2426
    25 == How Baslines are Calculated ==
     27== How Baselines are Calculated ==
    2628
    27 TODO How baslines are calculated. How to make usre of CI results.
     29While a tolerance percentage is specified manually in `*.T`, the baseline (i.e. expected) value of performance tests are recovered from previous runs of the performance tests (logged in git notes). Results [#FetchCiGitNotes fetched from CI] may also be used, in which case a CI [#MetricDataFields environment] is chosen based on the local machine architecture and os. Baselines are derived per `(Test, Way, Metric)` independently, by searching git notes as follows:
     30
     311. If the current commit message specifies an [#ExpectedPerformanceChanges expected change] for the metric, then stop. The Baseline is undefined.
     322. Move to the parent commit.
     333. If metrics for the given `(Test, Way, Metric)` exist in a git note for this commit (in ref space `perf`), then the baseline is the average value of those metrics.
     344. Else if metrics for the given `(Chosen CI Environment, Test, Way, Metric)` exist in a git note for this commit (in ref space `ci/perf`), then the baseline is the average value of those metrics.
     355. If the maximum search depth is reached then stop, the baseline is undefined. Else continue to step 1.
     36
     37== Expected Performance Changes ==
     38
     39In many cases, a new commit has expected performance changes. In order to allow the test suit to pass, these changes must be documented in the commit message in the format
     40
     41{{{
     42Metric (In|De)crease <metric(s)> <options>:
     43    <tests>
     44}}}
     45
     46where metrics and options are optional and allow you to specify a metric or list of metrics, the way, and test environment. Here are some examples:
     47
     48{{{
     49Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] (test_env='linux_x86', way='default'):
     50    Test012
     51    Test345
     52Metric Decrease 'bytes allocated':
     53    Test678
     54Metric Increase:
     55    Test711
     56}}}
     57
     58Upon failing some performance tests, the test runner will output  the string required to accept all changes. First double check that you really do expect those changes! If so, you can simply copy the text into the commit message and rerun the tests to ensure they pass.
     59
     60CAUTION: make sure you maintain the correct expected changes in your commit messages when squashing commits.
    2861
    2962== Comparing Commits ==