test-perf-summary v0.0.5

test-perf-summary

Pretty-print the results of Gaia's make test-perf.

Summary mode

The summary mode prints basic descriptive statistics like media, mean and standard deviation in a table.

$ test-perf-summary file1.json

settings                   Median  Mean  Stdev
-------------------------  ------  ----  -----
moz-chrome-dom-loaded         831   834     49
moz-chrome-interactive        832   834     49
moz-app-visually-complete    3619  3878   1725
moz-content-interactive      3620  3879   1725
moz-app-loaded               5142  5742   1761

Delta mode

The delta mode computes the differences between two runs of Gaia's test-perf and tests them for statistical significance (t-test with 0.05 alpha).

$ test-perf-summary file1.json file2.json

settings (means in ms)     Base  Patch  Δ     Sig?
-------------------------  ----  -----  ----  ----
moz-chrome-dom-loaded       834    803   -30  *   
moz-chrome-interactive      834    804   -30  *   
moz-app-visually-complete  3878   3745  -133      
moz-content-interactive    3879   3746  -133      
moz-app-loaded             5742   5715   -27      

In the example above, the test-perf measurements for the Settings app were only stable enough for the moz-chrome-dom-loaded and moz-chrome-interactive events. For these measurements it is valid to assume that the patch improved the performance by 30 milliseconds.

The remaining results, including the apparent 133 ms speed-up, are not significant and might be caused by a random instability of the data. Try increasing the sample size (via the RUNS variable; see below) and run test-perf again.

Installation

npm install -g test-perf-summary

Running Gaia perf tests

For best results, follow the Gaia performance tests guide on MDN. Connect your device to the computer and make sure you export the following variable (I use a local.mk file):

MARIONETTE_RUNNER_HOST=marionette-device-host

Then, run test-perf:

$ APPS="settings sms" RUNS=30 MOZPERFOUT=file1.json make test-perf

Running tests

Install gulp with npm install -g gulp and run:

gulp test