@thi.ng/bench v3.5.5

This project is part of the @thi.ng/umbrella monorepo.

• About
  • Status
  • Related packages
• Installation
• Dependencies
• Usage examples
• API
  • Benchmarking with statistics
  • Benchmark suites
  • Output formatting
• Authors
• License

About

Benchmarking utilities w/ various statistics & formatters (CSV, Markdown etc.).

Though no public API change (only additions), since v2.0.0 this library internally attempts to use high-res ES BigInt timestamps (in Node via process.hrtime.bigint()). If BigInt is not available in the target environment, timestamps are still only sourced via Date.now().

Status

STABLE - used in production

Search or submit any issues for this package

Related packages

• @thi.ng/csv - Customizable, transducer-based CSV parser/object mapper and transformer
• @thi.ng/hiccup-markdown - Markdown parser & serializer from/to Hiccup format

Installation

yarn add @thi.ng/bench

ES module import:

<script type="module" src="https://cdn.skypack.dev/@thi.ng/bench"></script>

Skypack documentation

For Node.js REPL:

# with flag only for < v16
node --experimental-repl-await

> const bench = await import("@thi.ng/bench");

Package sizes (gzipped, pre-treeshake): ESM: 1.39 KB

Dependencies

• @thi.ng/api

Usage examples

Several demos in this repo's /examples directory are using this package.

A selection:

API

Generated API docs

import { timed, bench, benchmark } from "@thi.ng/bench";

// test functions
const fib = (n) => n > 2 ? fib(n - 1) + fib(n - 2) : n > 0 ? 1 : 0;

const fib2 = (n) => {
    const res = [0, 1];
    for(let i = 2; i <= n; i++) {
        res[i] = res[i - 1] + res[i - 2];
    }
    return res[n];
};

// measure single execution time
timed(() => fib(40));
// 714ms
// 102334155
timed(() => fib2(40));
// 0ms
// 102334155

// measure 1mil iterations (default)
bench(() => fib(10), 1e6);
// 395ms
// 55

bench(() => fib2(10), 1e6);
// 53ms
// 55

Benchmarking with statistics

The benchmark() function executes a number of warmup runs, before executing the main measurement and producing a number of useful statistics: mean, median, min/max, 1st/3rd quartile, standard deviation (as percentage)...

See api.ts for configuration options.

Also see the formatting section below for other output options. This example uses the default format...

benchmark(() => fib(40), { title: "fib", iter: 10, warmup: 5 });
// benchmarking: fib
//         warmup... 3707.17ms (5 runs)
//         executing...
//         total: 7333.72ms, runs: 10
//         mean: 733.37ms, median: 733.79ms, range: [728.58..743.43]
//         q1: 730.98ms, q3: 735.03ms
//         sd: 0.54%

// also returns results:
// {
//   title: "fib",
//   iter: 10,
//   total: 7333.72402,
//   mean: 733.372402,
//   median: 733.794194,
//   min: 728.5808,
//   max: 743.432538,
//   q1: 730.980115,
//   q3: 735.025314,
//   sd: 0.542200865574415
// }

Benchmark suites

Multiple benchmarks can be run sequentially as suite (also returns an array of all results):

b.suite(
    [
        { title: "fib2(10)", fn: () => fib2(10) },
        { title: "fib2(20)", fn: () => fib2(20) },
        { title: "fib2(30)", fn: () => fib2(30) },
        { title: "fib2(40)", fn: () => fib2(40) },
    ],
    { iter: 10, size: 100000, warmup: 5, format: b.FORMAT_MD }
)

// |                   Title|    Iter|    Size|       Total|    Mean|  Median|     Min|     Max|      Q1|      Q3|     SD%|
// |------------------------|-------:|-------:|-----------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:|
// |                fib2(10)|      10|  100000|       54.34|    5.43|    5.15|    4.40|    8.14|    4.84|    6.67|   20.32|
// |                fib2(20)|      10|  100000|      121.24|   12.12|   12.13|   11.73|   12.91|   11.93|   12.35|    2.61|
// |                fib2(30)|      10|  100000|      152.98|   15.30|   14.51|   13.93|   20.77|   14.35|   16.35|   12.65|
// |                fib2(40)|      10|  100000|      164.79|   16.48|   15.60|   15.01|   19.27|   15.42|   18.80|    9.34|

Same table as actual Markdown:

Output formatting

The following output formatters are available. Custom formatters can be easily defined (see source for examples). Formatters are configured via the format option given to benchmark() or suite().

• FORMAT_DEFAULT - default plain text formatting
• FORMAT_CSV - Comma-separated values (w/ column header)
• FORMAT_MD - Markdown table format

Authors

Karsten Schmidt

If this project contributes to an academic publication, please cite it as:

@misc{thing-bench,
  title = "@thi.ng/bench",
  author = "Karsten Schmidt",
  note = "https://thi.ng/bench",
  year = 2018
}

License

© 2018 - 2021 Karsten Schmidt // Apache Software License 2.0