@mizchi/flaker

flaker

flaker is a test-intelligence toolkit for:

• sampling a smaller local test run from history and changed files
• detecting flaky tests in noisy CI environments
• measuring how well local sampled runs predict CI
• embedding the same core logic in MoonBit as a library

It is designed for repositories where:

• the full test suite is too expensive to run on every change
• CI failures are noisy because flaky tests are mixed with real regressions
• developers need a smaller local test run that still correlates well with CI

flaker helps answer:

• Which tests should I run for this change?
• How much can I shrink local execution without losing too much confidence?
• Which tests are actually flaky?
• How well does local sampled execution predict CI outcomes?

Upgrading from 0.0.x / 0.1.x? See docs/how-to-use.md#config-migration for the full key rename map. Starting with 0.2.0, the CLI refuses to start on legacy configs and points to the migration guide.

Upgrading from 0.4.x? See docs/migration-0.4-to-0.5.md or docs/migration-0.4-to-0.5.ja.md. 0.5.x keeps existing profiles working, but the recommended user-facing commands are now gate-oriented.

Upgrading from 0.12.x? 0.13.0 is a breaking release: --profile / [profile.*] / FLAKER_PROFILE are gone, replaced by --gate / [gate.iteration|merge|release] / FLAKER_GATE; the random, gbdt, and coverage-guided strategies, cluster_mode, model_path, [coverage], adaptive* keys, dev train, and the ops command group are removed. See docs/migration-0.12-to-0.13.md / docs/migration-0.12-to-0.13.ja.md.

Install as a CLI

pnpm add -D @mizchi/flaker

Or run it without installing:

pnpm dlx @mizchi/flaker --help

Requirements:

• Node.js 24+
• pnpm 10+

Install as a Claude Code plugin

This repo also ships a Claude Code plugin with two skills:

• flaker-setup Introduce flaker on a fresh repository. Day 0 → Week 4 onboarding flow, decision points, copy-paste commands, and pitfalls.
• flaker-management Operate flaker after setup. Advisory vs required gating, nightly triage, quarantine, flaky tag management, and staged Playwright E2E / VRT rollout.

# In Claude Code
/plugin marketplace add mizchi/flaker
/plugin install flaker@flaker

Then ask the agent something like:

• "新しいプロジェクトに flaker をセットアップしたい"
• "flaker の advisory を required に上げる条件を決めたい"
• "E2E VRT の nightly triage を設計したい"

The setup reference checklist lives at docs/new-project-checklist.ja.md and docs/new-project-checklist.md. The 0.4.x -> 0.5.x migration guide lives at docs/migration-0.4-to-0.5.ja.md and docs/migration-0.4-to-0.5.md. The user guide lives at docs/usage-guide.ja.md and docs/usage-guide.md. The operations guide lives at docs/operations-guide.ja.md and docs/operations-guide.md. The operations quick start lives at docs/flaker-management-quickstart.ja.md and docs/flaker-management-quickstart.md.

Use as a MoonBit Library

flaker also publishes a MoonBit library surface at mizchi/flaker.

The root package re-exports both:

• pure computation APIs
• the shared contract types they consume and return

If you prefer a stricter import boundary, the same types are still available from mizchi/flaker/contracts.

import {
  "mizchi/flaker" @flaker,
}

test "sample from historical runs" {
  let meta = @flaker.build_sampling_meta(
    [
      @flaker.SamplingHistoryRowInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
        status: "passed",
        retry_count: 0,
        duration_ms: 1200,
        created_at: "2026-04-03T00:00:00.000Z",
      },
    ],
    [
      @flaker.SamplingListedTestInput::{
        suite: "tests/login.spec.ts",
        test_name: "login works",
        task_id: Some("web-login"),
        filter: None,
        variant: None,
        test_id: None,
      },
    ],
  )

  let sampled = @flaker.sample_weighted(meta, count=1, seed=1UL)
  assert_eq(sampled.length(), 1)
}

The root library surface intentionally re-exports pure logic only:

• flaky detection: detect_flaky
• sampling: build_sampling_meta, sample_random, sample_weighted, sample_hybrid
• affected analysis: resolve_affected, build_affected_report, build_affected_report_from_input
• stable identity: create_stable_test_id, resolve_test_identity
• graph helpers: find_affected_nodes, expand_transitive, topological_sort
• report reducers: summarize_report, classify_report_diff, aggregate_report
• policy: summarize_quarantine, compute_quarantine_exit_code, run_config_check
• metrics: build_sampling_kpi

Contracts remain separate so the API boundary stays explicit and reusable from other packages.

Experimental Direct MoonBit CLI

The MoonBit command package at mizchi/flaker/cmd/flaker is now directly executable on both JS and native targets:

moon run src/cmd/flaker --target js -- --help
moon run src/cmd/flaker --target native -- --version

This direct entrypoint is intentionally narrow for now:

• --help
• --version
• JS bridge exports used by the npm CLI

Host and integration features such as GitHub collection, DuckDB-backed storage, artifact ingestion, and archive handling will be migrated incrementally through target-specific _js.mbt / _native.mbt modules.

Core Workflow

flaker is most useful when you repeat this loop:

1. Collect or import test results from CI and local runs.
2. Build history for flaky detection and sampling.
3. Ask flaker to choose a smaller local test set.
4. Run the sampled tests and compare local outcomes with CI.
5. Evaluate whether the sampling strategy is actually trustworthy.

Mental Model

flaker exposes sampling strategies and execution profiles, but operationally it is easier to reason about it in four layers:

• Gate: a decision boundary such as "can the author keep iterating?", "can this PR merge?", or "can we release?"
• Budget: the constraints that a gate must stay within
• Loop: a background routine that keeps a gate trustworthy over time
• Policy: the rules for how the system reacts when trust drops

In this model, sampling is not the primary concept. Sampling is an internal strategy used to keep a gate inside its budgets.

1. Gates

Most teams only need three gates:

• Iteration Gate: fast local feedback for the author
• Merge Gate: the PR or mainline gate
• Release Gate: a stricter full-suite or pre-release check

2. Budgets

A gate can have more than one budget:

• Time budget: how long humans are willing to wait
• Signal budget: how much false negative or flaky noise is acceptable
• Execution budget: CI minutes, parallelism, browser workers, compute cost
• Product performance budget: LCP, INP, bundle size, latency, or similar

This distinction matters because "performance budget" can mean either:

• execution performance of the test system itself
• performance of the product being tested

3. Loops

Loops are not usually gates. They exist to make gates reliable:

• Observation loop: scheduled full runs, history collection, holdout verification, KPI refresh
• Triage loop: quarantine review, @flaky tagging, promotion, demotion, owner assignment
• Incident loop: retry, confirm, diagnose, and fix after a real failure

4. Policies

Policies encode operational decisions:

• when to auto-quarantine
• when to exclude @flaky tests from normal execution
• when to promote an advisory check to required
• when to demote a required check back to advisory
• how retries are interpreted

Retries can help classify failures, but they are not proof of stability.

Mapping to flaker today

The current CLI and config already fit this model:

Mental model	Current flaker shape (0.14.0)
Iteration Gate	[gate.iteration]
Merge Gate	[gate.merge]
Release Gate	usually a full run, backed by [gate.release] or a dedicated release workflow
Observation loop	flaker apply (runs import --ci when GITHUB_TOKEN is set) + flaker status
Triage loop	flaker status --gate merge --detail + flaker status --list flaky (apply with [quarantine].auto=true also handles suggest/apply)
Incident loop	flaker debug retry / debug confirm / debug diagnose

If you describe flaker this way, the surface area becomes smaller:

• users choose gates and budgets
• operators run loops and maintain policies
• flaker chooses strategies such as affected, hybrid, or full

The ops command group, apply --emit/apply --target, and the old analysis primitives (analyze eval, analyze flaky-tag, policy quarantine) have all been removed; apply, status, and explain now cover the same ground. See docs/migration-0.12-to-0.13.md for the full mapping.

Quick Start

flaker.toml is the desired state. flaker apply reconciles the repo to it, idempotently. Works on fresh repos (no history) and mature repos alike — the reconciler handles the branching.

pnpm add -D @mizchi/flaker
pnpm flaker init --adapter <playwright|vitest|jest|junit> --runner <same or actrun>
# edit [affected].resolver in flaker.toml (workspace | glob | bitflow)
pnpm flaker doctor
pnpm flaker plan          # preview what apply will do
pnpm flaker apply         # converge (safe to re-run)
pnpm flaker status        # dashboard + promotion drift

flaker init generates [gate.iteration] / [gate.merge] / [gate.release] defaults out of the box. flaker apply detects missing CI history, runs a cold-start iteration gate, collects CI runs when GITHUB_TOKEN is set, and tunes sampling via calibrate when there are enough matched commits — in that order, skipping whatever isn't applicable.

The full Day 0 → Week 4 onboarding checklist lives at docs/new-project-checklist.ja.md / .md.

Canonical command forms (0.14.0)

The primary surface is: init, plan, apply, status, run, calibrate, doctor, debug, query, explain, import, report. dev is a hidden maintainer-only group.

Canonical (0.14.0)	Notes
flaker apply	reconciles the repo to flaker.toml; runs CI import + calibrate internally when needed
flaker calibrate [--window-days <n>] [--dry-run] [--json]	recommend and write [sampling]
flaker import --ci [--days <n>] [--branch-filter <branch>]	pull recent GitHub Actions results (needs GITHUB_TOKEN)
flaker status --markdown / flaker explain insights	weekly review artifacts
flaker debug retry / debug confirm / debug diagnose	incident investigation
flaker apply --output <file>	saves ApplyArtifact JSON
flaker plan --output <file>	saves PlanArtifact JSON

The full list of 0.12.x forms these replace (and why) is in docs/migration-0.12-to-0.13.md. See docs/migration-0.12-to-0.13.md for the full breaking-change list and error messages.

Initialize

flaker init --owner your-org --name your-repo

This creates flaker.toml.

Collect test history

From GitHub Actions (needs GITHUB_TOKEN):

export GITHUB_TOKEN=$(gh auth token)
flaker import --ci --days 30

From a local report file:

flaker import <file> --adapter playwright --commit "$(git rev-parse HEAD)"

If the imported artifact came from CI rather than a local run, mark it explicitly:

flaker import <file> --adapter playwright --source ci --commit <sha>

To execute through actrun, configure the workflow path explicitly:

[runner]
type = "playwright"
command = "pnpm exec playwright test -c playwright.config.ts"

[runner.actrun]
workflow = ".github/workflows/ci.yml"
local = true
trust = true
# job = "e2e"

Inspect health and operator state

flaker status
flaker status --gate merge --detail --json
flaker status --list flaky
flaker status --list quarantined

AI-assisted analysis primitives exist for lower-level detail:

flaker explain reason
flaker explain insights
flaker explain cluster
flaker status --markdown
flaker status --markdown > .artifacts/flaker-review.md

The markdown mode is meant for weekly review notes. It summarizes:

• health score
• flaky counts
• local/CI sampling correlation
• average sample ratio
• average saved minutes
• fallback rate

Sample tests before pushing

flaker run --dry-run --strategy hybrid --count 25
flaker run --dry-run --strategy affected --changed src/foo.ts
flaker run --dry-run --gate iteration --changed src/foo.ts

Sample and execute

flaker run --strategy hybrid --count 25
flaker run --strategy affected --changed src/foo.ts
flaker run --gate iteration --changed src/foo.ts
flaker run --runner actrun

flaker run stores the local sampled run and records sampling telemetry so you can later measure:

• P(CI pass | local pass)
• P(CI fail | local fail)
• false negatives / false positives
• average saved minutes

When flaker has no local history yet, the sampling summary now explains the cold-start fallback and suggests the next steps to build usable history. The first flaker run can still sample from the runner's listed tests, record that local run, and use the new history on the next run, so a fresh project can try the package immediately.

Execution Gates

User-facing CLI usage always goes through gates. The old profile-based flags and config sections were removed in 0.13.0 — see docs/migration-0.12-to-0.13.md.

Gate	Config section	Purpose
iteration	[gate.iteration]	Fast local feedback for the author
merge	[gate.merge]	PR / mainline gate
release	[gate.release]	Full or near-full verification

# Auto-detects the gate: merge on CI, iteration otherwise
flaker run

# Explicit gate selection
flaker run --gate iteration
flaker run --gate merge
flaker run --gate release

Configure in flaker.toml:

[runner]
type = "playwright"
command = "pnpm exec playwright test -c playwright.config.ts"
flaky_tag_pattern = "@flaky"

[gate.release]
strategy = "full"

[gate.merge]
strategy = "hybrid"
sample_percentage = 30
skip_flaky_tagged = true

[gate.iteration]
strategy = "affected"
max_duration_seconds = 60
fallback_strategy = "weighted"
skip_flaky_tagged = true

adaptive and the other adaptive_* sampling keys were removed in 0.13.0. Run flaker calibrate periodically instead of tuning sample_percentage automatically.

This supports a simple Playwright workflow:

• release runs all E2E tests and accumulates history
• merge / iteration exclude tests tagged with @flaky
• flaker status --list flaky surfaces quarantine and flaky-tag candidates for operator review

Recommended @flaky loop:

1. Start with full E2E execution so flaker can accumulate baseline history.
2. When a Playwright test is known to be flaky, tag it with @flaky. flaker detects both Playwright tags and @flaky embedded in the test title.
3. Keep release as full execution, and use merge / iteration with skip_flaky_tagged = true. For Playwright, flaker passes --grep-invert @flaky to normal runs.
4. Run flaker apply regularly (it auto-quarantines when [quarantine].auto = true), then review:

flaker status --list flaky
flaker status --markdown > .artifacts/flaker-weekly.md

Data flows downstream: release observation accumulates history → merge uses it for smarter sampling → iteration uses dependency graph for fast feedback. Run flaker calibrate periodically to re-tune sample_percentage from recent history (the adaptive auto-tuning flag was removed in 0.13.0).

For local dogfooding, the practical loop is:

flaker run --dry-run --gate iteration --changed src/foo.ts
flaker run --gate iteration --changed src/foo.ts

CI Integration

PR Comments

Post test results directly on pull requests:

- name: Run flaker
  run: flaker run --gate merge

- name: Post PR comment
  if: github.event_name == 'pull_request'
  run: |
    flaker report summary --adapter vitest --input report.json --pr-comment \
      | gh pr comment ${{ github.event.pull_request.number }} --body-file -
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Auto-create Issues for Quarantined Tests

When flaky tests are auto-quarantined, create tracking issues:

flaker quarantine suggest --json --output .artifacts/quarantine-plan.json
flaker quarantine apply --from .artifacts/quarantine-plan.json --create-issues

This creates a reviewed plan first, then applies it and optionally opens GitHub Issues via gh CLI. Requires gh to be installed and authenticated.

Self-Host Rollout

The repo now ships two GitHub-native self-host lanes:

• ci.yml: a self-host-advisory job runs on pull requests, executes flaker run --gate merge, snapshots kpi / eval, and updates a sticky PR comment.
• nightly-self-host.yml: a scheduled job rebuilds recent CI history with flaker import --ci, runs flaker run --gate release, and updates a rolling issue labeled flaker-self-host.

Both lanes render the same promotion-readiness summary from scripts/self-host-review.mjs. The current default is still advisory: the PR job is non-blocking, and the nightly workflow carries the long-form trend.

Promote flaker run --gate merge to a required check only after the nightly issue shows all five of the following. The primary signal is flaker status (the Promotion drift section reports ready when all five thresholds are met). For the exact numeric values, use flaker status --gate merge --detail --json — this is the authoritative output. (flaker gate review merge --json is the legacy alias, DEPRECATED in 0.7.0, removed in 0.8.0.)

• matched commits >= 20 — commits where both a gated local/CI run and a release/full run exist in the same window, so that local sampling outcomes can be compared against a ground-truth full run. Increases as the nightly --gate release accumulates history.
• false negative rate <= 5% — share of commits where the merge gate passed but the full run failed (sampling missed a real regression). Measured over the same matched-commit window.
• pass correlation >= 95% — P(full run passes | merge gate passes) on matched commits. Same metric referenced elsewhere in this README as P(CI pass | local pass).
• holdout FNR <= 10% — FNR measured on the holdout slice defined by [sampling] holdout_ratio. Tests in the holdout slice are excluded from the sampled run so that their outcomes can be used to audit whether the sampler's verdict generalizes. Guards against sampler overfitting to the visible slice.
• data confidence reaches moderate or high — derived signal combining matched-commit count, history window coverage, and flaky-noise level. Rough rule of thumb: low until ~10 matched commits, moderate around 20–40 with FNR/correlation green, high beyond 40 with stable noise. Exact boundaries come from the status --gate merge --detail --json output, not from config.

Recommended Usage Model

Start with advisory mode, not CI gating.

The most practical rollout looks like this:

1. flaker run --gate release in a nightly scheduled workflow (full test + data accumulation)
2. flaker run --gate merge on PR push (selective execution, posts PR comment)
3. flaker run --gate iteration during development (fast feedback)
4. Review flaker status and flaker status --gate merge --detail --json weekly
5. Only tighten the workflow after local-to-CI correlation looks strong

This works best in repositories with:

• long CI times
• flaky tests
• structured reports such as Playwright JSON, JUnit XML, or Vitest JSON

Main Commands

Collection and import

flaker import --ci --days 30
flaker import <file> --adapter playwright

[coverage] and the coverage-guided strategy were removed in 0.13.0; see docs/coverage-guided-sampling.md.

Sampling and execution

flaker run --dry-run --strategy weighted --count 20
flaker run --dry-run --strategy affected
flaker run --dry-run --strategy hybrid --count 50

flaker run --strategy hybrid --count 50
flaker run --strategy affected --changed src/foo.ts

The random, gbdt, and coverage-guided strategies were removed in 0.13.0; use weighted, affected, hybrid, or full.

Flag precedence

Resolution order (highest to lowest):
  1. Explicit CLI flag          (--strategy, --percentage, --count)
  2. [gate.<name>] in flaker.toml      (via --gate or auto-detection)
  3. [sampling] in flaker.toml         (project default)
  4. Built-in defaults

Notes:
  --count overrides --percentage when both are given
  --changed overrides git auto-detection
  --dry-run suppresses execution, still records selection telemetry
  --explain can be combined with --dry-run or a real run

Analysis

flaker status --list flaky
flaker explain reason
flaker status --markdown
flaker calibrate
flaker query "SELECT * FROM test_results LIMIT 20"

dev train (GBDT model training) was removed in 0.13.0 along with the gbdt strategy.

Confirm suspected failures

# Remote: triggers workflow_dispatch, waits for result
flaker debug confirm "tests/api.test.ts:handles timeout"
flaker debug confirm "tests/api.test.ts:handles timeout" --repeat 10

# Local: runs via test runner directly
flaker debug confirm "tests/api.test.ts:handles timeout" --runner local

Output classification (based on --repeat N runs, default N=5):

• BROKEN — fails on every repeat (failures == N). Treat as a real regression.
• FLAKY — fails on some but not all repeats (0 < failures < N). Candidate for @flaky tag or quarantine.
• TRANSIENT — passes on every repeat (failures == 0). The original CI failure did not reproduce here; either CI-environment specific or a one-shot noise event.

Use --repeat 10 (or higher) when you suspect a low-rate flake that the default N=5 might miss. Higher N trades wall time for classification confidence.

Requires .github/workflows/flaker-confirm.yml for remote mode — generated by flaker init. If your repo predates this file, re-run flaker init --force or copy the template from templates/flaker-confirm.yml in this repo.

Retry CI failures locally

# Re-run failed tests from most recent CI failure
flaker debug retry

# From a specific workflow run
flaker debug retry --run 12345678

Fetches the test result artifact from the failed CI run, identifies failed tests, and re-runs them locally. Reports which failures reproduce (real regressions) vs which don't (CI-specific or flaky).

Policy and ownership

flaker quarantine suggest --json --output .artifacts/quarantine-plan.json
flaker quarantine apply --from .artifacts/quarantine-plan.json --create-issues
flaker policy check
flaker exec affected --changed src/foo.ts

Reporting

flaker report summary --adapter vitest --input report.json --markdown
flaker report summary --adapter vitest --input report.json --pr-comment
flaker report diff --base base.json --head head.json

Minimal Configuration

[repo]
owner = "your-org"
name = "your-repo"

[storage]
path = ".flaker/data.duckdb"

[adapter]
type = "playwright"

[runner]
type = "playwright"
command = "pnpm exec playwright test"
flaky_tag_pattern = "@flaky"

[affected]
resolver = "workspace"

[flaky]
window_days = 14
detection_threshold_ratio = 0.02

[quarantine]
auto = true
flaky_rate_threshold_percentage = 30
min_runs = 10

[sampling]
strategy = "hybrid"
sample_percentage = 30
holdout_ratio = 0.1
co_failure_window_days = 90

[gate.release]
strategy = "full"

[gate.merge]
strategy = "hybrid"
sample_percentage = 30
skip_flaky_tagged = true

[gate.iteration]
strategy = "affected"
max_duration_seconds = 60
fallback_strategy = "weighted"
skip_flaky_tagged = true

Docs

• 新規プロジェクト導入チェックリスト (ja)
• New Project Onboarding Checklist
• Usage Guide (ja)
• Usage Guide
• Operations Guide (ja)
• Operations Guide
• Detailed Command Reference
• Using flaker with jev-test-filter / (ja)
• Migrating from 0.12.x to 0.13.0 (ja)
• Migrating from 0.12.x to 0.13.0
• Why flaker
• Design Partner Rollout

For contributing and dogfood workflows, see docs/contributing.md.

Release

The published npm entry point is:

node_modules/@mizchi/flaker/dist/cli/main.js

The package is built with Rolldown and includes the bundled MoonBit core in dist/moonbit/flaker.js.