0.3.2 • Published yesterday

test-harness-kit

Licence

Apache-2.0

Version

0.3.2

Deps

Vulns

Weekly

Summary Dependency Versions

test-harness-kit

Portable, self-maintaining tests your AI agent can actually run.

test-harness-kit is the reference implementation of TRIF (Test Recipe Interchange Format): an open format that captures an application's testing knowledge once, as portable human-readable recipes, so any AI agent, automation engine, or human can execute a test from a tiny slice of context — and tell a real product bug apart from its own stale assumption.

Status: v0.3 — usable, honest, pre-1.0. trif/0.1 is frozen; trif/0.2 adds the interchange layer and the binding-first runtime (a recipe graduates to a compiled script and runs with no model; on drift it falls back and heals — validated end-to-end, see the Wikipedia run report). The benchmark results are honestly mixed (see TestVoyager). Feedback and entries welcome.

Why

Natural-language browser agents can test an app they've never seen — but they pay two costs on every run:

Re-exploration. The agent re-derives navigation, selectors, and timing from scratch each time. Knowledge from run N evaporates before run N+1.
Verdict conflation. When a step fails, the agent can't tell "the app has a bug" from "the app changed and my assumption is stale" — so it either cries wolf or silently adapts to a regression.

TRIF fixes both by putting the knowledge back into a versioned artifact: explore once, write a cookbook; then any executor runs one recipe at O(1-test) context cost, grades it against an explicit oracle, and returns one of four verdicts — with drift detected and named, not silently absorbed.

flowchart LR
  EX["Explore once<br/>(funded)"] -->|writes| CB["Cookbook<br/>recipes · anchors · map · healing"]
  CB -->|one recipe, ~1.5k ctx| RUN["Any executor<br/>playwright · cdp · agent · human"]
  RUN -->|graded oracle| V["PASS · FAIL-BUG · FAIL-STALE · BLOCKED"]
  V -->|friction| LEDGER["Freshness ledger"]
  LEDGER -->|maintainer patch| CB

Install

npm install -g test-harness-kit      # global CLI
# or run without installing:
npx test-harness-kit start           # guided onboarding
# or from source (zero dependencies):
git clone https://github.com/huytieu/test-harness-kit
node test-harness-kit/cli/test-harness-kit.mjs

Requires Node ≥ 18. The CLI has zero runtime dependencies. To run recipes against a live app you also need one browser engine — Playwright (npm i -g playwright && npx playwright install chromium) or a Chrome started with --remote-debugging-port=9222.

Two ways to use it

You run a cookbook, not a URL. So there are two situations, and they need different help:

A. Hand it to a coding agent. The package ships a top-level SKILL.md — a kit-level Agent Skill. Drop the kit (or just that file) where your agent looks for skills and tell it what you want:

cp SKILL.md ~/.claude/skills/test-harness-kit/     # Claude Code / Codex / Cursor
# then, to the agent:  "author a TRIF cookbook for https://my-app.example and run the smoke suite"

The skill teaches the agent the whole workflow — decide, author a cookbook by exploring the app once, run it binding-first, and heal on drift — plus when a model is and isn't needed. Authoring (the one step that needs an agent) is written up in docs/authoring.md.

B. Run it yourself from the terminal / CI. Once a cookbook exists (you authored one, or you use a bundled one), no agent is involved: recipes with a compiled binding run as plain Playwright scripts — $0, no model, no API key. See Quick start and the binding-first execution model.

New here? Read docs/concepts.md for the mental model, then pick a lane above.

Quick start (60 seconds)

Guided tour — works anywhere:

npx test-harness-kit start        # or just `npx test-harness-kit` in a terminal

On your own app — init writes a cookbook into the current directory, then you operate on that path:

npx test-harness-kit init my-app --url https://my-app.example
#   scaffolded ./my-app.trif
npx test-harness-kit validate my-app.trif
#   PASS: my-app.trif conforms to TRIF 0.1
npx test-harness-kit list          # cookbooks in the current directory

init scaffolds a valid skeleton (one placeholder surface + recipe); you then explore the app once and write the real recipes — see docs/authoring.md, or hand the job to an agent via SKILL.md. After that, run any recipe forever.

Explore the bundled sample cookbooks — clone the repo and point commands at cookbooks/<name>.trif:

git clone https://github.com/huytieu/test-harness-kit && cd test-harness-kit
node cli/test-harness-kit.mjs list cookbooks
#   spliteasy     v1.0.1  recipes:26  surfaces:3   freshness:ok  cookbooks/spliteasy.trif
#   the-internet  v1.0.2  recipes:16  surfaces:11  freshness:ok  cookbooks/the-internet.trif
#   … (11 sites)

node cli/test-harness-kit.mjs show cookbooks/the-internet.trif TC-13   # read one recipe
node cli/test-harness-kit.mjs run  cookbooks/the-internet.trif TC-13 --adapter cdp
#   TC-13  PASS/clean  1.0s

Running needs a browser engine up — e.g. Chrome with --remote-debugging-port=9222 for the cdp adapter. Harder cases carry a capability: TC-16 exercises Shadow DOM and needs a shadow-dom-capable adapter, otherwise it honestly reports BLOCKED/capability.

Core concepts

A cookbook is a versioned directory, <app>.trif/:

my-app.trif/
  trif.json          # manifest: app URL, budgets, roles, provenance
  SKILL.md           # the router — a valid Agent Skills file (pack-specific facts only)
  recipes/TC-*.md    # one self-contained test each (~200 tokens): steps + oracle
  map/anchors.json   # UI target registry: role + name + note + engine hints
  map/surfaces/      # per-surface operating notes
  healing.md         # what to do when a recipe goes stale
  coverage.json      # freshness ledger (ok | suspect | stale per surface)

Four verdicts, never blended — this is the point:

Verdict	Meaning
`PASS`	the app satisfies the intent
`FAIL-BUG`	a real product defect (quoted evidence)
`FAIL-STALE`	the app drifted; the cookbook is out of date, not the app
`BLOCKED`	couldn't run (auth wall, app down, missing capability)

Anchor resolution ladder — a recipe never hard-codes a selector. Each @anchor resolves by: engine hint (fast) → semantic role+name → agentic fallback → miss (never improvised past silently). A hint going stale degrades gracefully instead of breaking.

Honesty rules (trif/0.2) — every executed step records how it resolved its target; resolving off the recorded hint without declaring it is non-conformant. Silent healing is a lint failure, and a PASS must declare which oracle assertions actually fired:

node cli/lint-result.mjs result.json --strict

Four doors

The same cookbook is consumed four ways — pick whichever fits your agent:

Door	How
Agent Skills	A cookbook's `SKILL.md` is a valid skill router. Drop the folder into Claude Code / Codex / Cursor and the agent auto-loads it.
CLI	`npx test-harness-kit run ...` — for CI, cron, and any shell-capable agent on any model.
Interchange	`ready` / `compile-binding` / `import` — NL recipe script (below).
Raw files	It's markdown and JSON. Read a recipe, run it by hand, diff a healing patch in a PR.

Interchange layer — NL Recipe Script

The "IF" in TRIF. A stable recipe can graduate to a real engine script (script speed, no model), and an existing script suite can lift up into draft recipes. A recipe compiles to both shipped engines by default — Playwright and Katalon Studio — from the same source, so you can validate it on either side:

# which recipes are deterministic enough to compile to a script?
node cli/test-harness-kit.mjs ready cookbooks/the-internet.trif
#   15 ready / 1 not-ready

# compile a recipe to BOTH default engines (no --engine); keyed to the pack version
node cli/test-harness-kit.mjs compile-binding cookbooks/cellphone.trif TC-02
#   compiled TC-02 (playwright) -> cookbooks/cellphone.trif/bindings/playwright/TC-02.spec.ts
#   compiled TC-02 (katalon)    -> cookbooks/cellphone.trif/bindings/katalon/TC-02.groovy
# (playwright = @playwright/test spec; katalon = Katalon Studio Groovy / WebUI keywords.
#  --engine playwright | katalon to pick one.)

# reverse: lift an existing Playwright spec into a draft recipe for review
node cli/test-harness-kit.mjs import legacy/checkout.spec.ts --out drafts/

Bindings are never authoritative — on drift they go suspect and are recompiled. Plain-text assertions round-trip cleanly on import; attribute-level assertions surface as explicit TODOs rather than guessed syntax. Compiled bindings are byte-reproducible (no timestamp; keyed to a fingerprint of the recipe + its anchors), so the sample cookbooks ship theirs under cookbooks/<name>.trif/bindings/playwright/ as runnable examples.

Execution model — binding-first, self-optimizing

run tries three lanes in order, so tests get faster and cheaper over time without ever losing truthfulness:

flowchart LR
  R["run TC"] --> F{fresh binding?}
  F -->|yes| B["① Binding lane<br/>compiled script · fast · $0 model"]
  B -->|pass| V1["PASS (lane: binding)"]
  B -->|fail → drop as suspect| I
  F -->|no / stale| I["② Intent lane<br/>recipe via adapter · grades oracle"]
  I -->|drift| H["③ Healing<br/>FAIL-STALE + patch"]
  I -->|pass & deterministic-ready| G["graduate → recompile binding"]

Binding lane — if a fresh compiled binding exists (recipe and anchors unchanged since compile), run the native script: fast, deterministic, no model. A green run is the verdict.
Intent lane — on a miss, a stale binding, or a binding failure (never a verdict on its own — the script is dropped as suspect), execute the recipe through the adapter and grade the oracle. This is the source of truth for bug-vs-drift.
Healing — on drift the intent lane emits FAIL-STALE + a patch proposal.

When the intent lane passes a deterministic-ready recipe, it graduates — recompiles the binding so the next run takes the fast lane. The loop closes itself.

npx test-harness-kit run my-app.trif TC-001 --adapter cdp
#   TC-001  PASS/clean  1200ms  [binding:playwright]        # fast lane
#   TC-001  binding recipe or anchors changed -> intent lane
#   TC-001  PASS/clean  7300ms  [intent:cdp] -> graduated to playwright binding
# flags: --no-binding  --binding-engine <e>  --no-graduate

The binding engine is the user's (resolved at runtime); if it isn't installed the lane is skipped silently and the intent lane runs. Other engines (Selenium, Cypress, Katalon Groovy) plug in behind the same runBinding(pack, case, engine) contract.

Seen end-to-end: bench/reports/wikipedia-e2e.html runs the Wikipedia cookbook through the whole loop with real numbers — first agent run, graduation to scripts (7–38× faster at $0 model), a live drift case that fell back to FAIL-STALE, and the "pay to learn, not to run" cost model.

TestVoyager — the benchmark

bench/ is TestVoyager, a vendor-neutral benchmark and leaderboard for AI web test-execution tools. Where WebVoyager scores whether an agent can complete a web task, TestVoyager scores whether a tool can test one: return the right verdict on a live app, catch a real bug, and survive hard UI.

Any tool can enter — the contract is {url, intent} → verdict; it does not require adopting TRIF. A raw browser agent, Scout, the Katalon AI runner, or a hand-written suite all compete on the same suite.
Two axes: lifecycle (first-run · repeat-stability · self-heal · verdict fidelity) × difficulty (shadow DOM · mislabelled objects · complex flows · confusing situations).
~115 cases across 11 public sites, ~15% traps, frozen oracles, per-site reporting, and a scorer that is a pure function of the entrant's log.

open bench/leaderboard.html          # current standings + charts
node bench/leaderboard.mjs           # regenerate from bench/entrants/

Spec and how to submit an entry: bench/BENCHMARK.md. Results are honestly mixed and reported in full — a testing standard that hides its own results has no business grading anyone else's.

CLI reference

test-harness-kit                       guided onboarding (TTY) / playbook (piped)
test-harness-kit start                 guided human onboarding
test-harness-kit list [dir|cookbook]   inventory cookbooks / one cookbook's recipes
test-harness-kit show <cb> <TC-id>     one recipe, prologue inlined, anchors expanded
test-harness-kit search <cb> "<q>"     ranked matching lines
test-harness-kit init <name> --url <u> scaffold a new valid cookbook
test-harness-kit validate <cb>         TRIF 0.1 conformance check
test-harness-kit run <cb> <TC..> --adapter <playwright|cdp|agentic|human>
test-harness-kit ready <cb>            deterministic-ready badge per recipe (0.2)
test-harness-kit compile-binding <cb> <TC..> --engine playwright   recipe → script (0.2)
test-harness-kit import <spec> [--out <dir>]   script → draft recipe (0.2)
test-harness-kit stale <cb> [--max-age <days>] freshness report
test-harness-kit verify <cb> --adapter <e>     re-check anchors against the live app
test-harness-kit heal <cb> list|apply <patch.json> [--human-signed]
test-harness-kit budgets               self-test the token budgets

Environment: THK_CDP_URL (CDP endpoint, default http://localhost:9222), THK_MODEL_CMD (model command for compile). The cdp adapter needs a Chrome started with --remote-debugging-port (a raw HTTP /json/version endpoint) — the chrome://inspect checkbox mode does not expose one.

Repo layout

spec/          trif-0.1.md (frozen) + trif-0.2-additions.md (draft)   — CC-BY-4.0
schemas/       cookbook / result / patch / coverage JSON Schemas
cli/           zero-dep reference CLI: run, validate, compile-binding, import, lint
adapters/      playwright · cdp · agentic (any computer-use model) · human checklist
cookbooks/     sample cookbooks (spliteasy, the-internet, …)
docs/          concepts, quickstarts (agent-skills / CLI / MCP / raw), evolution
bench/         TestVoyager — benchmark suite, oracles, scorer, leaderboard
fixtures/      validator + linter test suites

Contributing

Enter the benchmark: implement {url, intent} → verdict, run the suite, PR a results JSONL + a row in bench/entrants/index.json. Scores are recomputed from your log — no self-reported numbers. See bench/BENCHMARK.md.
Improve a baseline: the reference adapters and baselines are written to be good; an entry that beats them by improving the baseline is the benchmark working as intended.
npm test runs the packaging guard, conformance, validator, and linter.

License

Apache-2.0 for the kit (cli/, adapters/, schemas/, tooling) · CC-BY-4.0 for the spec documents under spec/. See LICENSE.

Keywords

testing ai-agents test-automation interchange-format trif agent-skills playwright e2e