@bernardo75/ai-test-analyzer

ai-test-analyzer

AI-powered analysis of automated test reports. Feed it an Allure results directory, a Playwright JSON report or a JUnit XML file and it classifies every failure as:

Verdict	Meaning
product_bug	The product misbehaves; the test is right
broken_test	The test is stale or badly built — includes a diagnosis and a recommended fix
flaky	Non-deterministic failure (timeouts, races, ordering) — includes a stabilization fix
environment_issue	Infra problem (network, DNS, 5xx dependencies, disk)

For Allure it can also enrich the report itself: every failed test gets an " AI Analysis" section, a filterable ai-verdict label, and the Categories tab groups failures by verdict.

Powered by Anthropic Claude behind a minimal provider interface. Disabled by default — with the feature flag off, every call is a strict no-op (zero network, zero tokens, zero file writes).

Install

npm install @bernardo75/ai-test-analyzer

Requires Node.js ≥ 20.

Quickstart

import { analyze } from "@bernardo75/ai-test-analyzer";

// Enable via env: AI_ANALYZER_ENABLED=true, ANTHROPIC_API_KEY=sk-ant-...
const result = await analyze({
  reportPath: "./allure-results",
  format: "allure", // "allure" | "playwright" | "junit"
});

if (result.status === "completed") {
  for (const [testId, entry] of Object.entries(result.verdictsByTest)) {
    if ("verdict" in entry) {
      console.log(testId, entry.verdict, entry.confidence);
      if (entry.verdict === "broken_test") {
        console.log("  diagnosis:", entry.diagnosis);
        console.log("  fix:", entry.recommendedFix);
      }
    } else {
      console.log(testId, "analysis failed:", entry.reason);
    }
  }
  console.log(result.summary);
}

Enrich the Allure report

import { analyzeAndEnrich } from "@bernardo75/ai-test-analyzer";

// Run BEFORE `allure generate`:
const { analysis, enrichment } = await analyzeAndEnrich({
  reportPath: "./allure-results",
  format: "allure",
});
// then: npx allure generate ./allure-results -o ./allure-report

Enrichment is strictly additive and idempotent: it only appends content (attachments, labels, description blocks, a [ai-verdict:*] message suffix, categories.json entries and an ai.analyzer.* block in environment.properties), never deletes or replaces existing results, and re-running it is a no-op.

Feature flag

The analyzer is off by default. Precedence:

1. config.enabled (programmatic — wins in both directions)
2. AI_ANALYZER_ENABLED env var ("true" / "1" to enable)
3. Default: disabled

With the flag off, analyze() / enrichAllureResults() / analyzeAndEnrich() return {status: "disabled"} without reading files, calling the network or consuming tokens — safe to leave installed in CI.

Configuration

await analyze({
  reportPath: "./allure-results",
  format: "allure",
  config: {
    enabled: true,                 // overrides AI_ANALYZER_ENABLED
    apiKey: process.env.MY_KEY,    // overrides ANTHROPIC_API_KEY
    model: "claude-sonnet-5",       // default
    maxConcurrency: 4,             // parallel analyses (default 4)
    maxGroups: 25,                 // hard cost cap per run (default: unlimited)
    pricing: { inputPerMTok: 5, outputPerMTok: 25 }, // cost-estimate override
    failureContext: {              // extra failure evidence sent to the LLM
      pageSnapshot: true,          //   Playwright error-context (text, ~8k cap)
      screenshot: true,            //   failure screenshot (vision providers only)
    },
    provider: myCustomProvider,    // inject your own AnalysisProvider
  },
});

Cost controls

• Only failures are analyzed — a green run costs nothing.
• Deduplication: failures sharing a normalized signature (masked error + own stack frames) are analyzed once; the verdict fans out to every test. 40 failures with 5 root causes = 5 LLM calls.
• Prompt caching: the system prompt is cached across groups of a run.
• maxGroups: hard ceiling of analyses per run.
• Usage report: every run returns totals — failures, groups, duplicates avoided, token usage and an estimated cost in USD.
• Failure evidence: when the report carries a Playwright error-context page snapshot and/or a failure screenshot, they are included in the analysis (size-capped, gated by config.failureContext) — decisive for telling a renamed selector (broken_test) apart from a missing feature (product_bug). The screenshot is used by AnthropicProvider (vision); ClaudeCodeProvider uses the text snapshot only.

result.summary;
// {
//   totalFailures: 40, groupsAnalyzed: 5, groupsFailed: 0,
//   duplicatesAvoided: 35,
//   usage: { inputTokens, outputTokens, cacheCreationInputTokens, cacheReadInputTokens },
//   estimatedCostUsd: 0.19, model: "claude-sonnet-5", durationMs: 41200
// }

API

Export	Description
analyze(options)	Parse + classify. Returns AnalysisResult (disabled | nothing_to_analyze | completed). Never rejects on provider failures — failed groups are reported per-test as analysis_failed.
enrichAllureResults(dir, analysis, options?)	Additive, idempotent enrichment of an allure-results directory.
analyzeAndEnrich(options)	Both steps in one call (Allure only).
AnthropicProvider	Production provider (official @anthropic-ai/sdk, structured outputs, prompt caching).
MockProvider	Deterministic provider for your own tests — zero network.
Errors	ReportNotFoundError, ReportParseError, ConfigurationError, ProviderError.

Credentials are never logged, persisted or written into enriched reports. The only data sent to the LLM is the failure context: error message, stack trace, steps and a ±20-line snippet of the failing test file.

What the analysis looks like in Allure

Each failed test gets:

• An attachment AI Analysis with verdict, confidence, reasoning and — for broken tests — diagnosis and recommended fix.
• A label ai-verdict=<verdict> you can filter/search by.
• A summary block appended to the test description.
• Grouping under Categories: "Product Bugs (AI)", "Broken Tests (AI)", "Flaky (AI)", "Environment Issues (AI)".

And the report home shows the run summary in the Environment widget (ai.analyzer.* keys in environment.properties): model, groups analyzed, duplicates avoided, token usage and estimated cost. User-authored properties are preserved.

Experimental: ClaudeCodeProvider (no API key)

If the machine has Claude Code installed and authenticated — including via a Claude Pro/Max subscription — you can route the analysis through it instead of the API:

import { analyze, ClaudeCodeProvider } from "@bernardo75/ai-test-analyzer";

const result = await analyze({
  reportPath: "./allure-results",
  format: "allure",
  config: {
    enabled: true,
    provider: new ClaudeCodeProvider(), // uses the local `claude` CLI headless
  },
});

For CI, generate a long-lived credential with claude setup-token and expose it to the runner. Read this before using it in a pipeline:

• Subscriptions are personal. Sharing one account across a team pipeline violates Anthropic's usage terms — use an API key (workspace-owned) for team CI. This provider is meant for personal projects and evaluation.
• Subscription rate-limit windows are shared with your interactive usage; CI runs may fail unpredictably when the window is exhausted.
• No structured outputs: the verdict is parsed from text and re-validated against the same schema — invalid responses become per-group analysis failures, never crashes.
• Usage/cost reporting depends on what the CLI returns (may be zeros).

Options: cliPath (default "claude"), model (passed as --model, default "sonnet"), timeoutMs (default 120000).

Examples

Runnable scripts in examples/: mock analysis (no cost), real analysis, enrichment end-to-end, multi-format, and an accuracy-evaluation harness against a human-labeled dataset.

License

MIT