@shop-os/qa-judge

@shop-os/qa-judge

An SDK-independent image quality judge owned by the backend workspace. The approved SpaceLab Goal Mode plan calls for a shared package for Ask AI and subsequent Enterprise judging. The host injects its evaluator and image access; this package owns frozen evidence contracts, response validation and deterministic acceptance.

API

import { createJudge, EvaluatorResponseSchema, hashImageBytes } from "@shop-os/qa-judge";
import type { Evaluator, EvidenceAccess, JudgeRequest } from "@shop-os/qa-judge";

// Implement these ports in the host, where authorization and model configuration live.
declare const evaluator: Evaluator;
declare const evidenceAccess: EvidenceAccess;
declare const request: JudgeRequest;

const judgment = await createJudge({ evaluator, evidenceAccess }).judge(request, {
    signal: AbortSignal.timeout(120_000),
});

The public entry exports all schemas and their inferred types, createJudge, hashImageBytes, JudgeError, the constant JUDGE_INSTRUCTIONS, and version constants. It builds ESM and CommonJS for Node 24+. Its only runtime dependency is Zod 4.

Frozen request

JudgeRequest contains:

• identity: test, operation and SKU IDs plus goal, input, output and workflow hashes.
• goal: description, expected output, deliverables and criteria.
• productImages: this SKU's product truth.
• styleReferences: optional style examples, separated from product truth.
• outputs: the actual required final output set, with deliverableId, nodeId and nodeRunId on every output. The host applies its output policy and canonical node bindings before calling. The package never assigns deliverables by array position.

Every image reference has a globally unique id, a SHA-256 hex contentHash and an optional untrusted label. Asset access belongs in the injected port, so the request contains no fetch URLs or credentials. Use actual image-byte hashes, not URL hashes. The host owns the other frozen hashes; the package returns them verbatim and computes an additional fingerprint covering the complete goal, exact evidence and check versions.

Each criterion has id, description, required and allowMinorDeviation. required: true means non-negotiable; combining it with minor tolerance is invalid. Optional deliverableIds restricts a criterion to those deliverables. Otherwise it applies to every supplied output. Every criterion still requires a result.

Each deliverable has an exact count, optional pixel width/height, and optional aspectRatio: { width, height }. mediaType, when present, must be "image". Translate a UI ratio such as "4:5" into the numeric object before invoking the package. Image IDs, criterion IDs and deliverable IDs cannot repeat.

Evidence port

EvidenceAccess.resolve(reference, { role, signal }) authorizes access and returns:

{
    status: "available",
    id: reference.id,
    contentHash: reference.contentHash,
    bytes,
    metadata: {
        source: "decoded",
        decoder: { name: "sharp", version: decoderVersion },
        width,
        height,
        mediaType: "image/png",
    },
}

The host must decode those exact bytes. Metadata copied from a prompt, file name, generator's requested settings, or unverified upload declarations does not qualify. The package validates metadata shape, verifies its reference identity and recomputes the SHA-256 over the bytes before invoking the evaluator. It copies the bytes and checks them again after evaluation.

For an unavailable frozen asset, return { status: "unavailable", id, contentHash, reason }. The judgment becomes unknown unless a measurable failure already exists. No visual call runs against incomplete required evidence. Style references are optional at setup; once included in the frozen baseline, they must remain accessible for that judgment. Transport failures can throw so the host applies technical recovery.

Evaluator port

Evaluator.evaluate(request, { signal }) receives:

• instructions: the package's constant evaluation policy.
• untrustedContext: the frozen request as data.
• evidence: exact image bytes, decoded metadata and fixed roles product_truth | style_reference | output.
• inputFingerprint: copy this into the structured model response.

The host can use AI SDK 6/ModelKit generateText with Output.object({ schema: EvaluatorResponseSchema }). Keep instructions in the instruction channel. Put the context and labeled image parts in ordinary user content; never promote reference text, OCR, filenames or captions into instructions. A style reference supplies appearance/composition, never another SKU's identity.

Return { response, metadata, usage }. response is validated by the package. metadata contains evaluatorId, evaluatorVersion, provider, model, nullable modelVersion and nullable providerRequestId. usage contains nullable inputTokens, outputTokens, totalTokens and costUsd. The adapter supplies actual metadata and usage separately from the model response; unknown values are null.

The model response contains only inputFingerprint and checks. Every check includes criterionId, verdict, severity, observed evidence, complete evaluated outputIds, affectedOutputIds, and nullable suggestedChange. Coverage and defects are distinct: for a criterion covering two images with one defect, both IDs appear in outputIds and only the defective image appears in affectedOutputIds.

Acceptance and errors

The package accepts no overall model score or model-supplied aggregate verdict. It validates every criterion exactly once, exact output coverage, affected-output identity, and severity consistency. Missing, duplicate and invented identities throw JudgeError with code invalid_evaluator_response.

Evidence	Aggregate result
Every criterion and measurable check passes	pass
Only minor failures on explicitly tolerant, non-required criteria	pass_with_minor_deviations, with explicit accepted deviations
Any non-tolerated failure, missing/extra deliverable, wrong decoded size or ratio	fail
Valid uncertain judgment or unavailable evidence, without a known failure	unknown

Major/critical failures always block. Unknown is never converted to a minor deviation. Accepted deviations are returned only when the entire result is accepted. The report includes independent measurable checks, exact input identity, evidence metadata, evaluator metadata, actual usage and package/policy/schema/check versions.

invalid_request is a setup/integration error. Invalid decoded evidence and malformed evaluator replies are technical errors, not quality verdicts. For malformed replies with valid call metadata, JudgeError.call preserves measured metadata and usage. Provider/access exceptions and cancellation propagate. This package retries nothing. The host persists technical failures and charges according to its existing policies.

Host responsibilities

The host owns authorization, image decoding, criterion confirmation, output-node bindings, durable orchestration, persistence, billing, technical recovery and publication. It also owns the bounded critical-defect confirmation policy: two agreeing judgments on unchanged inputs with at most one tiebreaker. A confirmation uses a new operation ID; the input fingerprint deliberately excludes that call ID so unchanged evidence remains comparable. A valid unknown judgment needs attention, not endless retries.

Offline fixtures verify the software boundary; they do not establish visual-model accuracy or prompt-injection resistance. Choose the host's model using human-reviewed evaluation data before enabling automatic corrections.

Release

Build and test this package independently, then use pnpm pack for local consumer integration. External consumers use the published @shop-os/qa-judge version. The first public release needs registry bootstrap/trusted-publisher setup by the release owner; this change does not create that external configuration or publish the package. No provider SDK, credentials, database code, workflow controller or UI dependencies are included in the distributable.