# @shop-os/qa-judge

Latest version **0.1.0** (published 2026-09-24) · 0 weekly downloads

## Install

```sh
npm install @shop-os/qa-judge
pnpm add @shop-os/qa-judge
yarn add @shop-os/qa-judge
bun add @shop-os/qa-judge
```

## Health

**Score 65/100 (B)** — status: active.

Positive: has types; esm support; no vulnerabilities; recently updated; high maintenance score.

Warnings: low downloads; pre 1.0.

## Facts

| | |
|---|---|
| Version | 0.1.0 |
| Published | 2026-09-24 |
| First published | 2026-09-24 |
| Weekly downloads | 0 |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Node | >=24 |
| Dependencies | 1 |
| Unpacked size | 151 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Maintainers | vikas-shopos, soham-shopos, shashwa7-shopos |

## Links

- npm: https://www.npmjs.com/package/@shop-os/qa-judge
- Repository: https://github.com/Shop-OS/shopos-backend-ts
- npm.io page: https://npm.io/package/@shop-os/qa-judge

## Dependencies (1)

- [zod](https://npm.io/package/zod.md) ^4.4.3

## Recent versions

- 0.1.0 (latest) — 2026-09-24

## README

# @shop-os/qa-judge

An SDK-independent image quality judge owned by the backend workspace. The approved
SpaceLab Goal Mode plan calls for a shared package for Ask AI and subsequent
Enterprise judging. The host injects its evaluator and image access; this package
owns frozen evidence contracts, response validation and deterministic acceptance.

## API

```ts
import { createJudge, EvaluatorResponseSchema, hashImageBytes } from "@shop-os/qa-judge";
import type { Evaluator, EvidenceAccess, JudgeRequest } from "@shop-os/qa-judge";

// Implement these ports in the host, where authorization and model configuration live.
declare const evaluator: Evaluator;
declare const evidenceAccess: EvidenceAccess;
declare const request: JudgeRequest;

const judgment = await createJudge({ evaluator, evidenceAccess }).judge(request, {
    signal: AbortSignal.timeout(120_000),
});
```

The public entry exports all schemas and their inferred types, `createJudge`,
`hashImageBytes`, `JudgeError`, the constant `JUDGE_INSTRUCTIONS`, and version constants.
It builds ESM and CommonJS for Node 24+. Its only runtime dependency is Zod 4.

### Frozen request

`JudgeRequest` contains:

- `identity`: test, operation and SKU IDs plus goal, input, output and workflow hashes.
- `goal`: description, expected output, deliverables and criteria.
- `productImages`: this SKU's product truth.
- `styleReferences`: optional style examples, separated from product truth.
- `outputs`: the actual required final output set, with `deliverableId`, `nodeId` and
  `nodeRunId` on every output. The host applies its output policy and canonical node
  bindings before calling. The package never assigns deliverables by array position.

Every image reference has a globally unique `id`, a SHA-256 hex `contentHash` and an
optional untrusted `label`. Asset access belongs in the injected port, so the request
contains no fetch URLs or credentials. Use actual image-byte hashes, not URL hashes.
The host owns the other frozen hashes; the package returns them verbatim and computes
an additional fingerprint covering the complete goal, exact evidence and check versions.

Each criterion has `id`, `description`, `required` and `allowMinorDeviation`.
`required: true` means non-negotiable; combining it with minor tolerance is invalid.
Optional `deliverableIds` restricts a criterion to those deliverables. Otherwise it
applies to every supplied output. Every criterion still requires a result.

Each deliverable has an exact `count`, optional pixel `width`/`height`, and optional
`aspectRatio: { width, height }`. `mediaType`, when present, must be `"image"`.
Translate a UI ratio such as `"4:5"` into the numeric object before invoking the package.
Image IDs, criterion IDs and deliverable IDs cannot repeat.

### Evidence port

`EvidenceAccess.resolve(reference, { role, signal })` authorizes access and returns:

```ts
{
    status: "available",
    id: reference.id,
    contentHash: reference.contentHash,
    bytes,
    metadata: {
        source: "decoded",
        decoder: { name: "sharp", version: decoderVersion },
        width,
        height,
        mediaType: "image/png",
    },
}
```

The host must decode those exact bytes. Metadata copied from a prompt, file name,
generator's requested settings, or unverified upload declarations does not qualify.
The package validates metadata shape, verifies its reference identity and recomputes
the SHA-256 over the bytes before invoking the evaluator. It copies the bytes and
checks them again after evaluation.

For an unavailable frozen asset, return `{ status: "unavailable", id, contentHash,
reason }`. The judgment becomes `unknown` unless a measurable failure already exists.
No visual call runs against incomplete required evidence. Style references are optional
at setup; once included in the frozen baseline, they must remain accessible for that
judgment. Transport failures can throw so the host applies technical recovery.

### Evaluator port

`Evaluator.evaluate(request, { signal })` receives:

- `instructions`: the package's constant evaluation policy.
- `untrustedContext`: the frozen request as data.
- `evidence`: exact image bytes, decoded metadata and fixed roles
  `product_truth | style_reference | output`.
- `inputFingerprint`: copy this into the structured model response.

The host can use AI SDK 6/ModelKit `generateText` with
`Output.object({ schema: EvaluatorResponseSchema })`. Keep `instructions` in the
instruction channel. Put the context and labeled image parts in ordinary user content;
never promote reference text, OCR, filenames or captions into instructions. A style
reference supplies appearance/composition, never another SKU's identity.

Return `{ response, metadata, usage }`. `response` is validated by the package.
`metadata` contains `evaluatorId`, `evaluatorVersion`, `provider`, `model`, nullable
`modelVersion` and nullable `providerRequestId`. `usage` contains nullable
`inputTokens`, `outputTokens`, `totalTokens` and `costUsd`. The adapter supplies actual
metadata and usage separately from the model response; unknown values are null.

The model response contains only `inputFingerprint` and `checks`. Every check includes
`criterionId`, `verdict`, `severity`, observed `evidence`, complete evaluated `outputIds`,
`affectedOutputIds`, and nullable `suggestedChange`. Coverage and defects are distinct:
for a criterion covering two images with one defect, both IDs appear in `outputIds`
and only the defective image appears in `affectedOutputIds`.

## Acceptance and errors

The package accepts no overall model score or model-supplied aggregate verdict.
It validates every criterion exactly once, exact output coverage, affected-output
identity, and severity consistency. Missing, duplicate and invented identities throw
`JudgeError` with code `invalid_evaluator_response`.

| Evidence                                                                          | Aggregate result                                                |
| --------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| Every criterion and measurable check passes                                       | `pass`                                                          |
| Only minor failures on explicitly tolerant, non-required criteria                 | `pass_with_minor_deviations`, with explicit accepted deviations |
| Any non-tolerated failure, missing/extra deliverable, wrong decoded size or ratio | `fail`                                                          |
| Valid uncertain judgment or unavailable evidence, without a known failure         | `unknown`                                                       |

Major/critical failures always block. Unknown is never converted to a minor deviation.
Accepted deviations are returned only when the entire result is accepted. The report
includes independent measurable checks, exact input identity, evidence metadata,
evaluator metadata, actual usage and package/policy/schema/check versions.

`invalid_request` is a setup/integration error. Invalid decoded evidence and malformed
evaluator replies are technical errors, not quality verdicts. For malformed replies
with valid call metadata, `JudgeError.call` preserves measured metadata and usage.
Provider/access exceptions and cancellation propagate. This package retries nothing.
The host persists technical failures and charges according to its existing policies.

## Host responsibilities

The host owns authorization, image decoding, criterion confirmation, output-node
bindings, durable orchestration, persistence, billing, technical recovery and publication.
It also owns the bounded critical-defect confirmation policy: two agreeing judgments
on unchanged inputs with at most one tiebreaker. A confirmation uses a new operation
ID; the input fingerprint deliberately excludes that call ID so unchanged evidence
remains comparable. A valid unknown judgment needs attention, not endless retries.

Offline fixtures verify the software boundary; they do not establish visual-model
accuracy or prompt-injection resistance. Choose the host's model using human-reviewed
evaluation data before enabling automatic corrections.

## Release

Build and test this package independently, then use `pnpm pack` for local consumer
integration. External consumers use the published `@shop-os/qa-judge` version. The
first public release needs registry bootstrap/trusted-publisher setup by the release
owner; this change does not create that external configuration or publish the package.
No provider SDK, credentials, database code, workflow controller or UI dependencies
are included in the distributable.

---
_Source: https://npm.io/package/@shop-os/qa-judge · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._