# @saccadejs/core

> saccade.js core: webcam eye tracking in the browser (MediaPipe face landmarks → ONNX eye embedding → ridge calibration), with a screen-to-webcam timing loopback

Latest version **0.3.0** (published 2026-09-23) · MIT license · 0 weekly downloads

## Install

```sh
npm install @saccadejs/core
pnpm add @saccadejs/core
yarn add @saccadejs/core
bun add @saccadejs/core
```

## Health

**Score 75/100 (B)** — status: active.

Positive: has types; esm support; no vulnerabilities; has provenance; recently updated; high maintenance score; high quality score.

Warnings: low downloads; large bundle; pre 1.0.

## Facts

| | |
|---|---|
| Version | 0.3.0 |
| Published | 2026-09-23 |
| First published | 2026-09-22 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Dependencies | 2 |
| Unpacked size | 21 MB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| Provenance | attested (GitHub Actions) |
| GitHub stars | 0 |
| Author | Josh de Leeuw |
| Maintainers | jodeleeuw |
| Keywords | jsPsych, eye tracking, webcam, gaze, saccade.js |

## Links

- npm: https://www.npmjs.com/package/@saccadejs/core
- Repository: https://github.com/jspsych/saccadejs
- Homepage: https://github.com/jspsych/saccadejs/blob/main/packages/core/README.md
- Issues: https://github.com/jspsych/saccadejs/issues
- npm.io page: https://npm.io/package/@saccadejs/core

## Dependencies (2)

- [onnxruntime-web](https://npm.io/package/onnxruntime-web.md) ^1.29.0
- [@mediapipe/tasks-vision](https://npm.io/package/@mediapipe/tasks-vision.md) ^0.10.35

## Recent versions

- 0.3.0 (latest) — 2026-09-23
- 0.2.0 — 2026-09-22

## README

# @saccadejs/core

Webcam eye tracking in the browser: MediaPipe face landmarks → a 36×144 eye crop →
an ONNX embedding → a ridge map from embeddings to screen position, fit from
your own calibration points. Plus a screen-to-webcam timing loopback that measures the one lag
JavaScript cannot see.

This is the jsPsych-agnostic core. For jsPsych experiments use
[`@saccadejs/extension`](https://saccade.jspsych.org/reference/extension/) and the `saccade-*` plugins,
which wrap everything here.

## Install

```
npm install @saccadejs/core
```

The package ships `models/eye_embedding.onnx` (opset 17, input `eye_image` float32
`[1,36,144,1]` in 0–255; outputs `embedding` float32 `[1,128]` and `cal_weight` float32
`[1,1]`, that frame's quality score in [0, 1]). The embedding width is not fixed at 128 —
the fit takes its width from the model — but the crop and its preprocessing are exact; see
[The model](https://saccade.jspsych.org/models/). `onnxruntime-web` and
`@mediapipe/tasks-vision` are dependencies of this package; the `<script>` build loads them
from a CDN instead (see [Asset hosting](#asset-hosting)).

## Usage

### 1. Bundler (ESM)

```js
import { SaccadeTracker, defaultGrid13, runCalibration, runValidation } from "@saccadejs/core";

const tracker = new SaccadeTracker({ smoothingFrames: 1 });
await tracker.init(); // camera prompt + model load
tracker.start();

// Show each target yourself; the driver owns the timing.
await runCalibration(
  tracker,
  defaultGrid13(),
  { settleMs: 1000, captureMs: 500 },
  {
    showTarget(target, phase) {
      // target is {x, y} in viewport fractions, or null when the run is over.
      // phase is "settle" (eye still moving) or "capture" (samples being collected).
      draw(target, phase);
    },
  },
);
tracker.fitCalibration(); // lambda defaults to lambdaFor(nPoints)

tracker.onFrame((f) => {
  if (f.gaze) console.log(f.gaze.x, f.gaze.y, f.time.capture);
});
```

`Gaze` coordinates are **viewport fractions**, origin top-left; multiply by
`window.innerWidth` / `innerHeight` for pixels.

#### Showing load progress

`init()` is a multi-second wait the first time — the eye model alone is about 20 MB — so pass
`onProgress` to tell the participant what is happening rather than leaving them at a blank
screen. It receives a `SaccadeProgress` (`{ stage, loaded?, total? }`) as init walks its
stages, in this order: `"camera"` (the permission prompt), `"mediapipe"`, `"landmarker"`,
`"ort"`, `"model"`, `"session"`, `"ready"`. Only `"model"` carries byte counts, and `total` is
present only when the file's size is known: always for a published release, and otherwise only
when the server sent it uncompressed with a `Content-Length`. The `.onnx` is fetched once and
reused for every execution-provider attempt. Passing no callback changes nothing about `init()`;
`tracker.onProgress(cb)` subscribes after construction.

```js
const tracker = new SaccadeTracker({
  onProgress: ({ stage, loaded, total }) => {
    if (stage === "model" && total) {
      const mb = (n) => (n / 1e6).toFixed(1);
      setStatus(`Downloading eye model ${mb(loaded)} / ${mb(total)} MB`);
    } else {
      setStatus(stage);
    }
  },
});
await tracker.init();
```

### 2. Plain `<script>` tag

The browser build exposes everything as the global `Saccade` and bundles neither
onnxruntime-web nor MediaPipe — it fetches them at `init()` from jsDelivr.

```html
<script src="https://unpkg.com/@saccadejs/core/dist/index.browser.min.js"></script>
<script>
  const tracker = new Saccade.SaccadeTracker();
  tracker.init().then(({ ep }) => {
    console.log("running on", ep); // "webgpu" or "wasm"
    tracker.start();
  });
</script>
```

### 3. jsPsych

Register the extension and use the plugins; you never touch this API directly.

```js
const jsPsych = initJsPsych({ extensions: [{ type: jsPsychExtensionSaccade }] });
```

See the [documentation site](https://saccade.jspsych.org/) for the extension and the
`saccade-preview` / `saccade-calibrate` / `saccade-validate` / `saccade-time-sync` plugins.

## API sketch

|                                                                                                     |                                                                                                                                                 |
| --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `new SaccadeTracker(opts)`                                                                          | `assets`, `video: {width, height}`, `smoothingFrames` (default 1), `executionProviders` (default `["webgpu", "wasm"]`), `onFrame`, `onProgress` |
| `init()`                                                                                            | camera + MediaPipe + ONNX, warmed up. Idempotent. Resolves `{ep, videoWidth, videoHeight}`                                                      |
| `start()` / `stop()` / `dispose()` / `running`                                                      | the frame loop                                                                                                                                  |
| `video`                                                                                             | the live, **unmirrored** camera element — display it if you like (CSS-mirror the preview, never the pixels the model sees)                      |
| `onFrame(cb)`                                                                                       | returns an unsubscribe function                                                                                                                 |
| `onProgress(cb)`                                                                                    | replays the latest load-progress report, then every new one; returns an unsubscribe function                                                    |
| `nextFrame()` / `nextEmbedding()` / `nextSample()`                                                  | one-shot promises; `nextSample()` pairs the embedding with the model's weight for it                                                            |
| `addCalibrationPoint(target, embeddings, weights?)`, `clearCalibration()`, `getCalibrationPoints()` | calibration set                                                                                                                                 |
| `fitCalibration({lambda, center, calHead})`                                                         | solves the ridge map, returns `{lambda, nPoints, weighting}`                                                                                    |
| `getCalWeighting()`                                                                                 | how the last fit weighted its rows: `"model"`, `"head"` or `"uniform"`                                                                          |
| `getCurrentGaze()`                                                                                  | latest `{gaze, time}` or null                                                                                                                   |
| `sampleLuminance()`                                                                                 | whole-frame mean luminance, what the loopback correlates                                                                                        |
| `defaultGrid13()`, `trainingGrid20()`, `validationGrid9()`, `lambdaFor(n)`                          | grids and the ridge penalty (3 at ≤ 9 points, else 1)                                                                                           |
| `runCalibration`, `runValidation`, `runLoopback`                                                    | drivers                                                                                                                                         |
| `extractEyeCrop`, `clahe`, `resizeBilinearCv`, `rgbaToGray`, `cropBBox`                             | the preprocessing, exported for testing                                                                                                         |
| `solveRidge`, `predict`, `calWeight`, `fitRidge`                                                    | the ridge fit                                                                                                                                   |
| `estimateLagEdges`, `estimateLag`, `sparseSchedule`, `seededRandom`, `splitHalves`, `intervalStats` | pure loopback analysis                                                                                                                          |

### Keep `tracker.video` in the document

Chrome only delivers camera frames (`requestVideoFrameCallback`) for a video element that is
actually **rendered** — in the document and not `display: none`. A hidden or detached element
stops the tracker silently: no error, no frames, and every `nextFrame()` waiter (a calibration
capture, the timing loopback) hangs forever. Move the element wherever you like, but hide it
with `opacity: 0` and/or a 2×2 px size — never with `display: none`, `visibility: hidden`, or by
unmounting it.

The tracker defends itself on both sides: if the element is not in the document it re-attaches
it to a tiny invisible holder of its own on `document.body`, and if rVFC stops arriving anyway
the frame loop falls back to a ~300 ms timer tick (frames then carry
`time.source === "callback"`) until it starts again.

As a last line of defense, `runCalibration` and `runValidation` take a `timeoutMs` in their
`CollectOptions` (default `5000`, `0` to wait indefinitely). If one camera frame takes longer
than that, the run rejects with `no camera frames for 5000 ms` rather than hanging, so a caller
has something to put on screen. `withFrameTimeout(promise, ms)` is exported for callers that run
their own capture loop.

The preprocessing is **bit-exact** with the Python pipeline the model was trained on (OpenCV
BT.601 gray, `INTER_LINEAR` resize, CLAHE with clip 2 and 8×8 tiles); `test/crop.test.ts`
checks it byte for byte against fixtures generated by that pipeline. Do not "clean it up".

## Asset hosting

By default nothing needs to be served by you: the `.onnx` resolves next to the package when
the build itself is served from a `@saccadejs/core/dist/`, and otherwise from jsDelivr, as do the
onnxruntime-web wasm, the MediaPipe wasm, and Google's `face_landmarker.task`. To host them
yourself (offline labs, or to avoid a third-party request), pass `assets`:

```js
new SaccadeTracker({
  assets: {
    modelUrl: "/static/eye_embedding.onnx",
    ortWasmUrl: "/static/ort/", // directory with ort's .wasm/.mjs
    mediapipeWasmUrl: "/static/mediapipe/wasm", // directory
    faceLandmarkerUrl: "/static/face_landmarker.task",
  },
});
```

Copy `node_modules/onnxruntime-web/dist/*.{wasm,mjs}` and
`node_modules/@mediapipe/tasks-vision/wasm/*` into those directories. `ort.env.wasm.wasmPaths`
is set to `ortWasmUrl` for you, and `numThreads` is pinned to 1.

The `<script>`-tag build additionally needs the two libraries themselves as ES modules; it
takes `ort.bundle.min.mjs` from inside `ortWasmUrl` and the MediaPipe bundle from jsDelivr.
Override those with `ortModuleUrl` / `mediapipeModuleUrl` if you are fully offline.

## Timing: what `FrameTime` means

Every frame carries a `time: FrameTime`, all on the `performance.now()` clock:

- **`capture`** — when the camera delivered the frame, from `requestVideoFrameCallback`'s
  `captureTime` (falling back to `receiveTime`, then to the callback time; `source` says
  which). This is the timestamp to record with a gaze sample, not the time the prediction
  became available.
- **`meanCapture`** — because gaze is computed from a ring buffer of the last `smoothingFrames`
  embeddings, the smoothed estimate refers to the _mean_ capture time of that buffer, not to the
  newest frame. At the default of 1 it equals `capture`; with `smoothingFrames: 5` at 30 fps it is
  ≈ 67 ms before `capture`.
- **`emit`** — when the frame reached your callback. `emit - capture` is pipeline latency, and
  only matters for gaze-contingent designs.
- **`dropped` / `presentedFrames`** — camera frames the loop never saw.

What none of these can see is **display lag + camera lag**: the delay from a `requestAnimationFrame`
paint to photons leaving the panel, plus photons to `captureTime`. That sum is a per-machine
constant, and it is what you must subtract from `capture` to align gaze
with stimulus onsets. `runLoopback(tracker)` measures it: it flashes a full-viewport panel on a
sparse schedule (levels held 0.5–1 s — at most one flash per second, well under the WCAG 2.3.1
limit, and not a flicker), correlates the camera's luminance edges against the flip times, and
returns `lagMs` with an honest uncertainty (`plateauWidthMs`) and a `verdict` of `"OK"`,
`"INCONCLUSIVE"` or `"UNRELIABLE"` with a `reason`. Only trust `lagMs` when the verdict is
`"OK"`.

## License

MIT

---
_Source: https://npm.io/package/@saccadejs/core · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._