@saccadejs/core

@saccadejs/core

Webcam eye tracking in the browser: MediaPipe face landmarks → a 36×144 eye crop → an ONNX embedding → a ridge map from embeddings to screen position, fit from your own calibration points. Plus a screen-to-webcam timing loopback that measures the one lag JavaScript cannot see.

This is the jsPsych-agnostic core. For jsPsych experiments use @saccadejs/extension and the saccade-* plugins, which wrap everything here.

Install

npm install @saccadejs/core

The package ships models/eye_embedding.onnx (opset 17, input eye_image float32 [1,36,144,1] in 0–255; outputs embedding float32 [1,128] and cal_weight float32 [1,1], that frame's quality score in [0, 1]). The embedding width is not fixed at 128 — the fit takes its width from the model — but the crop and its preprocessing are exact; see The model. onnxruntime-web and @mediapipe/tasks-vision are dependencies of this package; the <script> build loads them from a CDN instead (see Asset hosting).

Usage

1. Bundler (ESM)

import { SaccadeTracker, defaultGrid13, runCalibration, runValidation } from "@saccadejs/core";

const tracker = new SaccadeTracker({ smoothingFrames: 1 });
await tracker.init(); // camera prompt + model load
tracker.start();

// Show each target yourself; the driver owns the timing.
await runCalibration(
  tracker,
  defaultGrid13(),
  { settleMs: 1000, captureMs: 500 },
  {
    showTarget(target, phase) {
      // target is {x, y} in viewport fractions, or null when the run is over.
      // phase is "settle" (eye still moving) or "capture" (samples being collected).
      draw(target, phase);
    },
  },
);
tracker.fitCalibration(); // lambda defaults to lambdaFor(nPoints)

tracker.onFrame((f) => {
  if (f.gaze) console.log(f.gaze.x, f.gaze.y, f.time.capture);
});

Gaze coordinates are viewport fractions, origin top-left; multiply by window.innerWidth / innerHeight for pixels.

Showing load progress

init() is a multi-second wait the first time — the eye model alone is about 20 MB — so pass onProgress to tell the participant what is happening rather than leaving them at a blank screen. It receives a SaccadeProgress ({ stage, loaded?, total? }) as init walks its stages, in this order: "camera" (the permission prompt), "mediapipe", "landmarker", "ort", "model", "session", "ready". Only "model" carries byte counts, and total is present only when the file's size is known: always for a published release, and otherwise only when the server sent it uncompressed with a Content-Length. The .onnx is fetched once and reused for every execution-provider attempt. Passing no callback changes nothing about init(); tracker.onProgress(cb) subscribes after construction.

const tracker = new SaccadeTracker({
  onProgress: ({ stage, loaded, total }) => {
    if (stage === "model" && total) {
      const mb = (n) => (n / 1e6).toFixed(1);
      setStatus(`Downloading eye model ${mb(loaded)} / ${mb(total)} MB`);
    } else {
      setStatus(stage);
    }
  },
});
await tracker.init();

2. Plain <script> tag

The browser build exposes everything as the global Saccade and bundles neither onnxruntime-web nor MediaPipe — it fetches them at init() from jsDelivr.

<script src="https://unpkg.com/@saccadejs/core/dist/index.browser.min.js"></script>
<script>
  const tracker = new Saccade.SaccadeTracker();
  tracker.init().then(({ ep }) => {
    console.log("running on", ep); // "webgpu" or "wasm"
    tracker.start();
  });
</script>

3. jsPsych

Register the extension and use the plugins; you never touch this API directly.

const jsPsych = initJsPsych({ extensions: [{ type: jsPsychExtensionSaccade }] });

See the documentation site for the extension and the saccade-preview / saccade-calibrate / saccade-validate / saccade-time-sync plugins.

API sketch

new SaccadeTracker(opts)	assets, video: {width, height}, smoothingFrames (default 1), executionProviders (default ["webgpu", "wasm"]), onFrame, onProgress
init()	camera + MediaPipe + ONNX, warmed up. Idempotent. Resolves {ep, videoWidth, videoHeight}
start() / stop() / dispose() / running	the frame loop
video	the live, unmirrored camera element — display it if you like (CSS-mirror the preview, never the pixels the model sees)
onFrame(cb)	returns an unsubscribe function
onProgress(cb)	replays the latest load-progress report, then every new one; returns an unsubscribe function
nextFrame() / nextEmbedding() / nextSample()	one-shot promises; nextSample() pairs the embedding with the model's weight for it
addCalibrationPoint(target, embeddings, weights?), clearCalibration(), getCalibrationPoints()	calibration set
fitCalibration({lambda, center, calHead})	solves the ridge map, returns {lambda, nPoints, weighting}
getCalWeighting()	how the last fit weighted its rows: "model", "head" or "uniform"
getCurrentGaze()	latest {gaze, time} or null
sampleLuminance()	whole-frame mean luminance, what the loopback correlates
defaultGrid13(), trainingGrid20(), validationGrid9(), lambdaFor(n)	grids and the ridge penalty (3 at ≤ 9 points, else 1)
runCalibration, runValidation, runLoopback	drivers
extractEyeCrop, clahe, resizeBilinearCv, rgbaToGray, cropBBox	the preprocessing, exported for testing
solveRidge, predict, calWeight, fitRidge	the ridge fit
estimateLagEdges, estimateLag, sparseSchedule, seededRandom, splitHalves, intervalStats	pure loopback analysis

Keep tracker.video in the document

Chrome only delivers camera frames (requestVideoFrameCallback) for a video element that is actually rendered — in the document and not display: none. A hidden or detached element stops the tracker silently: no error, no frames, and every nextFrame() waiter (a calibration capture, the timing loopback) hangs forever. Move the element wherever you like, but hide it with opacity: 0 and/or a 2×2 px size — never with display: none, visibility: hidden, or by unmounting it.

The tracker defends itself on both sides: if the element is not in the document it re-attaches it to a tiny invisible holder of its own on document.body, and if rVFC stops arriving anyway the frame loop falls back to a ~300 ms timer tick (frames then carry time.source === "callback") until it starts again.

As a last line of defense, runCalibration and runValidation take a timeoutMs in their CollectOptions (default 5000, 0 to wait indefinitely). If one camera frame takes longer than that, the run rejects with no camera frames for 5000 ms rather than hanging, so a caller has something to put on screen. withFrameTimeout(promise, ms) is exported for callers that run their own capture loop.

The preprocessing is bit-exact with the Python pipeline the model was trained on (OpenCV BT.601 gray, INTER_LINEAR resize, CLAHE with clip 2 and 8×8 tiles); test/crop.test.ts checks it byte for byte against fixtures generated by that pipeline. Do not "clean it up".

Asset hosting

By default nothing needs to be served by you: the .onnx resolves next to the package when the build itself is served from a @saccadejs/core/dist/, and otherwise from jsDelivr, as do the onnxruntime-web wasm, the MediaPipe wasm, and Google's face_landmarker.task. To host them yourself (offline labs, or to avoid a third-party request), pass assets:

new SaccadeTracker({
  assets: {
    modelUrl: "/static/eye_embedding.onnx",
    ortWasmUrl: "/static/ort/", // directory with ort's .wasm/.mjs
    mediapipeWasmUrl: "/static/mediapipe/wasm", // directory
    faceLandmarkerUrl: "/static/face_landmarker.task",
  },
});

Copy node_modules/onnxruntime-web/dist/*.{wasm,mjs} and node_modules/@mediapipe/tasks-vision/wasm/* into those directories. ort.env.wasm.wasmPaths is set to ortWasmUrl for you, and numThreads is pinned to 1.

The <script>-tag build additionally needs the two libraries themselves as ES modules; it takes ort.bundle.min.mjs from inside ortWasmUrl and the MediaPipe bundle from jsDelivr. Override those with ortModuleUrl / mediapipeModuleUrl if you are fully offline.

Timing: what FrameTime means

Every frame carries a time: FrameTime, all on the performance.now() clock:

• capture — when the camera delivered the frame, from requestVideoFrameCallback's captureTime (falling back to receiveTime, then to the callback time; source says which). This is the timestamp to record with a gaze sample, not the time the prediction became available.
• meanCapture — because gaze is computed from a ring buffer of the last smoothingFrames embeddings, the smoothed estimate refers to the mean capture time of that buffer, not to the newest frame. At the default of 1 it equals capture; with smoothingFrames: 5 at 30 fps it is ≈ 67 ms before capture.
• emit — when the frame reached your callback. emit - capture is pipeline latency, and only matters for gaze-contingent designs.
• dropped / presentedFrames — camera frames the loop never saw.

What none of these can see is display lag + camera lag: the delay from a requestAnimationFrame paint to photons leaving the panel, plus photons to captureTime. That sum is a per-machine constant, and it is what you must subtract from capture to align gaze with stimulus onsets. runLoopback(tracker) measures it: it flashes a full-viewport panel on a sparse schedule (levels held 0.5–1 s — at most one flash per second, well under the WCAG 2.3.1 limit, and not a flicker), correlates the camera's luminance edges against the flip times, and returns lagMs with an honest uncertainty (plateauWidthMs) and a verdict of "OK", "INCONCLUSIVE" or "UNRELIABLE" with a reason. Only trust lagMs when the verdict is "OK".

License

MIT