@exellix/jobs-db

@exellix/jobs-db

Mongo-backed data tier for the Exellix job queue. This is the only package that imports mongodb. It exposes a small JobRunStore interface plus a native-driver implementation whose claim is a single atomic findOneAndUpdate.

A JobRun is one graph run on one input — the durable queue work-unit. It is distinct from a graph task node and from an ai-task (@exellix/ai-tasks). See temp/jobs/ for the full design.

Install

npm install @exellix/jobs-db

Usage

import { createJobRunStore } from '@exellix/jobs-db';

const { store, close } = await createJobRunStore({
  mongoUri: process.env.MONGO_URI,   // or set MONGO_URI in the environment
  // runtimeDb: 'exellix-jobs',       // default
  // configDb:  'exellix',            // default
  // ensureIndexes: true,             // default
});

const run = await store.claim('worker-1');   // atomic claim, or null
await store.close();                          // via close()

For tests, use the in-memory implementation:

import { createMemoryJobRunStore } from '@exellix/jobs-db';
const store = createMemoryJobRunStore();

JobRunStore interface

interface JobRunStore {
  insertMany(runs: JobRun[]): Promise<void>;
  update(id: string, patch: Partial<JobRun>): Promise<void>;
  updateMany(filter: JobRunFilter, patch: Partial<JobRun>): Promise<{ modifiedCount: number }>;

  claim(workerId: string, opts?: ClaimOpts): Promise<JobRun | null>;   // atomic

  get(id: string): Promise<JobRun | null>;
  find(filter: JobRunFilter): Promise<JobRun[]>;
  count(filter: JobRunFilter): Promise<number>;
  hasItem(jobDefId: string, itemId: string): Promise<boolean>;          // dedup

  decrementPendingDeps(id: string): Promise<JobRun | null>;             // atomic
  close(): Promise<void>;
}

Collections

Collection	DB	Purpose
job_runs	runtime (exellix-jobs)	the work-unit; source of truth
job_defs	config (exellix)	one doc per JobDef
graphs	config (exellix)	graph authoring JSON / per-graph defaults
jobs_app_settings	runtime	singleton app settings (maxConcurrentJobs, admission interval)
job_run_events (optional)	runtime	thin append-only audit log

App settings (jobs_app_settings)

One document (id: 'default') caps how many job runs are admitted app-wide:

Field	Default	Role
maxConcurrentJobs	10	Max admitted runs (pending + running, claimable)
queueAdmissionIntervalMs	15000	How often held runs are promoted when capacity frees

Edited via jobs-ui → Settings (dashboard) or JobsAppSettingsStore.save(). See parallel execution for sizing guidance.

Job context sources (ContextSource)

Work definitions, on-demand runs, and sync graph execute can attach optional context sources — linked-object and same-object Memorix slices resolved before dispatch into jobMemory.context:

Type	Shape	Role
LinkedObjectContextSource	kind: 'linked', join fields, optional filter, mandatory	Resolve related records (e.g. subnets inferences joined on data.subnetIp)
SameObjectContextSource	kind: 'same-object', contentTypes[]	Load other content types from the same record id

Exported types: ContextSource, LinkedObjectContextSource, SameObjectContextSource, ContextFilter, ContextSourceQueryOptions (from context-sources.ts). Resolution logic lives in @exellix/jobs; the dispatcher receives pre-resolved jobMemory only.

The (jobDefId, itemId) index (by_job_item) is intentionally non-unique — one item produces one job run per graph. Dedup is enforced by hasItem at enqueue time, not by a unique constraint.

Why a separate package

@x12i/xronox-store has no atomic conditional update (findOneAndUpdate), which forced the old Execution Matrix into in-process withClaimLock. A durable cross-process queue needs structural atomicity, so the data tier uses the native mongodb driver and is kept out of the queue logic.

Scripts

npm run build      # tsc → dist/
npm test           # unit + live (live runs only when MONGO_URI is set)
npm run test:live  # live Mongo integration tests only

Live tests read MONGO_URI from the environment, falling back to ../../graph-packages/graph-engine/.env. Each run uses an isolated exellix-jobs-live-* database that is dropped on teardown.

License

exellix-license