@x12i/memorix-pipeline

@x12i/memorix-pipeline

Durable pipeline DAG engine (memorix-pipeline/v2): captured plans, ordered step attempts, port registry (Memory, map, write, relationships, compose, enrich, …), retries, and run context.

Install

npm install @x12i/memorix-pipeline@3.0.2

Audience	Use this package?
Remote app	No — /api/pipelines
Platform / service	Yes

Exports

import {
  createPipelineRuntime,
  createLivePortRegistry,
  validatePipelineDefinition,
  createMongoRunLedger,
} from "@x12i/memorix-pipeline/v2";

Main entry also re-exports v2 surfaces from . where applicable; prefer ./v2 for new code.

Lifecycle

pipeline definition (metadata)
  → start run (capture plan + fingerprint)           # immediate execute
  → enqueue / lease / executeLeased                 # repeated-work queue
  → Work evaluate (selector → frozen batch → runs)   # continuous work
  → execute steps (attempts with input/output / DataRef)
  → mutating steps emit mutationRefs (persisted on attempt)
  → retry / resume

Repeated work (Jobs Manager foundation)

Studio authors pipelines. Runtime Work definitions (operational, not Magit metadata) evaluate selectors again and again and create ordinary PipelineRun rows with workId / batchId / itemRef / admission.

Concern	Mechanism
Continuous Work	PipelineWorkDefinition in pipeline-runtime stores
Admission	PipelineRun.admission (held | ready | leased) on the run ledger
Step leases	Existing step LeaseStore
HTTP product API	/api/jobs/v1 contract frozen (P1); handlers in P3

Do not introduce a parallel job_runs collection. One execution = one extended PipelineRun.

Pack schedules (design) ≠ Memory pull schedules ≠ Work cadence (runtime factory).

Step kinds include: memory-pull, map-normalize, validate, identity-dedupe, write-record, relationship-discover, relationship-write, compose, enrich, analyze, decide, workflow, run-pipeline (nested child run), runx, …

Step targets (composition)

Authoring steps may carry a unified target:

target.kind	Compiles to	Role
service	analyze / enrich / … + remote portConfig	Pack worker service
ai-service	analyze + ai-services/1	AI companion item
connector	memory-pull	Connector + source pull
pipeline	run-pipeline	Nested pipeline (child PipelineRun)
runx	runx	Runx capability (runx.run)

Nested pipelines use child runs with lineage (parentRunId, pipelineCallStack) and cycle/depth guards (default max depth 8).

Pipeline as a service

Route	Role
POST /api/pipelines/:pipelineId/invoke	Capability-shaped invoke
POST /api/capability/invoke	serviceId: "pipeline:{id}" (or bare installed id)
GET /api/pipelines/services	Virtual service catalog entries

Quick example

import { createPipelineRuntime, validatePipelineDefinition } from "@x12i/memorix-pipeline/v2";

validatePipelineDefinition(definition);
const rt = createPipelineRuntime({
  orgId,
  ledger,
  ports: createLivePortRegistry({ /* data, memory, relationships, mappings */ }),
});
const run = await rt.start({ agentIds, pipelineId, definition, input });
const attempts = await rt.getStepAttempts(run.runId);

HTTP

Route	Role
GET /api/pipelines/definitions	From effective metadata
GET /api/pipelines/services	Pipelines as virtual services
POST /api/pipelines/start	Start a run
POST /api/pipelines/:pipelineId/invoke	Capability-shaped invoke
POST /api/capability/invoke	pipeline:* service invoke
GET /api/pipelines/runs/:runId	Run + attempts
POST .../retry-step · .../resume	Recovery

Related

Package	Role
@x12i/memorix-pipeline-runtime	Artifacts / leases / DLQ helpers
@x12i/memorix-data-journey	Reconstructs the data cone from attempts + mutationRefs
@x12i/memorix-mapping	map-normalize execution

Docs

• Package docs under docs/
• docs/mutation-refs-capture.md
• Ebook: Building Pipelines
• Hub: https://docs.memorix.x12i.com

npm test