@x12i/countex-pipelines
Generic pre-aggregated processing counters for records moving through pipelines.
Countex-pipelines tracks status transitions (in_progress → done | failed) and serves rollups by subject, pipeline, and dimension — sub-second when the hot layer is warm, with no source-document scans at query time.
Package policy: Generic only. No host-product types, client names, or embedded scan logic. The host supplies historical replay via CountexRebuildSource when needed.
Default deployment: single machine, single process
The simplest starting config needs one durable database — MongoDB or PostgreSQL — and no Redis.
export MONGO_URI="mongodb://127.0.0.1:27017"
# or, instead of Mongo:
# export COUNTEX_POSTGRES_URL="postgres://countex:countex@127.0.0.1:5432/countex"
# REDIS_URL is optional — omit it for single-machine mode
COUNTEX_POSTGRES_URL (or postgresUrl) selects Postgres. If MONGO_URI is already set, a generic POSTGRES_URL used by another database does not switch this package. See migrating from MongoDB.
import { createCountexFromEnv } from "@x12i/countex-pipelines";
const client = await createCountexFromEnv();
await client.initialize({ tenantId: "tenant-1", fromCounterStore: true });
await client.ingest({
tenantId: "tenant-1",
subject: "entities",
recordId: "rec-42",
pipelineId: "enrich",
status: "done",
occurredAt: new Date().toISOString(),
});
const rollup = await client.rollup({ tenantId: "tenant-1" });
console.log(rollup.bySubject, rollup.byPipeline);
await client.close();
When REDIS_URL is absent, countex-pipelines uses an in-process MemoryHotStore. The durable database remains the source of truth; on restart the hot layer is empty and is re-warmed from it automatically on initialize or first query. Do not run multiple processes with MemoryHotStore — use Redis for multi-server.
| Config | Hot layer | When to use |
|---|---|---|
MONGO_URI or COUNTEX_POSTGRES_URL only |
In-process memory | One machine, one process (default) |
Durable URL + REDIS_URL |
Shared Redis | Multiple servers, same counters |
COUNTEX_READ_THROUGH=1 |
None (read durable) | Minimal infra, low volume |
Postgres ingest runs dedup, the record write, and rollup increments in one transaction. A crash cannot leave a record applied without its counters.
Three consumption surfaces (choose one)
1. Primitives (library-first)
Import pure functions and wire storage yourself:
import {
classifyRecord,
classifyPipeline,
buildDedupKey,
MemoryHotStore,
MongoDurableStore,
PostgresPipelineStore,
PipelinesEngine,
} from "@x12i/countex-pipelines";
// or: import from "@x12i/countex-pipelines/primitives"
2. Embedded facade (in-process)
import { createCountexFromEnv } from "@x12i/countex-pipelines";
// or: import from "@x12i/countex-pipelines/client"
3. HTTP API (optional separate package)
Run as a standalone service for remote or non-Node consumers:
npm install @x12i/countex-pipelines-server
MONGO_URI=... countex-pipelines-server
See @x12i/countex-pipelines-server.
Ingest is write-through to the durable store — HTTP 200 means durably recorded.
Counter model
| Field | Role |
|---|---|
tenantId |
Isolation scope |
subject |
Host-defined subject type |
recordId |
Item being processed |
pipelineId |
Host-defined processor / workflow id |
dimensions |
Optional tags for rollup axes |
| Metric | succeeded | pending | failed | running (derived) |
Classification (per record)
| Bucket | Rule |
|---|---|
| succeeded | At least one pipeline done, and no pipeline failed |
| failed | At least one pipeline failed (wins over done) |
| pending | No terminal status yet |
| running | Per-pipeline: latest status is in_progress |
Starting from the middle
- Redis/memory empty, durable store has counters →
initialize({ tenantId, fromCounterStore: true })reloads and warms the hot layer. - Counters never existed, host has history → implement
CountexRebuildSource.scan()and callinitialize({ tenantId, fromSource })(in-process only). - Process restart → same as (1); the durable store is the source of truth.
Install
npm install @x12i/countex-pipelines
# mongodb and pg are dependencies of this package
# optional for multi-server:
npm install ioredis
Node 18+.
Docs
License
MIT