rest-pipeline-js
Flexible, modular pipeline orchestrator for REST APIs — sequential and parallel stages, retry with backoff, response caching, rate limiting, auth provider, stream stages (SSE / AsyncIterable), plugin system, and Vue / React integrations — all with a single dependency (axios).
Contents
- Features
- Installation
- Demo
- Examples
- Quick start
- createRestClient
- Custom cache backend (CacheStore)
- Distributed rate limiting (RateLimiterStore)
- Idempotency keys
- Request tracing
- Auth Provider
- Log Sanitization
- RequestExecutor
- Circuit breaker
- Distributed circuit breaker (CircuitBreakerStore)
- PipelineOrchestrator
- Error recovery (errorHandler + recoverStep)
- Parallel stages
- Global middleware
- Pause / Resume
- Export / Import state
- Pipeline metrics
- Correlating a run (runId)
- createPipeline() + pipe() builder
- validatePipelineConfig()
- Plugin system
- Persist adapter
- Stream stages
- HTTP Adapter
- Vue integration
- React integration
- Entry points
- Architecture
- Bundle size & peer dependencies
- Development
Features
createRestClient()— full-featured HTTP client built on top of axios: retry with exponential backoff andRetry-Aftersupport, response caching with a pluggableCacheStorebackend (incl. targetedinvalidateCache()), rate limiting (concurrency + req/interval) with a pluggable distributedRateLimiterStore, circuit breaker with a pluggable distributedCircuitBreakerStore, auth provider with automatic 401 refresh and optional token caching, request cancellation by key, custom HTTP adapters- Request tracing — W3C
traceparentheader generation plus aTracingProviderhook (duck-typed against OpenTelemetry'sSpanAPI) for wiring in a real tracing backend - Idempotency keys —
Idempotency-Keyheader on mutating requests, manual or auto-generated per logical request across retry attempts PipelineOrchestrator— sequential and parallel stage execution; each stage hascondition,before,request,after,errorHandlerhooks (all receive the pipeline'sAbortSignal);sharedDatapool shared across all stages- Error recovery —
errorHandlercan returnrecoverStep(data)to turn a failed stage back into a successful one and keep the pipeline going, instead of only transforming the error - Global middleware —
beforeEach/afterEach/onErrorhooks that apply to every stage without modifying individual configs - Parallel groups — multiple stages run concurrently via
Promise.all, or through a bounded pool viaconcurrency; single failure stops the group - Pause / Resume / Abort —
pause()waits after the current stage;resume()continues;abort()cancels the current HTTP request and propagates itsAbortSignalinto every stage hook so customrequest/before/afterfunctions can cancel their own work too - Export / Import state — serialize
stageResults+ logs to a plain object; restore on the next page load - Stream stages —
stream: async function*for SSE / anyAsyncIterable;onChunkcallback in real time; abort-aware - Pipeline metrics & run correlation —
onPipelineStart,onPipelineEnd,onStepDurationcallbacks, plus arunId(also ongetRunId(), log entries, and step events) shared by every callback/event from the same run createPipeline()/pipe()builder — short factory and fluent builder API for common patterns; in TypeScript,pipe().step()chains inferprev's type from the previous step automaticallyvalidatePipelineConfig()— catch duplicate keys, empty keys, type errors before runtime- Plugin system — install reusable behavior (logging, analytics, etc.); cleanup via
destroy() - Persist adapter — pluggable save/load interface; auto-save after each stage
- Log sanitization — mask sensitive headers (
authorization,x-api-key,cookie, …) in metrics callbacks, on by default - Vue integration —
usePipelineRunVue,usePipelineProgressVue, and more (import fromrest-pipeline-js/vue) - React integration —
usePipelineRunReact,usePipelineProgressReact, and more (import fromrest-pipeline-js/react) - Tree-shakeable —
sideEffects: false; Vue and React entry points are code-split
Installation
npm install rest-pipeline-js
Peer dependencies for framework integrations:
# Vue
npm install vue@>=3.3
# React
npm install react@>=19 react-dom@>=19
Demo
A multi-scenario interactive demo showcasing the key features of rest-pipeline-js. All demos use real public REST APIs.
git clone https://github.com/macrulezru/pipeline-js.git
cd pipeline-js
npm install
npm run demo:vue
Opens at http://localhost:3000. The demo app lives in the demo/ directory.
| Demo | What it shows |
|---|---|
| Flight Pipeline | 4-stage sequential pipeline with sharedData, pauseBefore/pauseAfter, middleware, boarding pass result |
| Parallel Loading | pipe() fluent builder with .parallel([]) — 3 sources queried simultaneously, timing breakdown |
| Retry & Recovery | Configurable flaky stage with exponential backoff, event log, abort(), pause/resume between stages |
| Cache & Rate Limit | createRestClient() with cache TTL — see server vs cache timing; rate limiter burst visualization |
| Idempotency & Tracing | A flaky mutation retried by RequestExecutor with autoIdempotencyKey — same key on every attempt; tracing.generateTraceparent correlating requests under one trace-id |
Examples
Focused, copy-pasteable snippets (as opposed to the interactive demo app above) live in examples/:
| File | Shows |
|---|---|
pagination-fanout.ts |
Fanning out many paginated requests in parallel with a concurrency cap |
edge-fetch-adapter.ts |
A custom HttpAdapter on native fetch for Cloudflare Workers / Deno / edge |
sse-stream.ts |
A StreamStageConfig step consuming a Server-Sent Events endpoint chunk by chunk |
redis-cache-store.ts |
A CacheStore backed by Redis, shared across multiple server instances |
redis-rate-limiter-store.ts |
A RateLimiterStore backed by Redis, shared across multiple server instances |
redis-circuit-breaker-store.ts |
A CircuitBreakerStore backed by Redis, shared across multiple server instances |
opentelemetry-tracing.ts |
W3C traceparent + a TracingProvider hook, correlated with a pipeline's runId |
idempotent-mutations.ts |
Idempotency-Key on mutating requests, manual or auto-generated across retries |
Quick start
import { createRestClient, PipelineOrchestrator } from "rest-pipeline-js";
// 1. Create a REST client
const client = createRestClient({
baseURL: "https://api.example.com",
retry: { attempts: 2, delayMs: 500, backoffMultiplier: 2 },
cache: { enabled: true, ttlMs: 60000 },
auth: {
getToken: async () => localStorage.getItem("token") ?? "",
onUnauthorized: async () => {
/* refresh token */
},
},
});
const res = await client.get("/users/1");
// 2. Run a pipeline
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
{
key: "fetchUser",
request: async ({ sharedData }) =>
client.get(`/users/${sharedData.userId}`),
},
{
key: "processData",
request: async ({ prev }) => ({ ...prev.data, processed: true }),
},
],
},
sharedData: { userId: 42 },
});
const result = await orchestrator.run();
console.log(result.success, result.stageResults);
createRestClient
createRestClient(config: HttpConfig): RestClient
Creates a REST client with advanced HTTP features.
Methods
| Method | Description |
|---|---|
get(url, config?) |
GET request |
post(url, data?, config?) |
POST request |
put(url, data?, config?) |
PUT request |
patch(url, data?, config?) |
PATCH request |
delete(url, config?) |
DELETE request |
request(url, config?) |
Generic request |
cancellableRequest(key, url, config?) |
Request cancellable by key |
cancelRequest(key) |
Cancel request by key |
clearCache() |
Clear this client's entire response cache (async) |
invalidateCache(matcher) |
Clear only cache entries whose URL matches matcher (substring, RegExp, or (info) => boolean); returns (Promise<number>) the number of entries removed |
getCircuitBreakerState() |
Promise<"closed" | "open" | "half-open" | null> — null if circuitBreaker isn't configured. Resolves synchronously (no real async work) unless circuitBreaker.store is set |
HttpConfig options
| Option | Description |
|---|---|
baseURL |
Base URL for all requests |
timeout |
Request timeout in ms |
headers |
Default headers |
withCredentials |
Include cookies |
retry.attempts |
Number of retry attempts |
retry.delayMs |
Base delay between retries in ms |
retry.backoffMultiplier |
Exponential backoff multiplier |
retry.retriableStatus |
HTTP status codes eligible for retry (e.g. [429, 500, 503]) |
retry.maxRetryAfterMs |
Max wait from Retry-After header in ms (default: 60000) |
cache.enabled |
Enable response caching for GET requests |
cache.ttlMs |
Cache TTL in ms |
cache.strategy |
"strict" (default) or "stale-while-revalidate" |
cache.staleMs |
Extra time after ttlMs a stale response may still be served (SWR strategy) |
cache.store |
Custom CacheStore backend (e.g. Redis) instead of the built-in in-memory TtlCache — see Custom cache backend |
rateLimit.maxConcurrent |
Max simultaneous requests |
rateLimit.maxRequestsPerInterval |
Max requests per time window |
rateLimit.intervalMs |
Time window size in ms |
rateLimit.store |
Custom RateLimiterStore backend (e.g. Redis) for a limit shared across server instances — see Distributed rate limiting |
rateLimit.key |
Bucket name when using a shared rateLimit.store (default: random per-instance id — without an explicit key, a store has no sharing effect) |
rateLimit.leaseMs |
Auto-expiry (ms) for a store-backed concurrency slot if its holder crashes without releasing (default: 30000) |
metrics.onRequestStart |
Callback on request start |
metrics.onRequestEnd |
Callback on request end (includes duration and bytes) |
auth.getToken |
Async function returning a Bearer token (called before every request, unless auth.tokenTtlMs is set) |
auth.onUnauthorized |
Optional async callback on 401 — refresh the token here; request is retried once |
auth.tokenTtlMs |
Cache getToken()'s result for this many ms instead of calling it before every request; invalidated automatically on 401 |
sanitizeHeaders |
Mask sensitive headers in metrics callbacks (default: true — secure by default) |
sensitiveHeaders |
Additional headers to mask (extends DEFAULT_SENSITIVE_HEADERS) |
adapter |
Custom HTTP adapter (e.g. native fetch) — replaces built-in axios |
circuitBreaker |
See Circuit breaker — { failureThreshold, openMs, successThreshold?, isFailure?, store?, key? } |
tracing.generateTraceparent |
Add a W3C traceparent header to every request (default: false) — see Request tracing |
tracing.provider |
TracingProvider hook creating a span per request — see Request tracing |
idempotencyHeaderName |
Header name used for RestRequestConfig.idempotencyKey (default: "Idempotency-Key") — see Idempotency keys |
autoIdempotencyKey |
Have RequestExecutor auto-generate an idempotency key per logical request (default: false) — see Idempotency keys |
Per-request cache override
const res = await client.get("/data", {
useCache: true,
cacheTtlMs: 30000,
cacheKey: "my-custom-key",
});
Targeted cache invalidation
clearCache() wipes the entire response cache. To invalidate only the entries affected by a mutation (e.g. after a POST/PUT/DELETE), use invalidateCache() instead — it accepts a substring, a RegExp, or a predicate over { method, url }, and resolves to how many entries were removed. Both methods are async (so a custom cache.store can be backed by a real network call):
await client.post("/users/1/orders", newOrder);
await client.invalidateCache("/users/1"); // substring match on the cached URL
await client.invalidateCache(/^https:\/\/api\.example\.com\/users\/\d+$/);
await client.invalidateCache(({ method, url }) => method === "GET" && url.includes("/orders"));
Custom cache backend (CacheStore)
By default, cache.enabled: true caches responses in an in-memory TtlCache scoped to that one client instance — fine for a browser SPA, but each server process has its own cold cache in a multi-instance deployment. Pass cache.store to use any backend implementing CacheStore instead — Redis, for example, so cached responses are shared across every instance:
import { createRestClient, type CacheStore, type ApiResponse } from "rest-pipeline-js";
const redisStore: CacheStore<ApiResponse<unknown>> = {
async get(key) {
const raw = await redis.get(key);
return raw ? JSON.parse(raw) : undefined;
},
async set(key, value, ttlMs) {
await redis.set(key, JSON.stringify(value), "PX", ttlMs);
},
async delete(key) { await redis.del(key); },
async clear() { await redis.flushdb(); },
// getStale/deleteWhere are optional — without them, the
// 'stale-while-revalidate' strategy and invalidateCache() gracefully
// degrade (see CacheStore's JSDoc) instead of throwing.
};
const client = createRestClient({
baseURL: "https://api.example.com",
cache: { enabled: true, ttlMs: 60_000, store: redisStore },
});
See examples/redis-cache-store.ts for the full annotated version.
Full example
import { createRestClient } from "rest-pipeline-js";
const client = createRestClient({
baseURL: "https://api.example.com",
timeout: 5000,
retry: {
attempts: 2,
delayMs: 500,
backoffMultiplier: 2,
retriableStatus: [429, 500, 503],
},
cache: { enabled: true, ttlMs: 60000 },
rateLimit: { maxConcurrent: 3, maxRequestsPerInterval: 10, intervalMs: 1000 },
auth: {
getToken: async () => localStorage.getItem("token") ?? "",
onUnauthorized: async () => {
/* refresh token here */
},
},
sanitizeHeaders: true,
});
const res = await client.get("/users/1");
console.log(res.data);
// PATCH support
await client.patch("/users/1", { name: "Alice" });
// Cancellable request
const req = client.cancellableRequest("my-key", "/search", {
params: { q: "foo" },
});
// Cancel it any time:
client.cancelRequest("my-key");
Distributed rate limiting (RateLimiterStore)
By default, rateLimit is enforced in-memory, scoped to that one client instance — fine for a browser SPA, but each server process enforces its own limit in a multi-instance deployment, so N instances effectively allow N× the configured limit. Pass rateLimit.store to share the limit across instances (e.g. via Redis):
import { createRestClient, type RateLimiterStore } from "rest-pipeline-js";
const redisRateLimiterStore: RateLimiterStore = {
async incrementWindow(key, intervalMs) {
const count = await redis.incr(key);
if (count === 1) await redis.pexpire(key, intervalMs);
return count;
},
async acquireConcurrencySlot(key, maxConcurrent, leaseMs) {
// Needs an atomic increment-if-below-cap (typically a small Lua script) —
// see examples/redis-rate-limiter-store.ts for a full sketch.
// ...
},
};
const client = createRestClient({
baseURL: "https://api.example.com",
rateLimit: {
maxRequestsPerInterval: 100,
intervalMs: 60_000,
store: redisRateLimiterStore,
key: "api-example-com", // shared bucket name across every instance
},
});
Notes:
- Without an explicit
key, everyRateLimiterinstance gets its own random key — astoreonly has a sharing effect once multiple limiters (across processes) use the samekey. incrementWindowis a fixed-window counter — it has the standard edge-of-window burst characteristic of any fixed-window rate limiter (as opposed to a sliding log). This is a deliberate simplicity trade-off; implement a sliding-window store yourself if you need stricter bounds.acquireConcurrencySlot(maxConcurrent) cannot be made exactly correct across processes without a central lock service — treat it as an approximate cap, the way most distributed semaphores work in practice.leaseMsbounds how long a slot is held if its holder crashes without releasing.
See examples/redis-rate-limiter-store.ts for the full annotated version.
Idempotency keys
Send an Idempotency-Key header on mutating requests (POST/PUT/PATCH/DELETE) so a backend that supports idempotency keys (Stripe, PayPal, and plenty of in-house APIs) can safely dedupe retried requests instead of double-applying them. The library only sends the header — deduplication is the backend's job.
// Manual: generate the key once per logical operation, reuse across attempts
const idempotencyKey = crypto.randomUUID();
await client.post("/orders", cart, { idempotencyKey });
// Custom header name
const client = createRestClient({ idempotencyHeaderName: "X-Idempotency-Key" });
RequestExecutor (the class that actually implements retry — see RequestExecutor; createRestClient()'s own client.post()/etc. don't retry on their own) can generate the key for you automatically:
import { RequestExecutor } from "rest-pipeline-js";
const executor = new RequestExecutor({
baseURL: "https://api.example.com",
autoIdempotencyKey: true, // generates one key per logical request, reused across every retry attempt
retry: { attempts: 2, delayMs: 300, backoffMultiplier: 2 },
});
await executor.execute("/orders", { method: "POST", data: { items: ["sku-1"] } });
autoIdempotencyKey only affects mutating methods (POST/PUT/PATCH/DELETE) and only generates a key if the caller didn't already provide one via idempotencyKey. See examples/idempotent-mutations.ts.
Request tracing
Two independent features:
tracing.generateTraceparent adds a W3C Trace Context traceparent header to every request (skipped if the request already sets one explicitly), so any backend/APM that understands trace context can correlate the call with the rest of a distributed trace:
const client = createRestClient({
baseURL: "https://api.example.com",
tracing: { generateTraceparent: true },
});
Pass traceId on a request to correlate multiple calls under one trace instead of a fresh random one each time — a pipeline's runId (UUID) with its dashes stripped is exactly the 32 hex characters the format needs:
await client.get("/users/1", { traceId: orchestrator.getRunId().replace(/-/g, "") });
tracing.provider wraps every request in a real span in your tracing system. Its shape (TracingProvider/TracingSpan) is deliberately a subset of OpenTelemetry's Span API (duck-typed — this package doesn't depend on @opentelemetry/api), so a real OTel SDK plugs in as a thin adapter:
import { trace } from "@opentelemetry/api";
import { createRestClient, type TracingProvider } from "rest-pipeline-js";
const tracer = trace.getTracer("my-app");
const otelProvider: TracingProvider = {
startSpan: (name, attributes) => tracer.startSpan(name, { attributes }),
};
const client = createRestClient({
baseURL: "https://api.example.com",
tracing: { generateTraceparent: true, provider: otelProvider },
});
startSpan(name, attributes) is called before each request; span.end() after; span.setStatus()/span.recordException() on error (both optional on TracingSpan — a minimal provider only needs end()). See examples/opentelemetry-tracing.ts for the full annotated version, including a dependency-free console-logging provider.
Auth Provider
Automatically inject an Authorization: Bearer <token> header before every request. On a 401 response, onUnauthorized is called (e.g. to refresh the token) and the request is retried once — preventing infinite loops.
const client = createRestClient({
baseURL: "https://api.example.com",
auth: {
getToken: async () => {
return localStorage.getItem("access_token") ?? "";
},
onUnauthorized: async () => {
const newToken = await refreshAccessToken();
localStorage.setItem("access_token", newToken);
},
},
});
// Authorization: Bearer <token> is added automatically to every request
const res = await client.get("/profile");
Caching the token
If getToken() is expensive (e.g. it talks to a secure storage or refresh endpoint), set tokenTtlMs to reuse the result across requests instead of calling getToken() before every single one. The cache is invalidated automatically on a 401, so the next request always re-fetches a fresh token before retrying:
const client = createRestClient({
baseURL: "https://api.example.com",
auth: {
getToken: async () => requestTokenFromSecureEnclave(), // expensive
onUnauthorized: async () => refreshAccessToken(),
tokenTtlMs: 5 * 60_000, // reuse for up to 5 minutes
},
});
Log Sanitization
Mask sensitive headers in metrics callbacks (onRequestStart / onRequestEnd) so they never appear in logs.
import { createRestClient, DEFAULT_SENSITIVE_HEADERS } from "rest-pipeline-js";
// DEFAULT_SENSITIVE_HEADERS includes: authorization, x-api-key, x-auth-token,
// cookie, set-cookie, proxy-authorization
const client = createRestClient({
baseURL: "https://api.example.com",
// sanitizeHeaders defaults to true — masking is on unless you opt out below.
sensitiveHeaders: ["x-internal-secret"], // extend the default list
metrics: {
onRequestStart: (info) => {
// info.requestHeaders — sensitive values replaced with "REDACTED"
console.log(info.requestHeaders);
},
},
});
// To see raw headers (e.g. local debugging only), opt out explicitly:
// createRestClient({ ..., sanitizeHeaders: false });
Use sanitizeHeadersMap directly:
import { sanitizeHeadersMap } from "rest-pipeline-js";
const safe = sanitizeHeadersMap(
{ authorization: "Bearer abc", "content-type": "application/json" },
["x-custom-secret"],
);
// { authorization: "REDACTED", "content-type": "application/json" }
RequestExecutor
Wrapper for REST requests with retry, timeout (via AbortController), Retry-After header support, and backoff.
import { RequestExecutor } from "rest-pipeline-js";
const executor = new RequestExecutor({
baseURL: "https://api.example.com",
retry: {
attempts: 3,
delayMs: 500,
backoffMultiplier: 2,
retriableStatus: [429, 500, 502, 503],
maxRetryAfterMs: 30000, // cap Retry-After at 30 s
},
});
// 5th arg: external AbortSignal (e.g. from orchestrator.abort())
const res = await executor.execute("/data", undefined, 3, 5000, signal);
When the server returns a Retry-After header (numeric seconds or HTTP-date), that delay takes priority over the backoff formula. Values exceeding maxRetryAfterMs are clamped to the cap. Timeout is enforced via AbortController — the actual HTTP request is cancelled, not just the promise.
Circuit breaker
Protect a failing backend (and your own app) from piling up retries/timeouts: after failureThreshold consecutive failures, the client stops calling the network entirely for openMs and rejects requests immediately with a CircuitOpenError (code: "CIRCUIT_OPEN"). After openMs, it lets a probe request through (half-open); success closes the circuit again, failure re-opens it.
import { createRestClient, CircuitOpenError } from "rest-pipeline-js";
const client = createRestClient({
baseURL: "https://api.example.com",
circuitBreaker: {
failureThreshold: 5, // open after 5 consecutive failures
openMs: 30_000, // stay open for 30s before probing again
successThreshold: 2, // need 2 successful probes to fully close
isFailure: (error) => error.status === undefined || error.status >= 500, // ignore 4xx
},
});
try {
await client.get("/flaky-endpoint");
} catch (err) {
if (err instanceof CircuitOpenError) {
// rejected locally — no network call was made
}
}
await client.getCircuitBreakerState(); // "closed" | "open" | "half-open" — async
- Works on top of retry, cache, rate limiting, auth, and custom
adapters — it sits around the actual network call, same as those features. - Each retry attempt (from
RequestExecutor/request.retry) counts as its own pass through the breaker, so a flaky endpoint with retries enabled opens the circuit faster, not slower. - Cancelled/aborted requests are never counted as failures.
- Not set by default — without
circuitBreaker, behavior is unchanged. getCircuitBreakerState()(and everyCircuitBreakermethod) isasync— it resolves synchronously (no real async work) unlesscircuitBreaker.storeis set (see below).
Distributed circuit breaker (CircuitBreakerStore)
By default, circuit breaker state lives in-memory, scoped to that one client instance — each server process needs its own failureThreshold consecutive failures before it opens, so a struggling backend in a multi-instance deployment absorbs N× as many failures as configured before anything trips. Pass circuitBreaker.store to share open/closed/half-open state across instances:
import { createRestClient, type CircuitBreakerStore } from "rest-pipeline-js";
const redisCircuitBreakerStore: CircuitBreakerStore = {
async get(key) {
const raw = await redis.get(key);
return raw ? JSON.parse(raw) : null;
},
async set(key, state, ttlMs) {
await redis.set(key, JSON.stringify(state), "PX", ttlMs);
},
// Optional: atomic increment, avoids a get+compute+set race between
// concurrent requests on different instances.
async incrementCounter(key, field, ttlMs) {
const n = await redis.incr(`${key}:${field}`);
if (n === 1) await redis.pexpire(`${key}:${field}`, ttlMs);
return n;
},
};
const client = createRestClient({
baseURL: "https://api.example.com",
circuitBreaker: {
failureThreshold: 5,
openMs: 30_000,
store: redisCircuitBreakerStore,
key: "api-example-com", // shared bucket name across every instance
},
});
As with rateLimit.key, an explicit key is what makes multiple CircuitBreaker instances (across processes) actually share state — without it, each gets its own random key. Without incrementCounter, the breaker falls back to get-compute-set, which can under-count failures under heavy concurrent load across instances but remains fail-safe. See examples/redis-circuit-breaker-store.ts for the full annotated version.
PipelineOrchestrator
Main class for building and managing a pipeline of sequential (and parallel) stages.
Constructor
new PipelineOrchestrator({
config, // PipelineConfig — stages and optional middleware
httpConfig?, // HttpConfig — HTTP client settings
sharedData?, // Record<string, any> — shared pool across all stages
options?, // { autoReset?: boolean }
})
Methods
| Method | Description |
|---|---|
run(onStepPause?, externalSignal?) |
Execute all stages. Returns { stageResults, success } |
rerunStep(stepKey, options?) |
Re-execute a single stage (respects condition, before, after, middleware) |
abort() |
Abort pipeline execution (cancels the current HTTP request via AbortSignal) |
isAborted() |
Check if pipeline was aborted |
pause() |
Pause after the current stage completes |
resume() |
Resume a paused pipeline |
isPaused() |
Check if pipeline is paused |
exportState() |
Serialize stageResults and logs to a plain object |
importState(state) |
Restore stageResults and logs from a snapshot |
getStageResults() |
Synchronous snapshot of all stage results |
getRunId() |
ID of the current/last run() or rerunStep() — see Correlating a run |
destroy() |
Run cleanup callbacks from all installed plugins |
subscribeProgress(listener) |
Subscribe to progress updates |
subscribeStageResults(listener) |
Subscribe to stageResults changes |
subscribeStepProgress(stepKey, listener) |
Subscribe to a specific stage's progress |
on(eventName, handler) |
Subscribe to any event (step:<key>:start|success|error|skipped|progress, log) |
onStepStart/Finish/Error(handler) |
Subscribe to stage lifecycle events |
getProgress() |
Get current progress snapshot |
getLogs() |
Get all pipeline logs (capped at options.maxLogs entries, if set — see below) |
clearStageResults() |
Reset results and progress |
Stage parameters (PipelineStageConfig)
Every hook below also receives signal: AbortSignal in its params object — the same signal used by orchestrator.abort(). Pass it down to fetch/axios/etc. inside request/before/after so cancellation actually stops in-flight work, not just the pipeline's bookkeeping.
| Parameter | Description |
|---|---|
key |
Unique stage identifier |
request({ prev, allResults, sharedData, signal }) |
Main stage function — return value becomes the stage result |
condition({ prev, allResults, sharedData, signal }) |
If returns false, stage is skipped with status "skipped" |
before({ prev, allResults, sharedData, signal }) |
Pre-processing hook — returned value replaces prev passed to request |
after({ result, allResults, sharedData, signal }) |
Post-processing hook — returned value replaces the stage result |
errorHandler({ error, key, sharedData, signal }) |
Per-stage error handler — see Error recovery below |
retryCount |
Override retry count for this stage |
timeoutMs |
Override timeout for this stage |
pauseBefore |
Delay in ms before executing request |
pauseAfter |
Delay in ms after executing request |
Error recovery (errorHandler + recoverStep)
By default, whatever errorHandler returns is wrapped into an ApiError and the stage stays "error" — it can transform/enrich the error but not turn the failure into a success. Return recoverStep(data) to recover the stage instead: it's committed exactly like a successful stage (status "success", afterEach middleware, metrics, persistence) and the pipeline continues normally.
import { recoverStep } from "rest-pipeline-js";
{
key: "fetchPrice",
request: async () => fetchPriceFromApi(),
errorHandler: ({ error }) => {
if (isNetworkError(error)) return recoverStep(0); // fall back to a default and continue
return error; // anything else: keep failing as before
},
}
Stage execution flow
condition? → false → [status: skipped] → next stage
↓ true
middleware.beforeEach
↓
pauseBefore
↓
before() hook
↓
request()
↓
after() hook
↓
pauseAfter
↓
middleware.afterEach
↓
[status: success] → next stage
On error at any point:
└─► stage.errorHandler (if set)
├─► returns recoverStep(data) → [status: success] → next stage
└─► otherwise → middleware.onError → [status: error] → stop
Full example
import { PipelineOrchestrator } from "rest-pipeline-js";
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
{
key: "fetchUser",
request: async ({ sharedData }) => {
const res = await fetch(`/api/users/${sharedData.userId}`);
return res.json();
},
},
{
key: "processData",
condition: ({ prev }) => prev !== null,
before: ({ prev }) => ({ ...prev, processed: true }),
request: async ({ prev }) => prev,
after: ({ result }) => ({ ...result, finishedAt: Date.now() }),
},
],
middleware: {
beforeEach: ({ stage }) => console.log("Starting:", stage.key),
afterEach: ({ stage, result }) =>
console.log("Done:", stage.key, result.data),
onError: ({ stage, error }) =>
console.error("Error in", stage.key, error),
},
},
httpConfig: {
baseURL: "https://api.example.com",
retry: { attempts: 2, delayMs: 1000, backoffMultiplier: 2 },
cache: { enabled: true, ttlMs: 60000 },
},
sharedData: { userId: 42 },
options: { autoReset: true },
});
orchestrator.subscribeProgress((progress) => {
console.log(
"Stage:",
progress.currentStage,
"Statuses:",
progress.stageStatuses,
);
});
orchestrator.on("step:fetchUser:success", (payload) => {
console.log("fetchUser done:", payload.data);
});
const result = await orchestrator.run();
console.log("Pipeline finished:", result.success);
console.log("Stage results:", result.stageResults);
Parallel stages
Group stages for concurrent execution using parallel:
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
// Sequential stage
{ key: "auth", request: async () => getToken() },
// Parallel group — all run concurrently
{
key: "load-data",
parallel: [
{ key: "loadUsers", request: async () => fetchUsers() },
{ key: "loadProducts", request: async () => fetchProducts() },
{ key: "loadSettings", request: async () => fetchSettings() },
],
},
// Sequential stage after the group
{ key: "render", request: async ({ allResults }) => render(allResults) },
],
},
});
- All stages in a
parallelgroup run simultaneously viaPromise.all— unlessconcurrencyis set (see below). - If any stage in the group fails, the pipeline stops and marks
success: false. - Each parallel stage has its own key and result in
stageResults. rerunStep(key)works for stages inside parallel groups too.
Limiting concurrency
For fan-out over many items (e.g. paginated fetches), set concurrency on the group to cap how many stages run at once instead of starting all of them immediately:
{
key: "fetch-all-pages",
parallel: pageNumbers.map((n) => ({
key: `page-${n}`,
request: async () => fetchPage(n),
})),
concurrency: 5, // at most 5 requests in flight at a time
}
Results land in stageResults under their own key regardless of concurrency, in the same shape as an unlimited group. With the pipe() builder: .parallel(stages, { concurrency: 5 }).
Global middleware
Apply hooks to every stage without modifying individual stage configs:
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
/* ... */
],
middleware: {
beforeEach: async ({ stage, index, sharedData }) => {
console.log(`[${index}] Starting: ${stage.key}`);
sharedData.startedAt = Date.now();
},
afterEach: async ({ stage, index, result, sharedData }) => {
const ms = Date.now() - sharedData.startedAt;
console.log(`[${index}] Done: ${stage.key} in ${ms}ms`, result.data);
},
onError: async ({ stage, error, sharedData }) => {
await reportError({ stage: stage.key, error, context: sharedData });
},
},
},
});
Middleware runs in addition to (not instead of) per-stage errorHandler.
Pause / Resume
Pause the pipeline after a stage and resume later:
const orchestrator = new PipelineOrchestrator({ config });
// Pause after step1 completes
orchestrator.on("step:step1:success", () => orchestrator.pause());
const runPromise = orchestrator.run();
// At some point later (e.g. after user confirmation):
await showConfirmDialog();
orchestrator.resume();
await runPromise;
pause()— pipeline waits after the current stage finishes (including events).resume()— continues from the next stage.abort()while paused unblocks the pipeline and terminates it.
Export / Import state
Save and restore the pipeline state across page reloads or sessions:
const orchestrator = new PipelineOrchestrator({ config });
await orchestrator.run();
// Save state
const snapshot = orchestrator.exportState();
localStorage.setItem("pipelineState", JSON.stringify(snapshot));
// Later — restore and inspect without re-running
const saved = JSON.parse(localStorage.getItem("pipelineState"));
const orchestrator2 = new PipelineOrchestrator({ config });
orchestrator2.importState(saved);
console.log(orchestrator2.getProgress()); // restored progress
console.log(orchestrator2.getLogs()); // restored logs (timestamps as Date objects)
exportState() returns { stageResults, logs } — a plain JSON-serializable object. Timestamps in logs are stored as ISO strings and restored as Date objects on importState.
Capping log growth (maxLogs)
logs grows by one entry per step event and is never trimmed automatically — fine for a single run(), but an orchestrator instance reused across many runs without autoReset (e.g. a long-lived SPA singleton) accumulates logs indefinitely. Set options.maxLogs to keep only the N most recent entries (oldest evicted first):
const orchestrator = new PipelineOrchestrator({
config: {
stages: [/* ... */],
options: { maxLogs: 500 },
},
});
Without maxLogs, behavior is unchanged from previous versions.
Pipeline metrics
Observe pipeline execution without modifying stage logic:
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
/* ... */
],
metrics: {
onPipelineStart: ({ timestamp, runId }) => {
console.log(`[${runId}] Pipeline started at`, new Date(timestamp).toISOString());
},
onPipelineEnd: ({ durationMs, success, stageResults, runId }) => {
analytics.track("pipeline_complete", { durationMs, success, runId });
},
onStepDuration: ({ stepKey, durationMs, status, runId }) => {
console.log(`[${runId}] [${stepKey}] ${status} in ${durationMs}ms`);
},
},
},
});
| Callback | Receives | Description |
|---|---|---|
onPipelineStart |
{ timestamp, runId } |
Fires at the beginning of run() |
onPipelineEnd |
{ durationMs, success, stageResults, runId } |
Fires when run() completes |
onStepDuration |
{ stepKey, durationMs, status, runId } |
Fires after every executed step |
Correlating a run (runId)
Every run() call generates a fresh runId (a UUID, or a timestamp-based fallback in environments without crypto.randomUUID), shared by all metrics callbacks, log entries (getLogs()), and step events (PipelineStepEvent.runId) produced during that run — including all attempts of pipelineRetry. rerunStep() generates its own separate runId. Use orchestrator.getRunId() to read the current/last one, or read runId off any event/log/metrics callback to correlate everything that happened during one execution in your logging/tracing backend:
orchestrator.on("log", (entry) => sendToLogBackend({ ...entry, runId: orchestrator.getRunId() }));
createPipeline() + pipe() builder
createPipeline() — short factory
import { createPipeline } from "rest-pipeline-js";
const orchestrator = createPipeline(
[
{ key: "fetchUser", request: async () => fetchUser() },
{ key: "process", request: async ({ prev }) => process(prev) },
],
{
httpConfig: { baseURL: "https://api.example.com" },
sharedData: { userId: 42 },
pipelineOptions: { continueOnError: false },
metrics: {
onStepDuration: ({ stepKey, durationMs }) =>
console.log(stepKey, durationMs),
},
},
);
pipe() — fluent builder
import { pipe } from "rest-pipeline-js";
const orchestrator = pipe()
.step({ key: "auth", request: async () => getToken() })
.step({ key: "fetchUser", request: async ({ prev }) => fetchUser(prev) })
.parallel([
{ key: "loadPosts", request: async () => fetchPosts() },
{ key: "loadNotifs", request: async () => fetchNotifications() },
])
.stream({
key: "liveUpdates",
stream: async function* () {
yield* subscribe("/events");
},
onChunk: (chunk) => updateUI(chunk),
})
.build({ httpConfig: { baseURL: "https://api.example.com" } });
| Builder method | Description |
|---|---|
.step(stage) |
Add a sequential stage |
.parallel(stages, options?) |
Add a parallel group (key/concurrency optional, see Limiting concurrency) |
.subPipeline(item) |
Embed a sub-pipeline as a stage |
.stream(stage) |
Add a stream stage (AsyncIterable) |
.build(options?) |
Create and return a PipelineOrchestrator |
.toConfig(options?) |
Return PipelineConfig without creating an orchestrator |
Typed chaining (TypeScript)
In TypeScript, pipe().step(...) tracks the type of prev across the chain: each .step()'s prev is typed as the previous step's return value (undefined for the very first step, matching the orchestrator's actual runtime behavior). .parallel() / .subPipeline() / .stream() don't change it — exactly like at runtime, where prev for the next step still comes from the last regular .step(), not from a parallel group's results:
const orchestrator = pipe()
.step({ key: "auth", request: async (): Promise<string> => getToken() })
.step({ key: "fetchUser", request: async ({ prev }) => fetchUser(prev) }) // prev: string — inferred, autocompletes
.step({ key: "oops", request: async ({ prev }) => prev.totallyNotAMethod() }) // ✗ compile error: wrong type for prev
.build();
This works whether or not you keep reassigning the chain (builder.step(...) without capturing the return value still mutates the same instance, just like before) — the typing is purely additive and doesn't change runtime behavior.
validatePipelineConfig()
Catch configuration errors before runtime:
import { validatePipelineConfig } from "rest-pipeline-js";
const { valid, errors } = validatePipelineConfig({
stages: [
{ key: "step1", request: async () => data },
{ key: "step1", request: async () => other }, // duplicate!
{ key: "", request: async () => other }, // empty key!
],
});
if (!valid) console.error(errors);
// ["[root] duplicate stage key: "step1"", "[root] stage key must be a non-empty string"]
Validates: duplicate keys, empty/invalid keys, empty stages array, invalid field types (request, condition, retryCount, timeoutMs), and recursively validates nested subPipeline configs.
Plugin system
Package reusable orchestrator behavior into plugins:
const loggingPlugin = {
name: "logging",
install(orchestrator) {
const off = orchestrator.on("log", (event) => {
if (event.type === "step:success") console.log("✓", event.stepKey);
if (event.type === "step:error")
console.error("✗", event.stepKey, event.error);
});
return () => off(); // cleanup on orchestrator.destroy()
},
};
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
/* ... */
],
options: { plugins: [loggingPlugin, analyticsPlugin] },
},
});
// Call when the orchestrator is no longer needed:
orchestrator.destroy();
install(orchestrator)— receives the orchestrator instance; may subscribe to events, set up middleware, etc.- If
installreturns a function, it is registered as a cleanup callback and invoked bydestroy().
Persist adapter
Automatically save and restore pipeline state across page reloads:
const localStorageAdapter = {
save: (state) => localStorage.setItem("pipeline", JSON.stringify(state)),
load: () => {
const raw = localStorage.getItem("pipeline");
return raw ? JSON.parse(raw) : null;
},
};
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
/* ... */
],
options: { persistAdapter: localStorageAdapter },
},
});
// run() loads saved state at start; saves after each completed step
await orchestrator.run();
The adapter interface:
type PipelineStateAdapter = {
save(state: PipelineExportedState): void | Promise<void>;
load(): PipelineExportedState | null | Promise<PipelineExportedState | null>;
};
Both methods may be async (useful for IndexedDB or remote storage).
Stream stages (SSE / AsyncIterable)
A stage whose stream function returns an AsyncIterable<T>. The orchestrator collects all emitted chunks into an array (the stage result). onChunk is called for each chunk in real time.
const orchestrator = createPipeline([
{ key: "auth", request: async () => getToken() },
{
key: "liveData",
stream: async function* ({ prev }) {
const source = new EventSource(`/api/stream?token=${prev}`);
yield* eventSourceToAsyncIterable(source);
},
onChunk: (chunk, sharedData) => {
sharedData.partial = (sharedData.partial ?? "") + chunk;
updateUI(sharedData.partial);
},
},
{
key: "finalize",
// allResults.liveData.data is the full array of chunks
request: async ({ allResults }) => allResults.liveData.data.join(""),
},
]);
- Respects
abort()— checks the abort signal between each chunk. - Supports
continueOnError— failed stream stages can be skipped like any other step. - Emits standard step events:
step:start,step:success,step:error.
HTTP Adapter (custom fetch / edge environments)
Replace the built-in axios client with any HTTP implementation:
const fetchAdapter = {
async request(config) {
const url = `${config.baseURL ?? ""}${config.url ?? ""}`;
const res = await fetch(url, {
method: config.method ?? "GET",
body: config.data ? JSON.stringify(config.data) : undefined,
headers: { "Content-Type": "application/json", ...config.headers },
signal: config.signal,
});
const data = await res.json();
return {
data,
status: res.status,
statusText: res.statusText,
headers: Object.fromEntries(res.headers.entries()),
};
},
};
const client = createRestClient({
baseURL: "https://api.example.com",
adapter: fetchAdapter,
// Auth, interceptors, sanitizeHeaders, metrics still work on top of the adapter
auth: { getToken: async () => token },
});
type HttpAdapter = {
request<T = unknown>(
config: RestRequestConfig & { baseURL?: string },
): Promise<ApiResponse<T>>;
};
When adapter is set, createRestClient() never calls axios.create() — the built-in axios instance simply isn't constructed, so adapter-only usage (e.g. in Cloudflare Workers / Deno) doesn't pay for it.
Vue integration
<script setup>
import {
PipelineOrchestrator,
usePipelineProgressVue,
usePipelineRunVue,
} from "rest-pipeline-js/vue";
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
/* ... */
],
},
});
const progress = usePipelineProgressVue(orchestrator);
const { run, running, result, error, abort, pause, resume, rerunStep } =
usePipelineRunVue(orchestrator);
</script>
<template>
<div>
<div>Current stage: {{ progress.currentStage }}</div>
<button @click="run()" :disabled="running">Start</button>
<button @click="abort()" :disabled="!running">Abort</button>
<button @click="pause()">Pause</button>
<button @click="resume()">Resume</button>
<div v-if="result">Done: {{ result }}</div>
<div v-if="error">Error: {{ error.message }}</div>
</div>
</template>
Composables (import from rest-pipeline-js/vue):
| Composable | Returns | Description |
|---|---|---|
usePipelineProgressVue(orchestrator) |
Ref<PipelineProgress> |
Reactive progress |
usePipelineRunVue(orchestrator) |
{ run, running, result, error, stageResults, abort, pause, resume, rerunStep, clearStageResults } |
Run pipeline and get reactive state |
usePipelineStepEventVue(orchestrator, stepKey, eventType) |
Ref<any> |
Last payload for a specific step event |
usePipelineLogsVue(orchestrator) |
Ref<log[]> |
Reactive logs |
useRerunPipelineStepVue(orchestrator) |
function |
Bound rerunStep |
useRestClientVue(config) |
ComputedRef<RestClient> |
Reactive REST client |
usePipelineStageResultVue(orchestrator, stepKey) |
Ref<PipelineStepResult | null> |
Reactive result of a single stage |
React integration
import { useRef } from "react";
import {
PipelineOrchestrator,
usePipelineProgressReact,
usePipelineRunReact,
} from "rest-pipeline-js/react";
const orchestrator = new PipelineOrchestrator({
config: {
stages: [
/* ... */
],
},
});
export function PipelineComponent() {
const progress = usePipelineProgressReact(orchestrator);
const [run, { running, result, error, abort, pause, resume, rerunStep, clearStageResults }] =
usePipelineRunReact(orchestrator);
return (
<div>
<div>Current stage: {progress.currentStage}</div>
<button onClick={() => run()} disabled={running}>
Start
</button>
<button onClick={() => abort()} disabled={!running}>
Abort
</button>
<button onClick={() => pause()}>Pause</button>
<button onClick={() => resume()}>Resume</button>
{result && <div>Done: {JSON.stringify(result)}</div>}
{error && <div>Error: {error.message}</div>}
</div>
);
}
Hooks (import from rest-pipeline-js/react):
| Hook | Returns | Description |
|---|---|---|
usePipelineProgressReact(orchestrator) |
PipelineProgress |
Reactive progress |
usePipelineRunReact(orchestrator) |
[run, { running, result, error, stageResults, abort, pause, resume, rerunStep, clearStageResults }] |
Run pipeline and get state |
usePipelineStepEventReact(orchestrator, stepKey, eventType) |
any |
Last payload for a specific step event |
usePipelineLogsReact(orchestrator) |
log[] |
Reactive logs |
useRerunPipelineStepReact(orchestrator) |
function |
Bound rerunStep |
useRestClientReact(config) |
RestClient |
Memoized REST client — recreated when config is a new object reference; memoize it yourself (useMemo/useState/module-level constant) to avoid recreating it every render |
usePipelineStageResultReact(orchestrator, stepKey) |
PipelineStepResult | null |
Result of a single stage |
Entry points
| Entry point | Use for | Contents |
|---|---|---|
rest-pipeline-js |
Core only | PipelineOrchestrator, createRestClient, types, utilities. No Vue/React. |
rest-pipeline-js/vue |
Vue projects | Core + Vue composables |
rest-pipeline-js/react |
React projects | Core + React hooks |
// Core only
import { createRestClient, PipelineOrchestrator } from "rest-pipeline-js";
// Vue
import { PipelineOrchestrator, usePipelineRunVue } from "rest-pipeline-js/vue";
// React
import {
PipelineOrchestrator,
usePipelineRunReact,
} from "rest-pipeline-js/react";
sideEffects: false — unused entry points are tree-shaken. react / react-dom are peerDependencies.
Architecture
rest-pipeline-js
│
├── createRestClient (HttpConfig) → RestClient
│ ├── RequestExecutor — retry + backoff + Retry-After + AbortController timeout
│ ├── CacheManager — in-memory TTL cache for GET responses
│ ├── RateLimiter — concurrency + req/interval sliding window
│ ├── CircuitBreaker — closed → open → half-open; rejects locally when open
│ ├── AuthProvider — Bearer injection; 401 refresh + one retry; optional tokenTtlMs cache
│ ├── MetricsCollector — onRequestStart / onRequestEnd callbacks
│ ├── HeaderSanitizer — masks sensitive headers before metrics callbacks
│ └── HttpAdapter — pluggable transport (default: axios; swap for fetch / edge)
│
├── PipelineOrchestrator (config, httpConfig?, sharedData?, options?)
│ ├── StageRunner — sequential execution loop; parallel via Promise.all
│ │ condition → pauseBefore → before → request → after → pauseAfter
│ ├── MiddlewareRunner — beforeEach / afterEach / onError across all stages
│ ├── EventBus — on() / emit(); step:start|success|error|skipped|progress, log
│ ├── ProgressTracker — subscribeProgress / subscribeStageResults / getProgress
│ ├── AbortController — abort() cancels current HTTP request via AbortSignal
│ ├── PauseController — pause() / resume() inter-stage checkpoints
│ ├── MetricsHooks — onPipelineStart / onPipelineEnd / onStepDuration
│ ├── StateSerializer — exportState() / importState() (stageResults + logs)
│ ├── PersistAdapter — pluggable save/load; auto-save after each stage
│ └── PluginManager — install() + destroy() lifecycle
│
├── PipelineBuilder (pipe())
│ .step() / .parallel() / .subPipeline() / .stream() → .build() / .toConfig()
│
├── createPipeline() — short factory wrapping new PipelineOrchestrator()
│
├── validatePipelineConfig() — duplicate keys, empty keys, type checks, recursive
│
├── /vue (separate entry point)
│ usePipelineRunVue / usePipelineProgressVue / usePipelineLogsVue
│ usePipelineStepEventVue / useRestClientVue / usePipelineStageResultVue
│
└── /react (separate entry point)
usePipelineRunReact / usePipelineProgressReact / usePipelineLogsReact
usePipelineStepEventReact / useRestClientReact / usePipelineStageResultReact
Bundle size & peer dependencies
| Entry point | Peer deps | Notes |
|---|---|---|
rest-pipeline-js |
— | Core — orchestrator, HTTP client, utilities. Depends on axios. |
rest-pipeline-js/vue |
vue ^3.3 |
Core + Vue composables |
rest-pipeline-js/react |
react ^19, react-dom ^19 |
Core + React hooks |
The package ships as tree-shakeable ESM (dist/esm/) and CommonJS (dist/cjs/), each with its own package.json ({"type":"module"} / {"type":"commonjs"}) so Node's native ESM resolver can load it directly — not just bundlers. The /vue and /react entry points are code-split — importing one does not bundle the other. Size is enforced in CI (see below); current brotli size per entry point is ~23 KB with all dependencies (axios for core; vue/react are peer deps, excluded).
Development
npm install
npm run build # tsc → dist/esm + dist/cjs, then writes the dist/*/package.json type markers
npm run verify:esm # loads every entry point with node's native ESM resolver (regression guard)
npm test # vitest — unit tests
npm run test:types # vitest --typecheck — type-level tests for pipe()'s TPrev threading
npm run test:coverage # vitest run --coverage, enforces the thresholds in vitest.config.ts
npm run lint # eslint .
npm run size # rebuilds, then checks brotli size per entry point against .size-limit.json
.github/workflows/ci.yml runs all of the above (lint, build, ESM-load check, tests, type tests, coverage, bundle size) on Node 20/22/24 for every push and pull request. (The published package itself only requires Node 18+ — see engines in package.json — but the dev toolchain, Vite 7 / Vitest 4, needs Node 20.19+/22.12+ just to load their own config, so CI can't verify Node 18 anymore.)
License
MIT
Author
Danil Lisin Vladimirovich aka Macrulez
GitHub: macrulezru · Website: macrulez.ru/en
Bugs and questions — issues
Support the project
Open source takes time and effort. If my work saves you time or brings value, consider supporting further development.
Thank you for being part of this journey.