@x12i/memorix-source-connector

@x12i/memorix-source-connector

Metadata-driven source connector runtime for memorix-connector/1. Definitions must set "role": "source". Peer: @x12i/memorix-connector-sdk@^1.

An installed provider definition supplies adapters, operations, response mappings, streams, workflow, credential policy, error mappings, and execution ceilings; this package validates it and returns a normal SDK defineConnector implementation.

npm install @x12i/memorix-source-connector

import { createMetadataConnector, validateDefinition } from "@x12i/memorix-source-connector";
import definition from "./connector.json" with { type: "json" };

const validation = validateDefinition(definition);
if (!validation.valid) throw new Error("Invalid connector metadata");

export default createMetadataConnector(definition);

Supported protocol adapters are governed HTTP, GraphQL and SOAP, plus explicitly permitted SFTP, read-only PostgreSQL/MySQL, and POP3 material-lease adapters. HTTP response forms include JSON, XML, text, exact binary staging, and ndjson (newline-delimited JSON with optional response.stream.maxBytes / onTruncation). XML list mapping uses shared JMESPath paths (itemsPath / itemPath) with stable attribute (@_…) handling. Request templates may use a host-supplied scope root, hydrate batch root, and forEachPaginated parent root. Optional session preconditions and workflow compose / poll / hydrate / forEachPaginated / cleanup steps, optional streams, applied offset/cursor pagination, and semantic profile handoff (semanticProfileId / acquisitionRouteId on stream and collect-like steps) are documented in the monorepo metadata contract. POP3 uses Credorix material-lease for username/password; host and port stay in non-secret configuration.

Product metadata (MGC-12)

Operations (and optionally 1:1 streams) may declare:

• goodFor / failureAffect — short string arrays for product use and failure impact
• requiredness — "must" (default when omitted) or "optional"
• requiresEntitlements — operator hints (SKU, OAuth scope, appliance feature); not a license API

Explorer DATA collections (X12I-MGC-04)

Default Explorer retention collection is {objectType}-raw. Override with stream explorerCollection; set dataRetention to "explorer" or "none" (omit to retain). Use objectCatalogStubs(def) / explorerCollectionFor(objectType) for catalog stubs without Map-1 joins. System collection connector-integrity-runs is the integrity ledger (ABS-03 schema).

Set connector cache.ttlSeconds so MCP mgc_fetch knows when _raw is stale. Optional agent.resources documents fetch identity keys; the catalog auto-derives resources from streams when omitted.

Vendor totals (MGC-13)

response.totalCountPath / optional truncatedPath feed page returnedCount, matchedCount, and truncated (matchedCount > returnedCount or flag or NDJSON cut → complete: false). Export classifyAcquisitionStream for host integrity status (complete / partial / failed / skipped / unsupported).

Parent-keyed hydrate merge (MGC-14)

hydrate with mergeIntoParent: true loads parent records from a prior saveAs, reads a child op per parentIdPath, stamps mergeMap fields onto each parent, and collects the parent stream. This is not compose (union of streams) and not a Level-1 abstract compile. | in mergeMap is first-non-empty. Child 404/410 leaves the parent metadata-only.

Foreign-id inspect (MGC-15)

idOwnership: "foreign" on collect/hydrate/forEachPaginated means the ids were not allocated this run. Default cleanup is "never"; pairing those ids with a kind: "cleanup" / delete step is a load-time error. Connector-allocated jobs still use create→poll→collect→cleanup always (MGC-07). Foreign inspect is not teardown. 404/410 skip as unsupported-by-retention.

The runtime never accepts executable JavaScript in metadata, never reads provider secrets, and never writes directly to Memorix storage. Provider requests use the host-supplied governed port; collection and atomic checkpoint advancement use ctx.collectStream; large content uses ctx.content.stage.

Only staged and installed definitions execute. The host must pin connector ID, semantic version, and definition hash.

Pausable acquisition (host ports)

executePausableWorkflow accepts a full SDK ConnectorPullContext or a thin host envelope (orgId, http, …). Missing workflow.run is self-hosted via createWorkflowRunner; missing source is {}. Missing http still fails closed (GOVERNED_HTTP_REQUIRED).

Hosts that are not an SDK pull host should build context with createPausableAcquisitionContext (HTTP wrap, auth headers, in-memory collectStream, workflow runner):

import {
  MetadataConnectorRuntime,
  createPausableAcquisitionContext,
} from "@x12i/memorix-source-connector";

const runtime = new MetadataConnectorRuntime();
const pullContext = createPausableAcquisitionContext({
  definition,
  orgId,
  pullId,
  http: hostHttpPort,
  authHeaders,
  signal,
});
const paused = await runtime.executePausableWorkflow(definition, pullContext, {
  signal: pauseSignal,
  authHeaders,
});