@x12i/memorix-retrieval

@x12i/memorix-retrieval

@x12i/memorix-retrieval is the read-side runtime data tier for Memorix.

It loads Catalox descriptors, inventories Mongo databases, resolves collection bindings, reads mapped and unmapped records, executes list/slice descriptors, composes item views, builds graph/inventory views, and returns Explorer/API-ready JSON.

Full data-plane landscape (consultation): docs/MEMORIX-DATA-PLANE-OVERVIEW.md

Positioning

Package	Role
@x12i/memorix-retrieval	Runtime reads/views — Catalox descriptors + Mongo reads → lists, items, inventory, slices, graph, health
@x12i/memorix-writer	Full-envelope payload writes
@x12i/memorix-descriptors	Descriptor writes, slice schema, inventory ignore policy
@x12i/memorix-completion	Partial $set on missing data fields (not full-envelope writes)
memorix-explorer-api	HTTP facade over retrieval + writer

Retrieval does not write descriptors and does not write Memorix payload documents.

Quick start

Set MONGO_URI. Database names, entity/event/knowledge routing, and collection resolution are handled internally.

MONGO_URI=mongodb://localhost:27017

Host apps (Catalox + Mongo)

import {
  createMemorixRetrievalStackFromEnv,
  fetchMemorixListForEntity,
  fetchMemorixItemForEntity,
} from "@x12i/memorix-retrieval";

const { client, appId } = await createMemorixRetrievalStackFromEnv({
  cataloxContext: { superAdmin: true, actor: { type: "service", id: "my-api" } },
});

const list = await fetchMemorixListForEntity(client, {
  entityName: "assets",
  page: { limit: 50, offset: 0 },
});

const item = await fetchMemorixItemForEntity(client, {
  entityName: "assets",
  entityId: "10.150.68.31",
});

await client.close?.();

Targets: entity, event, knowledge

Target	Default DB	Identity field	Env override
entity	memorix-entities	entityId	MEMORIX_ENTITIES_DB
event	memorix-events	eventId	MEMORIX_EVENTS_DB
knowledge	memorix-knowledge	knowledgeId	MEMORIX_KNOWLEDGE_DB

Shipped knowledge-target object types in retrieval seeds: knowledge-playbooks, content-documents (extracted documents — see @x12i/memorix-corpus).

import {
  resolveMemorixDatabaseName,
  resolveMemorixIdentityField,
  resolveMemorixTargetFromDescriptor,
} from "@x12i/memorix-retrieval";

Unified inventory

One API powers Catalox-first and DB-first Explorer inventory modes:

import { buildMemorixUnifiedInventory } from "@x12i/memorix-retrieval";

const inventory = await buildMemorixUnifiedInventory(client, {
  sourceLens: "catalox-first", // or "db-first"
  targets: ["entity", "event", "knowledge"],
  includeExactCounts: false,
  includeSamples: true,
});

Returns normalized rows, issues, per-target summaries, and suggested actions.

Every inventory result includes source (MemorixRuntimeSourceRef), targetBindings, and per-row sources explaining descriptor and Mongo provenance.

Source-aware runtime provenance

Retrieval never returns anonymous data. Every Explorer-facing result can explain where it came from:

import {
  getSourceAwareMemorixRetrievalHealth,
  type MemorixRuntimeSourceRef,
  type RuntimeSourceAware,
} from "@x12i/memorix-retrieval";

const health = await getSourceAwareMemorixRetrievalHealth(client, {
  includeInventory: true,
});

// health.source.sourceClass === "runtime-projection"
// health.mongo.targets.knowledge.status may be "not-configured" or "empty" (non-fatal)

Key types: MemorixRuntimeSourceRef, RuntimeSourceAware<T>, MemorixTargetDbBinding, MemorixRequiredUserInput.

Legacy Catalox catalogs (memorix-entities, memorix_entity_content_types, knowledge) are not used for descriptor-driven reads.

Catalox-first vs DB-first

• Catalox-first — start from entity descriptors and declared content types; append orphan Mongo collections.
• DB-first — start from Mongo collections (parsed with last-dash rule); append missing Catalox-declared collections.

Both lenses use the same MemorixUnifiedInventory shape.

Graph-run filters (_graphRuns) — MRX-FRS-001

Planning queries for Exellix jobs over reserved _graphRuns.<graphId> paths:

import {
  fetchMemorixListForEntity,
  compileGraphRunFilter,
} from "@x12i/memorix-retrieval";

const list = await fetchMemorixListForEntity(client, {
  entityName: "assets",
  graphRunFilter: { graphRunNotDone: { graphId: "graph-qcrbz6t" } },
  includeGraphRuns: true, // optional; default false on lists
  page: { limit: 100, offset: 0 },
});

Facet	Use case
graphRunNotDone	Records needing processing
graphRunStatus	Filter by in_progress / done / failed
graphRunCompletedBefore	Stale results
graphRunFailed	Retry candidates
graphRunStaleVersion	Graph version invalidation

Item reads include _graphRuns by default (includeGraphRuns defaults to true). Writer APIs: @x12i/memorix-writer — see memorix-writer/docs/GRAPH-RUNS.md.

Graph-run counts and processing overview

Count documents by graph-run filter without fetching bodies (countDocuments only — never paginated reads):

import {
  countByGraphRunFilter,
  getProcessingOverview,
} from "@x12i/memorix-retrieval";

const pending = await countByGraphRunFilter(client, {
  target: "entity",
  collection: "assets-snapshots",
  graphRunFilter: { graphRunNotDone: { graphId: "graph-qcrbz6t" } },
});

const overview = await getProcessingOverview(client, {
  appId: "memorix",
  objectTypes: ["assets", "subnets"],
  graphIds: ["graph-qcrbz6t"], // omit to discover graphIds from _graphRuns data
  estimated: true, // approximate recordTotal only; bucket counts stay exact
});

Returns byTarget (storage-tier rollups), byEntity (per object type), and byGraph (per graphId) without paginating documents.

Index requirement on each canonical collection:

db.<canonicalCollection>.createIndex({ _graphRuns: 1 }, { sparse: true, name: "idx_graphRuns" });

Nested path filters (_graphRuns.<graphId>.status) use the sparse top-level index as baseline; per-graph compound indexes remain opt-in via @x12i/memorix-writer ensureGraphRunIndexes.

Job-type run stamps (_jobTypeRuns) — MRX-CR-003

Parallel to _graphRuns, eligibility and reprocessing can use _jobTypeRuns.<jobTypeId> (sanitized map keys). Read helper:

import { getJobTypeRun } from "@x12i/memorix-mongo";

await getJobTypeRun(retrieval, "assets", recordId, jobTypeId, "core");

Writer stamp APIs: @x12i/memorix-writer — see memorix-writer/docs/JOB-TYPE-RUNS.md.

Baseline sparse index:

db.<canonicalCollection>.createIndex({ _jobTypeRuns: 1 }, { sparse: true, name: "idx_jobTypeRuns" });

Per-jobType compound indexes: @x12i/memorix-writer ensureJobTypeRunIndexes creates:

• { "_jobTypeRuns.<key>.status": 1, <idField>: 1 }
• { "_jobTypeRuns.<key>.completedAt": -1 }
• { "_jobTypeRuns.<key>.status": 1, "_jobTypeRuns.<key>.completedAt": -1 } (reprocessing filters)

Linked record resolution — MRX-CR-004

import { resolveLinkedRecordIds } from "@x12i/memorix-mongo";

const { entity, recordIds } = await resolveLinkedRecordIds(
  retrieval,
  sourceEntity,
  sourceDoc,
  relationshipKey,
);

Estimated counts: Pass estimated: true for large unfiltered totals (estimatedDocumentCount / $collStats). Graph-run bucket counts always use exact countDocuments even when estimated: true.

Entity identity field

import {
  resolveDefaultListDescriptorForEntity,
  resolveEntityIdentityField,
} from "@x12i/memorix-retrieval";

const { entity, list, identityField } = await resolveDefaultListDescriptorForEntity(
  client,
  "subnets",
);
// identityField === "subnetId" when declared on descriptor; else target default

Last-dash collection parsing

Collection names follow <objectName>-<contentType> where the last dash is the separator (object names may contain dashes):

import { parseMemorixCollectionName } from "@x12i/memorix-retrieval";

parseMemorixCollectionName("topology-cidr-graphs-scoped");
// objectName = "topology-cidr-graphs", contentType = "scoped"

Raw collection records

Read Mongo directly without a Catalox descriptor:

import { fetchMemorixRawCollectionRecords } from "@x12i/memorix-retrieval";

const raw = await fetchMemorixRawCollectionRecords(client, {
  target: "entity",
  collectionName: "assets-snapshots",
  page: { limit: 50, offset: 0 },
});

Open records from inventory rows with descriptor or raw fallback:

import { fetchMemorixCollectionRecords } from "@x12i/memorix-retrieval";

const records = await fetchMemorixCollectionRecords(client, {
  entityName: "assets",
  contentType: "snapshots",
  mode: "auto",
});

Slice execution

List descriptors with kind: "slice" are executable saved views:

Slice descriptors use the canonical shape from @x12i/memorix-descriptors (entityName, target, contentType, filter, ui.group). Create slices with createSliceDescriptor in descriptors; execute them here with fetchMemorixSliceRecords.

import {
  fetchMemorixSlices,
  fetchMemorixSliceRecords,
  fetchMemorixSliceItem,
  loadMemorixSliceDescriptor,
} from "@x12i/memorix-retrieval";

const slices = await fetchMemorixSlices(client, { includeHealth: true });
const page = await fetchMemorixSliceRecords(client, {
  sliceId: "critical-vulnerabilities",
  page: { limit: 50, offset: 0 },
});

Record display names

Human-readable titles for operators come from record.concept.title on stored documents (see MEMORIX-DATABASE-CONVENTIONS.md). Memorix read-tier APIs resolve display names centrally — hosts must not scan top-level id, name, or data.* fields.

import { resolveMemorixRecordDisplayName } from "@x12i/memorix-retrieval";

const displayName = resolveMemorixRecordDisplayName(record, {
  preferredLocale: "en", // optional; for localized concept.title objects
});
// string | undefined — never falls back to record id or identity fields

Supported concept.title shapes: plain string, string array (first non-empty entry), localized object ({ en: "...", he: "..." }). Legacy concept.name may still be read during transition.

List and item responses include optional top-level displayName (and list rows also expose recordId when resolvable):

• fetchMemorixList / fetchMemorixListForEntity → rows[].displayName
• fetchMemorixItem / fetchMemorixItemForEntity → displayName
• listMemorixEntityContentTypeDocuments → documents[].displayName

When concept.title is absent, displayName is omitted. Show the record id separately for disambiguation — do not substitute it as the title.

Easy retrieval helpers

Use these when you need identity + display name without re-implementing Memorix rules:

import {
  fetchMemorixRecordInfoForEntity,
  resolveMemorixRecordInfo,
  resolveMemorixRecordInfoFromRow,
  resolveMemorixRecordInfoFromItem,
  formatMemorixRecordLabel,
} from "@x12i/memorix-retrieval";

// One call: lookup by entity + recordId / entityId / eventId / knowledgeId
const info = await fetchMemorixRecordInfoForEntity(client, {
  entityName: "vulnerability-groups",
  recordId: "69d216ed358778e93215788d",
});
// → { recordId, displayName?, entityId?, identityField?, identityValue? }

// From list rows (already stamped with displayName)
const fromRow = resolveMemorixRecordInfoFromRow(listRow);

// From item responses
const fromItem = resolveMemorixRecordInfoFromItem(itemResponse);

// Picker / journal label — name when present, else id
const label = formatMemorixRecordLabel(info);
const withSuffix = formatMemorixRecordLabel(info, { includeIdSuffix: true });

Search: memorixLabelSearchFilter (used by raw entity reads) includes concept.title (and legacy concept.name) for string-valued titles. Localized-object search is best-effort on the concept.title path.

Record identity

import { resolveMemorixRecordIdentity } from "@x12i/memorix-retrieval";

const { id, identityField, source } = resolveMemorixRecordIdentity("knowledge", record);

Resolution order: descriptor identity field → target default (entityId / eventId / knowledgeId) → id → Mongo _id.

Health model

import { getMemorixRetrievalHealth } from "@x12i/memorix-retrieval";

const health = await getMemorixRetrievalHealth(client, { includeInventory: true });
// health.catalox, health.mongo.targets, health.descriptors, health.inventory

Logging

Retrieval uses @x12i/logxer ≥ 5.1.0 (dual ESM/CJS; use @x12i/logxer/package-levels or @x12i/logxer/diagnostics in browser-safe code — never the main entry). Pass host logging through constructors; retrieval forwards it to logxer as { stack } and does not manage host-wide registries.

Environment (this package only)

# Default when unset: warn
MEMORIX_RETRIEVAL_LOGS_LEVEL=info

# Silence retrieval logs
MEMORIX_RETRIEVAL_LOGS_LEVEL=off

Allowed values: off (or none / silent), error, warn, info, debug, verbose.

Downstream usage

Pass the same StackLoggingOptions object you use for other x12i packages:

import {
  createMemorixRetrieval,
  createMemorixRetrievalStackFromEnv,
  resolveMemorixRetrievalLogLevel,
  type StackLoggingOptions,
} from "@x12i/memorix-retrieval";

const logging: StackLoggingOptions = {
  packageLevels: {
    MEMORIX_RETRIEVAL: "debug",
    CATALOX: "warn",
  },
};

// Preferred: pass logging when creating the client / stack
const { client } = await createMemorixRetrievalStackFromEnv({
  logging,
  cataloxContext: { superAdmin: true, actor: { type: "service", id: "my-api" } },
});

// Or shorthand for this package only
await createMemorixRetrievalStackFromEnv({ logLevel: "info", ... });

// Introspection (optional stack argument; defaults to bound stack)
resolveMemorixRetrievalLogLevel(logging); // "debug"

Precedence for this package: parent logging stack → MEMORIX_RETRIEVAL_LOGS_LEVEL env → logxer default warn. Host apps configure bulk env / process registries via @x12i/logxer directly — not through memorix-retrieval.

Exports: createMemorixRetrievalLogger, bindMemorixRetrievalLogging, memorixRetrievalLogger, toMemorixRetrievalStackLogging, MEMORIX_RETRIEVAL_LOGS_LEVEL_ENV_KEY.

Documentation

Doc	Purpose
DATA-TIER-CONTRACT.md	Public API surface (what hosts may call)
XRONOX-DATA-TIER-REQUIREMENTS.md	Xronox API/env requirements for Memorix data tier
MEMORIX-CATALOX-CONTRACTS.md	Descriptor catalogs, JSON formats, cross-component sync
MEMORIX-DATABASE-CONVENTIONS.md	Mongo DB names, collections, env vars
EXPLORER-HOST-APIS.md	Graph, raw reads, inventory, slices (Explorer hosts)

Advanced (optional)

Option / env	Purpose
MEMORIX_ENTITIES_DB / MEMORIX_EVENTS_DB / MEMORIX_KNOWLEDGE_DB	Non-default database names
MEMORIX_ENTITIES_COLLECTION_* / MEMORIX_EVENTS_COLLECTION_* / MEMORIX_KNOWLEDGE_COLLECTION_*	Per-type collection overrides
CATALOX_APP_ID / MEMORIX_APP_ID	Catalox app namespace (default: memorix)
MEMORIX_RETRIEVAL_LOGS_LEVEL	Logxer verbosity for this package (warn default; off to silence, info/debug for more detail)
memorixDb	Code overrides for entity/event/knowledge DB names
xronox	Inject a pre-configured Xronox client instead of built-in Mongo reads

Entity descriptors declare target: "entity" | "event" | "knowledge" (default entity).

Catalox seeds

Source of truth: catalox-seeds/inputs/

npm run catalox:seed:build
npm run catalox:seed:memorix-retrieval:validate
npm run catalox:seed:memorix-retrieval:apply
npm run catalox:seed:ensure

Lists catalog (lists)

The list descriptor catalog id is lists (legacy: memorix-list-descriptors). Migrate existing Catalox data with memorix-descriptors/scripts/migrate-list-descriptors-catalog.mjs.

Compound sort

fetchMemorixList accepts sort: [{ property, direction }, ...]. Secondary sort keys must resolve to the same content type as the primary key. Ensure a matching compound index exists on the driver collection — there is no automatic index provisioning.

Analytics fields on lists

List descriptors may declare analytics[] with { field, collection, joinBy, metric }. Values are merged into rows per page via scoped aggregation (groupMemorixRecords).

Full record fetch

import { fetchMemorixFullRecord } from "@x12i/memorix-retrieval";

const full = await fetchMemorixFullRecord(client, {
  entityName: "assets",
  entityId: "abc-123",
  // contentTypes: ["snapshots", "analysis"], // optional subset
});
// full.contentTypes — raw document(s) per content type

Unlike fetchMemorixItemForEntity, this bypasses item descriptors and returns uncurated documents.

List filters

Request filters use MemorixFilterInput[]. A property is authorized when it is filterable: true on the entity descriptor and belongs to the list's leading content type. The list descriptor's filters[] array supplies pinned defaults only — it is not an allow-list for ad-hoc filters.

Build, test, and live smoke

npm run build
npm test

Live checks (require .env with MONGO_URI and Mongo Catalox credentials):

npm run mongo:check-collections
npm run mongo:inventory -- --sourceLens db-first --targets entity,event,knowledge
npm run mongo:inventory -- --sourceLens catalox-first --targets all
npm run mongo:inventory -- --includeExactCounts
npm run smoke:retrieval -- --health --source-aware
npm run smoke:retrieval -- --inventory --sourceLens db-first
npm run smoke:retrieval -- --raw-records --target entity --collection assets-snapshots --limit 5
npm run smoke:retrieval -- --collection-records --entity assets --contentType snapshots --mode auto
npm run smoke:retrieval -- --slice critical-vulnerabilities-list --limit 5
npm run smoke:retrieval -- --graph --sourceLens db-first --includeInventory

Publish

npm run build && npm test && npm publish