@x12i/memorix-completion

@x12i/memorix-completion

Folder name: this package lives in memorix-compeletion/ (typo). The npm package is @x12i/memorix-completion.

Fill missing fields in Memorix entity, event, and knowledge records by looking up data in source MongoDB databases, using JSON completion mappings or Catalox (--source catalox).

Mappings are plain JSON files (arrays of mapping objects). Use {{ENV.variableName}} tokens in strings; they resolve via @x12i/env-tokens at load time.

Install

npm install @x12i/memorix-completion

Quick start

Set MONGO_URI and run. Database names and entity/event routing are resolved internally (defaults: memorix-entities, memorix-events).

MONGO_URI=mongodb://localhost:27017
npx memorix-complete --mappings ./my-mappings.json

Source database for enrichment: set MEMORIX_SOURCE_DB or per-mapping source.databaseName.

See Memorix Database Conventions for shared naming rules across x12i peers.

Databases (advanced overrides)

Each mapping declares a target — either entity or event — which determines which Memorix database receives updates:

Target	Env variable	Default database
entity	MEMORIX_ENTITIES_DB	memorix-entities
event	MEMORIX_EVENTS_DB	memorix-events

Source records are read from a separate database configured per mapping (or via env defaults):

Env variable	Purpose
MEMORIX_SOURCE_DB / SOURCE_DATABASE_NAME	Default source database when not set in mapping
mapping.source.databaseName	Per-mapping source DB (supports {{ENV.*}})

Mapping JSON

Each mapping file is a JSON array. Example entry:

{
  "name": "complete-vulnerability-entity-missing-data",
  "entityType": "vulnerability",
  "target": "entity",
  "version": "1.0.0",
  "enabled": true,
  "targetCollection": "vulnerabilities",
  "targetSelector": {
    "sourceCollection": "vulnerabilities-information"
  },
  "source": {
    "databaseName": "{{ENV.sourceDatabaseName}}",
    "collection": "vulnerabilities-information"
  },
  "match": {
    "targetPath": "entityId",
    "sourcePath": "vulnerabilityId"
  },
  "writeRoot": "data",
  "mode": {
    "onlyFillMissing": true,
    "overwriteExisting": false,
    "createNestedObjects": true
  },
  "ignore": {
    "exactProperties": ["_id", "metadata"],
    "idProperties": true
  },
  "fields": [
    { "sourcePath": "enrichment", "targetPath": "enrichment" }
  ]
}

Field	Description
target	entity or event — selects Memorix database and record type
targetCollection	Memorix Mongo collection for target records (recommended)
targetCollectionCandidates	Extra names to try if targetCollection is omitted
targetSelector.sourceCollection	Source lineage selector; matches _system.provenance.source.collection
source.databaseName / collection	Source DB lookup (supports {{ENV.*}})
match.targetPath / sourcePath	Join key between Memorix record and source document
fields	Paths to copy from source → target data

For event mappings, set "target": "event" and point targetCollection at the appropriate collection in memorix-events.

CLI

After npm run build:

# Dry-run (default)
npx memorix-complete --mappings ./my-mappings.json

# Write to Mongo
npx memorix-complete --mappings ./my-mappings.json --write

# Filters
npx memorix-complete -m ./my-mappings.json --entity vulnerability --target entity --limit 100 --write

Environment:

• MONGO_URI — required Mongo connection (simple mode)
• MEMORIX_SOURCE_DB / SOURCE_DATABASE_NAME — source database when not in mapping
• MEMORIX_ENTITIES_DB / MEMORIX_EVENTS_DB — optional overrides (defaults shown below)
• MEMORIX_ENTITIES_COLLECTION_<ENTITY_TYPE> — optional entity collection override
• MEMORIX_EVENTS_COLLECTION_<ENTITY_TYPE> — optional event collection override
• MEMORIX_COMPLETION_MAPPINGS — default mappings file path for CLI
• MEMORIX_COMPLETION_TARGET — default target filter (entity, event, or all)

Programmatic API

Full Mongo pipeline (mappings file → read → complete → write)

import {
  runMongoCompletion,
  summarizeCompletionResults,
  loadCompletionMappingsFromFile
} from "@x12i/memorix-completion";

const report = await runMongoCompletion({
  mappings: "./completion-mappings.json", // or pre-loaded array
  dryRun: true,
  limit: 0, // 0 = all records per mapping
  filter: { entityType: "vulnerability", target: "entity" }
});

console.log(summarizeCompletionResults(report.results));

Host-provided I/O (any database)

import {
  runMemorixCompletion,
  loadCompletionMappingsFromFile
} from "@x12i/memorix-completion";

const mappings = loadCompletionMappingsFromFile("./completion-mappings.json");

const results = await runMemorixCompletion({
  sourceDatabaseName: mappings[0].source.databaseName,
  records,
  mappings,
  dryRun: false,
  sourceReader: async ({ databaseName, collection, matchPath, matchValue }) => {
    return findOne({ db: databaseName, collection, filter: { [matchPath]: matchValue } });
  },
  recordWriter: async ({ record, updatedData }) => {
    await updateRecord(record.recordId, { data: updatedData });
  }
});

Load mappings only

import { loadCompletionMappingsFromFile } from "@x12i/memorix-completion";

const mappings = loadCompletionMappingsFromFile("./maps.json", {
  env: process.env // default; resolves {{ENV.*}}
});

Mongo adapters (custom orchestration)

import {
  connectMongoCompletion,
  createMongoSourceReader,
  createMongoRecordWriter,
  loadRecordsFromCollection,
  resolveTargetCollectionName,
  resolveMemorixDbName
} from "@x12i/memorix-completion";

const entitiesDb = resolveMemorixDbName("entity");
const eventsDb = resolveMemorixDbName("event");

Core exports

Export	Description
runMongoCompletion	End-to-end runner from mapping JSON file(s)
runMemorixCompletion	Batch runner with injected readers/writers
completeMemorixRecord	Single-record completion
loadCompletionMappingsFromFile	Load + resolve {{ENV.*}} in mapping JSON
resolveJsonEnvTokens	Deep token resolution on any JSON value
filterCompletionMappings	Filter by entity, target, name, enabled, fields
resolveMemorixDbName	Resolve entity/event Memorix database from env
resolveDefaultSourceDbName	Resolve default source database from env

Scripts (this repo)

npm run build
npm test
npm run poc:live -- --dry-run --entity vulnerability --target entity --limit 10
npm run complete -- --mappings test/poc-data-fix.json --write

Publish

From the x12i workspace root:

./scripts/publish-memorix-packages.sh