1.33.0 • Published 1 week ago

@x12i/memorix-assembler

Licence

exellix-license

Version

1.33.0

Deps

Size

76 kB

Vulns

Weekly

Summary Dependency Versions

@x12i/memorix-assembler

Assembles a clean, data-only graph input from one or more Memorix content types.

This package owns all Memorix fetch / assembly / context logic so that execution code (workers, UI, HTTP handlers) has zero knowledge of Memorix internals.

Memorix concepts
Object types and content types
Collection naming
Identity and linkage within an object type
Cross-object-type linkage (context)
The 1:1 vs 1:n rule (fixed)
The data-only contract
Assembly output shape
Priority and merge order for 1:1 types
Property selection
Context resolution
Public API
Integration seams
Worked examples

Memorix is the operational data layer of Exellix. It organises records into typed collections and tracks processing history (which graphs have run, results, failures) directly on each record via _graphRuns.

Key actors:

Memorix retrieval (@x12i/memorix-retrieval) — reads records from MongoDB.
Memorix writer (@x12i/memorix-writer) — stamps _graphRuns back after execution.
Descriptors (@x12i/memorix-descriptors) — declares the shape of each object type.

2. Object types and content types

An object type (also called entity) is a top-level domain concept, e.g. subnets, assets, vulnerabilities.

Each object type is split into content types — logical partitions of its data. Common content types:

Postfix	Role	Cardinality
`core`	Primary identity record	1:1
`snapshots`	Latest snapshot of the full state	1:1
`events`	Timeline events for the object	1:n
`alerts`	Active alerts	1:n
`scores`	Computed risk / priority scores	1:n
…any other	Custom enrichment or processing output	1:n

Content type keys are declared in the entity descriptor (MemorixEntityDescriptor.contentTypes).

3. Collection naming

MongoDB collections for a Memorix entity follow the convention:

{collectionPrefix}-{contentTypePostfix}

Examples for subnets:

Content type	Collection name
`core`	`subnets-core`
`snapshots`	`subnets-snapshots`
`events`	`subnets-events`

collectionPrefix is declared in the descriptor (defaults to the entity name).

4. Identity and linkage within an object type

All content-type collections for the same object type share a single identity field that links records to each other:

Target type	Identity field
`entity`	`entityId`
`event`	`eventId`
`knowledge`	`knowledgeId`

Rule: use one field, do not reinvent. When the assembler needs to fetch the events collection for a subnets record, it simply queries { entityId: <value> } (where value was read from the core record).

The core record is always the identity anchor: we load it first (by recordId or _id), read its entityId, then use that value to join all other collections.

How `buildRecordLookupQuery` works

The initial core record lookup uses a broad OR filter to handle different record identifier patterns in use across the codebase:

{ $or: [{ _id: recordId }, { entityId: recordId }, { recordId: recordId }] }

Subsequent cross-collection queries use the specific identity field only, e.g.:

{ entityId: "subnet-abc-123" }

5. Cross-object-type linkage (context)

Context is cross-object data: e.g. vulnerabilities linked to an asset, or owners linked to a subnet.

Runtime injection uses jobMemory.context only (object bucket):

Hippox approved links (hippoxLinkIds / inline approved links) — object-type keys
Declarative contextQueries / contextSources — alias keys (and enrichment leaves)
Assembly promotions (host) — discovery, associated*, etc. under .context

Relation-based descriptor contextLinks are not resolved into job memory (no _legacy bucket). Migrate joins to Hippox.

const jobMemory = await resolveAssemblerContext(retrieval, {
  input: assembledData,
  sourceEntity: 'subnets',
  hippoxLinkIds: run.hippoxContext?.linkIds,
});
// → { context: { subnets?: …, zones?: … } }

This is entirely separate from same-object-type assembly:

Assembly = same object, multiple content types → merge into runtime.input
Context = different objects → runtime.jobMemory.context

6. The 1:1 vs 1:n rule (fixed)

This rule is fixed — not configurable.

Postfix	Cardinality	Semantics
`core`	1:1	One primary record per identity
`snapshots`	1:1	One latest snapshot per identity
anything else	1:n	Many records per identity

Why fixed? core and snapshots are structurally defined as singular by Memorix conventions. All other content types (events, alerts, scores, custom enrichments) naturally produce multiple records for a single entity.

7. The data-only contract

Every Memorix document has this shape in MongoDB:

{
  "_id":          "...",           // Mongo ObjectId or string
  "_memorixRef":  { ... },         // canonical pointer — used by graph-run tracker
  "_graphRuns":   { ... },         // processing history — DO NOT send to graph
  "narratives":   { ... },         // operational labels
  "createdAt":    "...",
  "updatedAt":    "...",
  "data": {                        // ← ONLY THIS is domain data
    "entityId": "subnet-abc-123",
    "hostname": "router-1",
    "location": { "city": "TLV" },
    ...
  }
}

The assembler always reads from data (or the descriptor-configured dataRoot) and never sends system fields to the graph:

Stripped fields: _id, __v, _memorixRef, _graphRuns, narratives, createdAt, updatedAt, lastSeen, firstSeen.

The only system field that IS added back is _memorixRef (synthesised from the assembled identity) so the graph-run tracker can stamp _graphRuns on the right record after execution.

8. Assembly output shape

Given a selection of ['core', 'snapshots', 'events'] for a subnets entity:

{
  "entityId": "subnet-abc-123",       // ← from core (1:1, top priority by default)
  "hostname":  "router-1",             // ← from core
  "lastSeen":  "2024-11-01T00:00:00Z", // ← from snapshots (1:1, merged at root)
  "events": [                          // ← 1:n, array always, named by postfix
    { "type": "change", "at": "...", ... },
    { "type": "alert",  "at": "...", ... }
  ],
  "_memorixRef": {                     // added by assembler for graph-run tracker
    "target":      "entity",
    "entityName":  "subnets",
    "contentType": "core",
    "recordId":    "subnet-abc-123"
  }
}

Key guarantees:

1:1 types are merged at the root in priority order (first-wins on collision).
1:n types are always arrays, even when empty or containing a single item.
1:n array keys are the content-type postfix (events, alerts, etc.).
No system fields are present except _memorixRef.

9. Priority and merge order for 1:1 types

When two 1:1 content types define the same key, the priority array decides which value to keep.

const selection: AssemblySelection = {
  objectType: 'subnets',
  contentTypes: [{ key: 'core' }, { key: 'snapshots' }],
  priority: ['snapshots', 'core'],  // snapshots wins on collision
};

Content types not listed in priority are appended after the listed ones in declaration order. Default priority (when priority: []) is the order of contentTypes in the selection, with the first entry winning.

10. Property selection

listSelectableProperties(retrieval, binding, contentTypeKey) samples up to 50 live documents from the content-type collection and infers the property tree from their data root.

Returns:

{
  objectType: string;
  contentType: string;
  properties: PropertyNode[];  // nested tree
  flatPaths: string[];         // all leaf + branch paths (use as default selectedProperties)
}

AssemblyContentTypeConfig.selectedProperties:

undefined (omitted) → include all properties (default).
[] → include nothing (edge case, effectively empty input from this type).
['hostname', 'location.city'] → include only these dot-notation paths.

Property projection is applied after extracting the data root, before merging.

11. Context resolution

import { resolveAssemblerContext } from '@x12i/memorix-assembler';

const jobMemory = await resolveAssemblerContext(retrieval, {
  input: assembledData,        // must contain _memorixRef for entity auto-inference
  sourceEntity: 'subnets',     // optional explicit override
  hippoxLinkIds: run.hippoxContext?.linkIds,
});
// → { context: { /* object-type / alias keys */ } }

Without Hippox links, returns { context: {} }. Relation contextLinks are ignored.

12. Public API

`assembleRecordData(retrieval, selection, recordId)`

Fetch and merge all selected content types for a record.

const data = await assembleRecordData(retrieval, {
  objectType: 'subnets',
  target: 'entity',
  contentTypes: [
    { key: 'core' },
    { key: 'snapshots', selectedProperties: ['lastSeen', 'status'] },
    { key: 'events' },
  ],
  priority: ['core', 'snapshots'],
}, 'subnet-abc-123');

Returns null when the record cannot be found in any collection.

`listSelectableProperties(retrieval, binding, contentTypeKey)`

Returns the property tree for one content type, sampled from live data.

const { properties, flatPaths } = await listSelectableProperties(retrieval, {
  entityName: 'subnets',
  collectionPrefix: 'subnets',
  target: 'entity',
}, 'events');

`resolveAssemblerContext(retrieval, params)`

Resolves cross-object-type context into jobMemory.context from Hippox approved links.

`readRecordIdentity(doc, target?)`

Reads the identity value from a raw Mongo document.

`buildIdentityQuery(identity)`

Builds the Mongo filter { entityId: value } from a resolved identity.

13. Integration seams

┌────────────────────────────────────────────────────────────────┐
│  Enqueue (on-demand / batch / work)                           │
│  ─ QueueSingleInput.assemblySelection → JobRun.assemblySelection│
└────────────────────────┬───────────────────────────────────────┘
                         │ persisted on JobRun
┌────────────────────────▼───────────────────────────────────────┐
│  Worker (runOne in @exellix/jobs)                             │
│  ─ if run.assemblySelection → deps.assembleInput(run)          │
│    → assembleRecordData(retrieval, selection, run.input.recordId)│
│  ─ resolvedInput used as runtime.input for executeGraph         │
└────────────────────────┬───────────────────────────────────────┘
                         │ runtime.input = assembled data
┌────────────────────────▼───────────────────────────────────────┐
│  executeGraph (graph-engine)                                  │
│  ─ receives { input: assembledData, jobMemory: context }        │
└────────────────────────┬───────────────────────────────────────┘
                         │ graph executes, result returned
┌────────────────────────▼───────────────────────────────────────┐
│  GraphRunTracker (createDynamicMemorixGraphRunTracker)         │
│  ─ reads _memorixRef from assembled input                       │
│  ─ stamps _graphRuns on the Memorix record (via writer)         │
└────────────────────────────────────────────────────────────────┘

Assembly deps wiring

Background worker (jobs/src/memorix-jobs-runtime.ts):

assembleInput: async (run) => {
  const selection = run.assemblySelection as AssemblySelection;
  if (!selection) return null;
  return assembleRecordData(retrieval, selection, String(run.input.recordId));
},

Inline execute (jobs-ui/src/factory/execution-service.ts): Same pattern. The dep is cached per worker-id singleton.

Server-side endpoints

Method	Path	Purpose
`GET`	`/api/assembler/properties`	Property tree for one content type
`POST`	`/api/assembler/preview`	Preview assembled data for a record

const selection: AssemblySelection = {
  objectType: 'assets',
  target: 'entity',
  contentTypes: [
    { key: 'core' },
    { key: 'snapshots' },
    { key: 'vulnerabilities' },
  ],
  priority: ['core', 'snapshots'],
};

const data = await assembleRecordData(retrieval, selection, 'asset-xyz');
// →
// {
//   entityId: 'asset-xyz',
//   hostname: 'web-1',
//   os: 'Ubuntu 22.04',
//   lastSeen: '2024-12-01T...',     // from snapshots (lower priority, doesn't overwrite core)
//   status: 'online',               // from snapshots (not in core → added)
//   vulnerabilities: [              // 1:n array
//     { cveId: 'CVE-2024-...', severity: 'high' },
//     { cveId: 'CVE-2024-...', severity: 'medium' },
//   ],
//   _memorixRef: { target: 'entity', entityName: 'assets', ... }
// }

Example B: Narrowing to selected properties

const selection: AssemblySelection = {
  objectType: 'subnets',
  target: 'entity',
  contentTypes: [
    { key: 'core', selectedProperties: ['entityId', 'cidr', 'location.country'] },
    { key: 'events', selectedProperties: ['type', 'at'] },
  ],
  priority: ['core'],
};
// → only the listed paths are included in the output

Example C: Events-only graph (zero 1:1 types)

const selection: AssemblySelection = {
  objectType: 'subnets',
  contentTypes: [{ key: 'alerts' }],
  priority: [],
};
// → { alerts: [{ severity: 'critical', ... }, ...], _memorixRef: { ... } }
// No root-level merging needed — alerts is 1:n, placed directly as array

Keywords

exellix memorix assembler content-types

@x12i/memorix-assembler

@x12i/memorix-assembler

Table of contents

1. Memorix concepts

2. Object types and content types

3. Collection naming

4. Identity and linkage within an object type

How `buildRecordLookupQuery` works

5. Cross-object-type linkage (context)

6. The 1:1 vs 1:n rule (fixed)

7. The data-only contract

8. Assembly output shape

9. Priority and merge order for 1:1 types

10. Property selection

11. Context resolution

12. Public API

`assembleRecordData(retrieval, selection, recordId)`

`listSelectableProperties(retrieval, binding, contentTypeKey)`

`resolveAssemblerContext(retrieval, params)`

`readRecordIdentity(doc, target?)`

`buildIdentityQuery(identity)`

13. Integration seams

Assembly deps wiring

Server-side endpoints

Enqueue path

14. Worked examples

Example A: Single entity, two 1:1 types + one 1:n

Example B: Narrowing to selected properties

Example C: Events-only graph (zero 1:1 types)

Keywords