@molecule/api-text-provenance-overlap

@molecule/api-text-provenance-overlap

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Text provenance by word overlap, for @molecule/api-text-provenance.

Indexes every three-word sequence the assistant produced in the sessions (its replies and the files it wrote) and every sequence the user typed, then marks a paragraph ai when at least minAiShare (default 0.5) of its words are covered by assistant sequences and by no user sequence. The turn that contributed the most sequences is the paragraph's source; its prompt is the user message just before it, and its model is the turn's. The example is the whole build step: a post's markdown file and its folder of transcript exports in, provenance.json out.

Quick Start

// provenance.ts — runs in Node at build time (a build script or a Vite plugin), never in the page.
import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from 'node:fs'
import { dirname, join } from 'node:path'

import {
  canReadTranscript,
  readTranscript,
  setProvider as setTranscriptReader,
} from '@molecule/api-agent-transcript'
import { provider as anyTranscript } from '@molecule/api-agent-transcript-autodetect'
import { attributeText, setProvider as setAttribution } from '@molecule/api-text-provenance'
import { provider as wordOverlap } from '@molecule/api-text-provenance-overlap'

setTranscriptReader(anyTranscript) // reads Claude Code, Codex and Molecule IDE exports
setAttribution(wordOverlap)

// One block of the post, in page order. `prompt` and `model` are on every ai span.
export interface ProvenanceSpan {
  text: string // the block's markdown: a paragraph, heading, list or code block
  origin: 'human' | 'ai'
  prompt?: string // the person's message the AI was answering, as typed
  model?: string // the model that wrote it, as the transcript names it
}

// What /<slug>/provenance.json holds.
export interface Provenance {
  aiShare: number // 0..1, the share of the post's words the AI wrote
  words: number
  aiWords: number
  prompts: string[] // the distinct prompts behind the ai spans, in page order
  spans: ProvenanceSpan[]
}

// The post's top-level blocks: front matter dropped, split on blank lines, never inside a code fence.
export function markdownBlocks(markdown: string): string[] {
  const body = markdown.replace(/^---\r?\n[\s\S]*?\r?\n---\r?\n/, '')
  const blocks: string[] = []
  let lines: string[] = []
  let inFence = false
  for (const line of body.split(/\r?\n/)) {
    if (/^\s*(`{3}|~{3})/.test(line)) inFence = !inFence
    if (!inFence && line.trim() === '') {
      if (lines.length > 0) blocks.push(lines.join('\n'))
      lines = []
    } else {
      lines.push(line)
    }
  }
  if (lines.length > 0) blocks.push(lines.join('\n'))
  return blocks
}

// Attribute one post from its markdown file and the folder holding its transcript exports.
// A missing or empty folder is a 100% human post; files that are not transcripts are skipped.
export function postProvenance(markdownFile: string, transcriptDir: string): Provenance {
  const blocks = markdownBlocks(readFileSync(markdownFile, 'utf8'))
  const files = existsSync(transcriptDir)
    ? readdirSync(transcriptDir, { withFileTypes: true })
        .filter((entry) => entry.isFile())
        .map((entry) => entry.name)
        .sort()
    : []
  const sessions = files
    .map((name) => ({ text: readFileSync(join(transcriptDir, name), 'utf8'), fileName: name }))
    .filter((input) => canReadTranscript(input))
    .map((input) => readTranscript(input))
  const result = attributeText({ paragraphs: blocks, sessions })
  return {
    aiShare: result.aiShare,
    words: result.words,
    aiWords: result.aiWords,
    prompts: result.prompts,
    spans: result.paragraphs.map((p): ProvenanceSpan => {
      const text = blocks[p.index]
      return p.origin === 'ai'
        ? { text, origin: 'ai', prompt: p.prompt, model: p.model }
        : { text, origin: 'human' }
    }),
  }
}

// Write provenance.json, creating its folder.
export function writeProvenance(outFile: string, provenance: Provenance): void {
  mkdirSync(dirname(outFile), { recursive: true })
  writeFileSync(outFile, `${JSON.stringify(provenance, null, 2)}\n`)
}

// In the build, for each PUBLISHED post (skip drafts), after the site's own build has written dist/:
// writeProvenance('dist/my-post/provenance.json', postProvenance('posts/my-post.md', 'transcripts/my-post'))

Type

provider

Installation

npm install @molecule/api-text-provenance-overlap @molecule/api-agent-transcript @molecule/api-text-provenance

API

Functions

shingles(ws, size)

The size-word sequences of a word list, as strings.

function shingles(ws: readonly string[], size?: number): string[]

• ws — The words.
• size — Sequence length.

Returns: One key per starting position.

words(text)

A text's words, as attribution compares them.

function words(text: string): string[]

• text — Any text: markdown, rendered prose, a file's content.

Returns: Its words, lowercase, in order.

Constants

provider

Attributes paragraphs by overlapping word sequences.

const provider: TextProvenanceProvider

SHINGLE

How many consecutive words make one comparison unit.

const SHINGLE: 3

Core Interface

Implements @molecule/api-text-provenance interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-text-provenance'
import { provider } from '@molecule/api-text-provenance-overlap'

export function setupTextProvenanceOverlap(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-agent-transcript ^1.0.0
• @molecule/api-text-provenance ^1.0.0

Runtime Dependencies

• @molecule/api-agent-transcript

• @molecule/api-text-provenance

• Do NOT write your own transcript parser, paragraph matcher or prompt pairing. Bond the packages exactly as the example does, and install all four: npm install @molecule/api-agent-transcript @molecule/api-agent-transcript-autodetect @molecule/api-text-provenance @molecule/api-text-provenance-overlap.

• Do NOT import these from page or client code. They are server-only and throw in a browser bundle. Run the example at build time; the page reads its output (the file, or the spans handed to the prerender).

• Do NOT write into dist/ before the site's own build — Vite empties it. Write after the build, or emit the JSON from a Vite plugin: this.emitFile({ type: 'asset', fileName: 'my-post/provenance.json', source: JSON.stringify(map) }).

• Do NOT look a span's prompt up in prompts by position. Every ai span carries its own prompt and model; prompts is the distinct list, for the count ("from 3 prompts").

• spans[i] is block i of markdownBlocks(file). Render the page from these same spans (see @molecule/app-margin-notes-react, whose example takes this file as its input) so the page and the map always agree.

• A block made of words the person typed — a heading copied from the prompt, a paragraph pasted into it — is human. That is correct, not a miss.

• Compares words only (letters and digits, lowercase), so markdown in the session (**bold**, ## heading, links) matches the rendered prose, and a Claude Code /export's re-wrapped text matches too. Pass the markdown blocks as they are; do not strip or render them first.

• Tolerates light edits: changing a word breaks at most three sequences, so a paragraph with a few words changed stays ai. A rewrite falls below minAiShare and becomes human.

• A paragraph shorter than three words is ai only when the whole phrase appears in an assistant text and in no user text.

• When several turns wrote the same words (a draft, then a revision), the turn with the most matching sequences wins, and the later one on a tie.