metamod v0.3.1
metamod
Composable and cachable processing units
npm i metamod
Why?
Metamod is a library to create performant and debuggable processing pipelines.
The goal is to divide (and conquer) complex tasks into the smallest processing units possible, called "mods".
Each mod is basically a simple (potentially async) function that takes requests and tokens (more on that later) and returns new requests and tokens. The requests are like parameters triggering the use of one or many mods for a specific "request type" (for example: "compile a code from TypeScript to JavaScript").
Tokens are stored values that can be reused by any mod later, without the need to add data to the requests. This is especially useful for configuration values. Mods explicitly declare which tokens they need (with readTokens
) and which tokens they update.
Mods can be pure, meaning they don't have side-effects, or impure, meaning they can have side-effects. Pure mods are cached, meaning they are only executed once for a given combination of request id, request data, token values and package versions - even between pipeline runs! Impure mods are not cached, meaning they are executed every time they are called. Typically mods calling the filesystem to read or write files are impure. Mod author can specifiy if a mod is pure or impure with the sideEffects
flag.
Mods are called automatically by Metamod in sequence for a given request id. All mods registered to a pipeline, either directly or via plugins, will be automatically called for requests of the corresponding type.
If multiple requests have the same id, they are grouped together and the mods are called only once for the group. This is useful for example to merge multiple content together.
Requests are processed in parallel and continuously as new requests are created by mods. (Note that multithreading is not supported yet, but it's planned. Until then, all processing is bound by the JavaScript thread.) Mods can however mark certain requests with the waitForNextStep
flag so they are queued until all current (and potentially future) requests are processed. Then the pipeline moves onto a new "step" if there are pending requests with the waitForNextStep
flag. This can be useful for example to wait for all source files to be processed before bundling them together.
Getting started
Note: currently Metamod is very barebone and low-level. Higher-level utility features are planned to make it easier to use for certain use cases such as processing files.
Anatomy of a mod
A mod is described by an object with a name
, a requestType
and other properties, plus a process
function which is where all the actual work will happen.
Example of impure mod (because it reads a file):
{
name: 'demo-read-file',
requestType: 'read-file',
sideEffects: true,
readTokens: ['srcDir'],
process: async ({ requests, tokens }) => {
const { id } = requests[0]
const content = await fs.readFile(path.resolve(tokens.get('srcDir'), id), 'utf8')
return {
requests: [
{
type: 'process-file',
id,
data: {
content,
},
},
],
}
},
}
Example of a pure (thus cacheable) mod:
{
name: 'demo-process-file',
requestType: 'process-file',
sideEffects: false,
process: async ({ requests }) => {
const { id, data } = requests[0]
const content = data.content.toUpperCase()
return {
requests: [
{
type: 'write-file',
id,
data: {
content,
},
},
],
}
},
}
You can use the createRequest(type, id, data?)
helper to create requests:
import { createRequest } from 'metamod'
const request = createRequest('read-file', 'hello.js', { content: 'console.log(`hello`)' })
The process
function provides requests
and tokens
as input, and you can return an object with new requests
and new tokens
as output:
return {
requests: [
{
type: 'read-file',
id: 'foo.txt',
}
],
// Can also be an array of tokens { id: string, value: any }[]
tokens: {
srcDir: 'src',
outDir: 'dist',
},
}
Defining a pipeline
A pipeline is a list of mods (or plugins adding mods). It has a special init
function called with an optional configuration object, that allows the pipeline to bootstrap the initial requests and tokens. This init
function is always called at the beginning of the pipeline.
import { definePipeline } from 'metamod'
const pipeline = definePipeline({
name: 'demo',
version: '0.0.0',
init: (config) => {
return {
requests: config.entryFiles.map(file => ({
type: 'read-file',
id: file,
})),
tokens: {
srcDir: config.srcDir,
outDir: config.outDir,
},
}
},
mods: [
// Mods go here
],
plugins: [
// Plugins go here
],
})
Running a pipeline
Use the runPipeline
function to run a pipeline. It takes a pipeline and a configuration object, and returns a promise that resolves when the pipeline is done.
import { runPipeline } from 'metamod'
await runPipeline(pipeline, {
srcDir: 'input',
outDir: 'output',
entryFiles: ['foo.txt'],
})
Debugging
Note: debugging features are still work in progress. For example, a fully interactive UI is planned.
You can enable logging with the DEBUG
environment variable:
DEBUG=metamod* node my-pipeline.mjs
Reusing mods
You can create generic mods by passing the request parameters (or even the entire requests) as request data:
import { createRequest } from 'metamod'
export const ReadFileMod = defineMod<{
requestType: string
}>({
name: 'read-file',
requestType: 'read-file',
sideEffects: true,
async process ({ requests }) {
const file = requests[0].id
const { requestType } = requests[0].data
if (fs.existsSync(file)) {
const content = await fs.promises.readFile(file, 'utf8')
return {
requests: [
createRequest(requestType, id, { content }),
],
}
}
},
})
You can even create an helper function to help create requests tailored for this mod:
export function requestReadFile (id: string, nextRequestType: string) {
return createRequest('read-file', id, {
requestType: nextRequestType,
})
}