@fnet/shell-flow

@fnet/shell-flow

Introduction

The @fnet/shell-flow project provides a powerful, expression-based shell command orchestration system for Node.js. Execute shell commands in sequence, parallel, or as background processes with built-in support for JSON operations, HTTP requests, file management, text transformations, and more. All builtin operations write to a dedicated runtime context ($) for clean, collision-free data management.

How It Works

At its core, @fnet/shell-flow accepts a configuration object that specifies the commands to run, the working environment, and error handling strategies. Commands can be executed in sequence, parallel, or as background processes. The process manager integrated into the system ensures that all spawned processes are tracked and can be cleanly terminated if necessary. Additionally, it supports the use of templates for environment variables, enabling dynamic command configurations based on the user's context.

Key Features

Core Execution Modes

• Sequential Command Execution: Run commands one after the other, halting on errors if required.
• Parallel Command Execution: Execute multiple commands simultaneously for increased efficiency.
• Background Execution: Fork commands to run in the background, allowing the main process to continue without waiting.
• Script Mode: Execute a sequence of commands within a single shared shell session.

Expression-Based Builtins

• JSON Operations: Parse, stringify, and extract JSON data (json::parse, json::get, json::stringify)
• HTTP Requests: Make GET, POST, PUT, DELETE requests (http::get, http::post)
• File Operations: Read, write, copy, delete files (file::read, file::write)
• Text Transformations: Uppercase, lowercase, trim, replace, split, join (txt::upper, txt::lower)
• Encoding/Hashing: Base64, URL encoding, SHA256, MD5 (encode::base64, hash::sha256)
• Time Operations: Timestamps, formatting, parsing (time::now, time::format)
• Capture & Retry: Capture command output to context (capture::) and retry with exponential backoff (retry::)

Advanced Features

• Exit Code Management: Always returns an exit code for proper shell orchestration and CI/CD integration
• Runtime Context ($): Dedicated namespace for builtin results, prevents naming collisions
• Template Variables: Dynamic value substitution with {{variable}} syntax
• Error Handling: Customizable policies: stop, continue, throw
• Output Capture: Store and access command outputs for further processing or logging
• Environment Management: Flexible per-command or global environment variable configuration
• Composable Expressions: Chain processors for complex workflows (e.g. retry::3 wrapping capture::logs)

Documentation

For comprehensive documentation, examples, and API reference, see fnet/how-to.md.

Conclusion

@fnet/shell-flow provides a powerful, expression-based tool for managing shell command execution within a Node.js application. It allows developers to streamline their workflow by running and controlling multiple commands efficiently, with robust error handling, output capturing, and built-in operations for JSON, HTTP, files, text transformations, and more. This makes it suitable for automation scripts, build systems, and various development tasks requiring programmatic shell command executions.

Developer Guide for @fnet/shell-flow

Overview

The @fnet/shell-flow library provides developers with a powerful, expression-based shell command orchestration system. Execute shell commands in sequence, parallel, or as background processes with built-in support for JSON operations, HTTP requests, file management, data transformations, and more. All builtin operations write to a dedicated runtime context ($) for clean, collision-free data management.

Installation

npm install @fnet/shell-flow
# or
yarn add @fnet/shell-flow

Key Features

Core Execution Modes

• Sequential Command Execution - Run commands one after the other
• Parallel Command Execution - Execute multiple commands simultaneously
• Background Execution (Fork) - Run long-running processes in background
• Script Mode - Execute commands in a single shell session

Expression-Based Builtins

• JSON Operations - Parse, stringify, and extract JSON data (json::parse, json::get)
• HTTP Requests - Make GET, POST, PUT, DELETE requests (http::get, http::post)
• File Operations - Read, write, copy, delete files (file::read, file::write)
• Text Transformations - Uppercase, lowercase, trim, replace, split, join (txt::*)
• Encoding/Hashing - Base64, URL encoding, SHA256, MD5 (encode::*, hash::*)
• Time Operations - Timestamps, formatting, parsing (time::now, time::format)
• Assertions - Validate values, files, and conditions in workflows (assert::equal, assert::exists)
• Prompts - Interactive user input with confirm and text types (prompt::confirm, prompt::text)
• Environment Variables - Read, set, check, list, and delete env vars at runtime (env::get, env::set)
• Capture & Retry - Capture command output and retry with backoff (capture::, retry::)
• Array Iteration - Loop over parsed arrays and run commands per item (each::)
• Pipe - Chain step outputs sequentially, mixing shell commands and builtins (pipe::)

Advanced Features

• Exit Code Management - Always returns exit code for proper shell orchestration and CI/CD integration
• Runtime Context ($) - Dedicated namespace for builtin results, prevents naming collisions
• Template Variables - Dynamic value substitution with {{variable}} syntax
• Conditional Execution - Skip steps based on runtime conditions (when)
• Error Handling - Customizable policies: stop, continue, throw
• Timeout - Time limits for steps, parallel, and fork groups
• Signal Handling - Graceful SIGINT/SIGTERM handling with automatic child process cleanup
• Output Capture - Store and access command outputs for processing
• Environment Management - Flexible environment variable configuration
• Composable Expressions - Nest expressions for complex workflows

Core Types

CommandGroup

{
  steps?: string[];           // Array of sequential commands
  parallel?: string[];        // Array of parallel commands
  fork?: string[];            // Array of background commands
  filemap?: object;           // Filemap configuration object
  onError?: "stop" | "continue" | "throw";  // Error handling policy
  env?: Record<string, any>;  // Environment variables
  wdir?: string;              // Working directory
  captureName?: string;       // Name to capture output
  useScript?: boolean;        // Whether to execute in script mode
  timeout?: number;            // Timeout in seconds (0 = disabled)
  retry?: boolean | {         // Optional retry config
    attempts?: number;        // default 3
    delay?: number;           // default 1000ms
    factor?: number;          // default 2
    maxDelay?: number;        // default 30000ms
    codes?: number[];         // default [1]
  };
}

Input Configuration

{
  commands?: (string | CommandGroup)[];  // Sequential commands
  parallel?: (string | CommandGroup)[];  // Parallel commands
  fork?: (string | CommandGroup)[];      // Background commands
  onError?: "stop" | "continue" | "throw";  // Global error policy
  env?: Record<string, any>;             // Global environment variables
  wdir?: string;                         // Global working directory (defaults to process.cwd())
  context?: Record<string, any>;         // Template context object
  retry?: boolean | {                    // Optional global retry config
    attempts?: number;
    delay?: number;
    factor?: number;
    maxDelay?: number;
    codes?: number[];
  };
  gracefulTimeout?: number;              // Graceful termination window in ms (default: 1500)
  processManager?: ProcessManager;       // External ProcessManager for shared lifecycle
}

Output

The library returns a result object containing execution metadata and captured data:

{
  exitCode: number;                          // Final exit code (0 = success, non-zero = error/manual exit)
  $?: Record<string, any>;                   // Runtime context with builtin operation results
  [captureName]?: {                          // Captured command outputs (if any)
    items: CaptureResult[];
  };
  error?: {                                  // Last error details (if any)
    message: string;
    command: string;
    code: number;
    onError: string;
  };
  errors?: Array<{...}>;                     // All errors that occurred
}

CaptureResult

{
  stdout: string;    // Command's standard output
  stderr: string;    // Command's standard error
  code: number;      // Exit code
}

Basic Usage

Sequential Commands

import shellFlow from '@fnet/shell-flow';

await shellFlow({
  commands: [
    'echo "First command"',
    'echo "Second command"'
  ]
});

Parallel Commands

await shellFlow({
  parallel: [
    'npm run test:unit',
    'npm run test:integration'
  ]
});

Background Processes

await shellFlow({
  fork: [
    'npm run watch',
    'npm run dev-server'
  ]
});

Advanced Features

Command Groups

await shellFlow({
  commands: [
    {
      steps: [
        'npm install',
        'npm run build'
      ],
      onError: 'stop',
      captureName: 'build_output'
    }
  ]
});

Mixed Execution Modes

await shellFlow({
  commands: [
    'echo "Starting build process"',
    {
      parallel: [
        'npm run test:unit',
        'npm run test:integration'
      ]
    }
  ],
  fork: [
    'npm run watch:css',
    'npm run watch:js'
  ]
});

Output Capture

Non-script steps capture produce an items array:

const result = await shellFlow({
  commands: [
    {
      steps: ['echo "Hello World"', 'echo "Second"'],
      captureName: 'greeting'
    }
  ]
});

console.log(result.greeting.items[0].stdout);  // First command stdout
console.log(result.greeting.items[0].stderr);  // First command stderr
console.log(result.greeting.items[0].code);    // First command exit code

Script mode (useScript: true) produces a single capture object:

const result = await shellFlow({
  commands: [
    {
      steps: ['echo "Hello World"', 'echo "Second"'],
      useScript: true,
      captureName: 'greeting'
    }
  ]
});

console.log(result.greeting.stdout);  // Combined script stdout
console.log(result.greeting.stderr);  // Combined script stderr
console.log(result.greeting.code);    // Script exit code

Environment Variables

await shellFlow({
  commands: [
    {
      steps: ['npm run build'],
      env: {
        NODE_ENV: 'production'
      }
    }
  ],
  env: {
    CI: 'true'
  }
});

Working Directory

await shellFlow({
  commands: [
    {
      steps: ['npm run build'],
      wdir: './packages/app'
    }
  ],
  wdir: '/project/root'  // Global working directory
});

Script Mode

await shellFlow({
  commands: [
    {
      steps: [
        'set -e',
        'echo "Starting build"',
        'npm run build'
      ],
      useScript: true,
      captureName: 'build_log'
    }
  ]
});

Expression Syntax

The library supports powerful expression-based commands using the processor::operation::contextName syntax. Expressions enable advanced operations like JSON parsing, HTTP requests, file operations, and more - all with automatic result storage in the runtime context ($).

Basic Expression Format

commands:
  - capture::logs: npm run test
  - retry::3: curl https://api.example.com
  - json::parse::data: "{{response}}"
  - txt::upper::result: "hello world"

Composable Expressions

Expressions can be nested for powerful workflows:

commands:
  - retry::3:
      capture::logs: npm run test
  - json::parse::data: "{{logs.items[0].stdout}}"
  - echo: "Status: {{$.data.status}}"

Runtime Context ($)

All expression-based builtins write their results to the runtime context ($), which is separate from user-defined context variables. This prevents naming collisions and provides a clean namespace for builtin results.

const result = await shellFlow({
  commands: [
    { 'http::get::response': 'https://api.example.com/users' },
    { 'json::parse::users': '{{$.response.body}}' },
    { echo: 'First user: {{$.users[0].name}}' }
  ]
});

// Result includes both capture data and runtime context
console.log(result);
// {
//   $: {
//     response: { status: 200, body: "..." },
//     users: [{ name: "John" }, ...]
//   }
// }

Key Points:

• User context: {{varName}}
• Runtime context: {{$.varName}}
• Final result always includes $ object with all builtin results

Template Variables

The library supports template variable substitution using context objects. Templates use the {{variable}} syntax with several features:

await shellFlow({
  commands: [
    'echo "Hello {{user.name}}"',
    'mkdir -p {{paths.output}}'
  ],
  context: {
    user: {
      name: 'John'
    },
    paths: {
      output: './dist'
    }
  }
});

Template Features

1. Nested Object Access

await shellFlow({
  commands: ['npm config set registry {{config.npm.registry}}'],
  context: {
    config: {
      npm: {
        registry: 'https://registry.npmjs.org'
      }
    }
  }
});

1. Array Access

await shellFlow({
  commands: ['deploy {{services[0].name}}'],
  context: {
    services: [
      { name: 'api' },
      { name: 'web' }
    ]
  }
});

1. Default Values

await shellFlow({
  commands: [
    'NODE_ENV={{env || production}}',
    'PORT={{port || 3000}}'
  ],
  context: {
    env: 'development'
  }
});

1. Strict Mode (use {{API_KEY!}})

await shellFlow({
  commands: [
    'curl -H "Authorization: {{API_KEY!}}" {{url}}'
  ],
  context: {
    API_KEY: process.env.API_KEY,
    url: 'https://api.example.com'
  }
});

1. Conditional Value (presence-based)

await shellFlow({
  commands: [
    'npm run {{isProd ? build:prod}}'
  ],
  context: {
    isProd: true
  }
});

Control Commands

The library provides built-in control commands for common operations:

Echo Command

await shellFlow({
  commands: [
    { echo: "Starting process..." },
    { echo: "User: {{user.name}}" }
  ],
  context: {
    user: { name: "John" }
  }
});

Sleep Command

await shellFlow({
  commands: [
    { echo: "Starting..." },
    { sleep: 2 },  // Wait for 2 seconds
    { sleep: "{{delay}}" }  // Dynamic delay from context
  ],
  context: {
    delay: 1
  }
});

Exit Command

await shellFlow({
  commands: [
    { echo: "Running tests..." },
    "npm test",
    { exit: "{{testsPassed ? 0 : 1}}" }  // Dynamic exit code
  ],
  context: {
    testsPassed: true
  }
});

Filemap Command

The filemap command allows you to map files from source directories to target directories with support for templating, symlinks, and multiple output formats.

await shellFlow({
  commands: [
    { echo: "Starting file mapping..." },
    { filemap: {
        target: "dist",
        sources: [
          {
            source: "templates",
            target: ".",
            symlink: false
          },
          {
            source: "assets",
            target: "assets",
            symlink: true
          }
        ]
      }
    },
    { echo: "File mapping completed" }
  ],
  context: {
    app: {
      name: "My App",
      version: "1.0.0"
    }
  }
});

Filemap Configuration

The filemap command accepts an object (not a file path string) with the following properties:

• target (required): The target directory path where processed files will be placed.
• sources (required): An array of source objects with the following properties:
  • source (required): The source from which to fetch files, supports multiple protocols and providers.
  • target (optional): Target subdirectory within the main target directory for output.
  • context (optional): Context data to be used with the templating engine for dynamic content rendering.
  • engine (optional): Template engine to use, defaults to 'njk' (Nunjucks).
  • symlink (optional): Determines whether to create symbolic links instead of copying files (default: false).
  • provider (optional): Custom provider configurations that can override defaults.
• output (optional): Output format; options include 'file', 'stdout', or 'json' (default: 'file').
• provider (optional): Default provider configurations for various source types.

Pause Command

The pause command pauses execution and waits for the user to press Enter before continuing. This is particularly useful for interactive scripts or when running background processes that you want to keep running until manually terminated.

await shellFlow({
  commands: [
    { echo: "Starting background processes..." },
    { fork: [
        "npm run server",
        "npm run watch"
      ]
    },
    { echo: "Background processes started" },
    { pause: "Press Enter to continue..." },
    { echo: "Cleaning up and exiting" }
  ]
});

The pause command can be used in two ways:

// With a custom message
{ pause: "Press Enter to continue..." }

// With a default message ("Press Enter to continue...")
{ pause: true }

Control commands must contain only their respective key (echo, sleep, exit, pause, or filemap). The exit command accepts values 0-127, and sleep accepts non-negative numbers for seconds.

Prompt Expression

The prompt:: expression asks interactive questions and stores answers in the runtime context ($). Unlike pause (which just waits), prompt captures user decisions for use with when and other builtins.

await shellFlow({
  commands: [
    // Shorthand: defaults to confirm (y/n)
    { 'prompt::approved': 'Deploy to production?' },
    // $.approved → true or false

    // Explicit confirm with default
    { 'prompt::confirm::deploy': {
        message: 'Deploy to production?',
        default: false
      }
    },

    // Text input
    { 'prompt::text::username': {
        message: 'Enter your name:'
      }
    },
    { echo: 'Hello, {{$.username}}!' },

    // Text with default value
    { 'prompt::text::branch': {
        message: 'Target branch:',
        default: 'main'
      }
    },

    // Compose with when for branching
    { 'prompt::should_deploy': 'Continue with deployment?' },
    { when: '{{$.should_deploy}}',
      steps: [
        { 'prompt::text::target': {
            message: 'Which environment?',
            default: 'staging'
          }
        },
        { echo: 'Deploying to {{$.target}}...' }
      ]
    }
  ]
});

Prompt Types:

• prompt::<name>: <question> — Shorthand, defaults to confirm (y/n)
• prompt::confirm::<name>: <question_or_config> — Yes/no question, returns true/false
• prompt::text::<name>: <question_or_config> — Free text input, returns string

Config Object:

{
  message: string;       // The question to display
  default?: any;         // Default value (shown in prompt, used if Enter pressed)
  abort?: boolean;       // If true, throw ShellError when user answers "no" (default: false)
}

Abort mode:

By default, prompt::confirm stores the result and continues — the developer decides what to do via when. With abort: true, a "no" answer throws a ShellError, stopping the workflow (respects onError policy).

await shellFlow({
  commands: [
    // Default: result in $, developer branches with when
    { 'prompt::approved': 'Include tests?' },
    { when: '{{$.approved}}', steps: [{ echo: 'Running tests...' }] },

    // Abort mode: "no" stops the workflow
    { 'prompt::confirm::danger': {
        message: 'DELETE all data?',
        abort: true
      }
    }
  ]
});

Relationship to pause:

• pause = "Wait for Enter" (no result, no decision, no $ context)
• prompt = "Ask a question" (result in $, composable with when)

Expression-Based Builtins

The library provides powerful expression-based builtins that write results to the runtime context ($). All builtins follow the format: processor::operation::contextName: input

Capture Expression

Capture command output and store it in the runtime context ($). The captured data is accessible both via result.$.name and directly as result.name on the returned result object.

const result = await shellFlow({
  commands: [
    { 'capture::logs': 'npm run test' },
    { echo: 'Test output: {{$.logs.items[0].stdout}}' }
  ]
});

// Both access paths are equivalent:
console.log(result.$.logs);   // via runtime context
console.log(result.logs);     // via root result

Retry Expression

Retry commands with automatic backoff:

await shellFlow({
  commands: [
    { 'retry::3': 'curl https://api.example.com' },
    { 'retry::5': {
        'capture::response': 'curl https://api.example.com'
      }
    }
  ]
});

Each Expression

Iterate over arrays and run a block of commands for each item. The current item is injected into the template context under the name you choose.

Syntax: each::<itemName>::<arrayPath>

await shellFlow({
  commands: [
    { 'json::parse::services': '[{"name":"api","port":3001},{"name":"web","port":3002}]' },
    { 'each::service::$.services': [
        { echo: 'Deploying {{service.name}} on port {{service.port}}...' },
        'npm run deploy -- --service={{service.name}}'
      ]
    }
  ]
});
// Output:
// Deploying api on port 3001...
// Deploying web on port 3002...

Key behaviors:

• The array must already be resolved in the runtime context ($) or user context
• Each item is available as {{itemName}} (or {{itemName.property}} for objects) inside the body
• An empty array is a no-op — the body is simply skipped
• All builtins (txt::, json::, capture::, etc.) work inside the loop body
• onError policy applies per-iteration, consistent with the rest of the pipeline
• Loop-scoped variables do not leak outside the each:: block

Pipe Expression

Chain step outputs sequentially — each step's output feeds into the next. Mixes shell commands and builtin expressions in a single pipeline.

Syntax: pipe::<contextName>

await shellFlow({
  commands: [
    // Shell commands: each step's stdout feeds the next
    { 'pipe::version': [
        'node --version'
      ]
    },
    { echo: 'Version: {{$.version}}' },

    // Mix shell commands and builtins
    { 'pipe::result': [
        'echo "hello world"',
        'txt::upper::_piped',
        'encode::base64::_piped'
      ]
    },
    { echo: 'Encoded: {{$.result}}' },

    // Shell → JSON parse
    { 'pipe::data': [
        'echo \'{"name":"John","age":30}\'',
        'json::parse::_piped'
      ]
    },
    { echo: 'Name: {{$.data.name}}' }
  ]
});

How it works:

• Steps execute sequentially within the pipe
• Shell commands: stdout is captured and passed as input to the next step (trailing newline trimmed)
• Builtin expressions: receive the previous step's output as input, store result in $ context
• The final value (last stdout or last builtin result) is stored in $.contextName
• Intermediate builtin results are also accessible in $ under their own context names

Key behaviors:

• Shell commands and builtins can be freely mixed in any order
• An error in any pipe step stops the pipe (respects onError policy)
• Works with all builtin processors: txt::, json::, encode::, hash::, etc.

JSON Operations

Parse, stringify, and extract JSON data:

await shellFlow({
  commands: [
    // Parse JSON string
    { 'json::parse::data': '{"user":{"name":"John","age":30}}' },
    { echo: 'Name: {{$.data.user.name}}' },

    // Extract JSON path
    { 'json::get::name': '$.data.user.name' },
    { echo: 'Extracted: {{$.name}}' },

    // Stringify object — template references to objects (e.g. {{$.data}}) are
    // automatically resolved to the actual object before stringification
    { 'json::stringify::output': '{{$.data}}' }
  ]
});

JSON Operations:

• json::parse::<name>: <json_string> - Parse JSON to object
• json::stringify::<name>: <template_or_value> - Convert object to JSON string; template references like {{$.name}} that resolve to objects are handled correctly
• json::get::<name>: <json_path> - Extract value using JSONPath

Text Operations

Text transformation operations using the txt:: processor (note the X - cross in the middle!):

await shellFlow({
  commands: [
    // Uppercase
    { 'txt::upper::upper': 'hello world' },
    { echo: '{{$.upper}}' },  // HELLO WORLD

    // Lowercase
    { 'txt::lower::lower': 'HELLO WORLD' },

    // Trim whitespace
    { 'txt::trim::trimmed': '  hello  ' },

    // Replace text
    { 'txt::replace::result': {
        input: 'hello world',
        search: 'world',
        replace: 'universe'
      }
    },

    // Split string to array
    { 'txt::split::items': {
        input: 'a,b,c',
        delimiter: ','
      }
    },

    // Join array to string
    { 'txt::join::result': {
        input: ['a', 'b', 'c'],
        delimiter: ','
      }
    }
  ]
});

Text Operations:

• txt::upper::<name>: <text> - Convert to uppercase (alias: uppercase)
• txt::lower::<name>: <text> - Convert to lowercase (alias: lowercase)
• txt::trim::<name>: <text> - Trim whitespace
• txt::replace::<name>: {input, search, replace} - Replace text (supports regex)
• txt::split::<name>: {input, delimiter} - Split string to array
• txt::join::<name>: {input, delimiter} - Join array to string

File Operations

Read, write, and manage files:

await shellFlow({
  commands: [
    // Write file
    { 'file::write::write_result': {
        path: '/tmp/data.txt',
        content: 'Hello World'
      }
    },

    // Read file
    { 'file::read::content': '/tmp/data.txt' },
    { echo: 'Content: {{$.content}}' },

    // Check if file exists
    { 'file::exists::check': '/tmp/data.txt' },
    { echo: 'Exists: {{$.check}}' },

    // Copy file
    { 'file::copy::copy_result': {
        source: '/tmp/data.txt',
        destination: '/tmp/backup.txt'
      }
    },

    // List directory
    { 'file::list::files': '/tmp' },

    // Delete file
    { 'file::delete::delete_result': '/tmp/data.txt' }
  ]
});

File Operations:

• file::read::<name>: <path> - Read file content
• file::write::<name>: {path, content} - Write content to file
• file::exists::<name>: <path> - Check if file exists (returns boolean)
• file::delete::<name>: <path> - Delete file
• file::copy::<name>: {source, destination} - Copy file
• file::list::<name>: <path> - List directory contents

HTTP Operations

Make HTTP requests:

await shellFlow({
  commands: [
    // GET request
    { 'http::get::response': 'https://api.example.com/users' },
    { echo: 'Status: {{$.response.status}}' },
    { echo: 'Body: {{$.response.body}}' },

    // POST request
    { 'http::post::create_result': {
        url: 'https://api.example.com/users',
        body: { name: 'John', email: 'john@example.com' },
        headers: { 'Content-Type': 'application/json' }
      }
    },

    // PUT request
    { 'http::put::update_result': {
        url: 'https://api.example.com/users/1',
        body: { name: 'John Updated' }
      }
    },

    // DELETE request
    { 'http::delete::delete_result': 'https://api.example.com/users/1' }
  ]
});

HTTP Operations:

• http::get::<name>: <url> or {url, headers}
• http::post::<name>: {url, body, headers?}
• http::put::<name>: {url, body, headers?}
• http::delete::<name>: <url> or {url, headers}

Response Format:

{
  status: 200,
  statusText: "OK",
  headers: { ... },
  body: "..." // Response body as string
}

Encoding Operations

Encode, decode, and hash data:

await shellFlow({
  commands: [
    // Base64 encode
    { 'encode::base64::encoded': 'hello world' },
    { echo: 'Encoded: {{$.encoded}}' },

    // Base64 decode
    { 'decode::base64::decoded': '{{$.encoded}}' },
    { echo: 'Decoded: {{$.decoded}}' },

    // URL encode
    { 'encode::url::url_encoded': 'hello world' },
    { echo: 'URL: {{$.url_encoded}}' },

    // URL decode
    { 'decode::url::url_decoded': '{{$.url_encoded}}' },

    // SHA256 hash
    { 'hash::sha256::hash': 'password123' },
    { echo: 'Hash: {{$.hash}}' },

    // MD5 hash
    { 'hash::md5::md5_hash': 'password123' }
  ]
});

Encoding Operations:

• encode::base64::<name>: <text> - Base64 encode
• decode::base64::<name>: <text> - Base64 decode
• encode::url::<name>: <text> - URL encode
• decode::url::<name>: <text> - URL decode
• hash::sha256::<name>: <text> - SHA256 hash (hex)
• hash::md5::<name>: <text> - MD5 hash (hex)

Time Operations

Work with timestamps and dates:

await shellFlow({
  commands: [
    // Get current timestamp
    { 'time::now::timestamp': null },
    { echo: 'Now: {{$.timestamp}}' },

    // Format timestamp
    { 'time::format::formatted': {
        timestamp: '{{$.timestamp}}',
        format: 'iso'  // iso, date, time, locale, or custom
      }
    },
    { echo: 'Formatted: {{$.formatted}}' },

    // Parse date string
    { 'time::parse::parsed': '2024-10-14' },
    { echo: 'Parsed: {{$.parsed}}' },

    // Add time
    { 'time::add::future': {
        timestamp: '{{$.timestamp}}',
        amount: 3600000,  // 1 hour in ms
        unit: 'milliseconds'
      }
    },

    // Calculate difference
    { 'time::diff::difference': {
        start: '{{$.timestamp}}',
        end: '{{$.future}}',
        unit: 'hours'
      }
    }
  ]
});

Time Operations:

• time::now::<name>: - Get current timestamp (milliseconds)
• time::format::<name>: {timestamp, format} - Format timestamp
  • Formats: iso, date, time, locale, or custom pattern
• time::parse::<name>: <date_string> - Parse date to timestamp
• time::add::<name>: {timestamp, amount, unit} - Add time
  • Units: milliseconds, seconds, minutes, hours, days
• time::diff::<name>: {start, end, unit} - Calculate difference

Assert Operations

Validate values, compare results, and check conditions within workflows. Assertions write their results to the runtime context ($) and throw a ShellError on failure — making them composable with onError policies.

await shellFlow({
  commands: [
    // Equality check
    { 'assert::equal::check': ['hello', 'hello'] },
    { echo: 'Passed: {{$.check.passed}}' },

    // Inequality check
    { 'assert::not_equal::check2': ['hello', 'world'] },

    // String contains
    { 'assert::contains::check3': ['hello world', 'world'] },

    // File existence
    { 'assert::exists::check4': './package.json' },

    // Truthy value
    { 'assert::truthy::check5': 'yes' },

    // Numeric comparisons
    { 'assert::gt::check6': [10, 5] },
    { 'assert::gte::check7': [10, 10] },
    { 'assert::lt::check8': [3, 7] },
    { 'assert::lte::check9': [5, 5] }
  ]
});

Assert Operations:

• assert::equal::<name>: [actual, expected] - Strict equality (string comparison)
• assert::not_equal::<name>: [actual, unexpected] - Inequality check
• assert::contains::<name>: [haystack, needle] - String or array contains
• assert::exists::<name>: <path> - File existence check
• assert::truthy::<name>: <value> - Truthy check (treats "false", "0", "", "null", "undefined" as falsy)
• assert::gt::<name>: [actual, expected] - Greater than (numeric)
• assert::gte::<name>: [actual, expected] - Greater than or equal (numeric)
• assert::lt::<name>: [actual, expected] - Less than (numeric)
• assert::lte::<name>: [actual, expected] - Less than or equal (numeric)

Result Format:

{
  passed: true,    // Whether the assertion passed
  actual: "hello", // The actual value (key varies by operation)
  expected: "hello" // The expected value
}

Using with other builtins:

Assertions are most powerful when combined with capture, JSON parsing, and other builtins for self-validating workflows:

await shellFlow({
  commands: [
    // Capture and validate command output
    { 'capture::version': 'node --version' },
    { 'assert::contains::ver_check': ['{{$.version.items.0.stdout}}', 'v'] },

    // Parse JSON and validate fields
    { 'json::parse::data': '{"status":"ok","count":42}' },
    { 'assert::equal::status': ['{{$.data.status}}', 'ok'] },
    { 'assert::gt::count': ['{{$.data.count}}', 0] },

    // Handle expected failures gracefully
    {
      steps: [
        { 'assert::equal::will_fail': ['actual', 'different'] }
      ],
      onError: 'continue'
    }
  ]
});

Environment Variable Operations

Read, set, check, list, and delete environment variables at runtime. Unlike the static env key on command groups, env:: operations are dynamic — they can use values from the $ runtime context and make changes visible to subsequent shell commands.

await shellFlow({
  commands: [
    // Read an env var into $ context
    { 'env::get::home': 'HOME' },
    { echo: 'Home: {{$.home}}' },

    // Set an env var (visible to subsequent commands)
    { 'env::set::result': ['MY_VAR', 'my_value'] },

    // Set from $ context (e.g., captured output)
    { 'capture::ver': 'node --version' },
    { 'env::set::result2': ['NODE_VER', '{{$.ver.items.0.stdout}}'] },

    // Check if env var exists
    { 'env::exists::check': 'MY_VAR' },
    { echo: 'Exists: {{$.check}}' },  // true

    // List env vars matching a pattern
    { 'env::list::vars': 'MY_*' },

    // Delete an env var
    { 'env::delete::del': 'MY_VAR' },
    { echo: 'Deleted: {{$.del.deleted}}' }  // true
  ]
});

Environment Operations:

• env::get::<name>: <varName> - Read env var into $ context (returns null if not set)
• env::set::<name>: [varName, value] or {name, value} - Set env var in process.env
• env::exists::<name>: <varName> - Check if env var is defined (returns boolean)
• env::list::<name>: <pattern> - List env vars matching a glob pattern (supports * wildcard)
• env::delete::<name>: <varName> - Remove env var from process.env

Key difference from static env: The static env key on command groups sets env vars before commands run. env:: operations happen during execution, can read values dynamically, and can propagate captured output or computed values to subsequent shell commands.

Exit Control

The library supports controlled process termination using the exit command. The exit code can be specified directly or using template variables:

// Simple exit
await shellFlow({
  commands: [
    'echo "Done"',
    { exit: 0 }  // Success exit
  ]
});

// Using template variables
await shellFlow({
  commands: [
    'npm run test',
    { exit: '{{testResult.code || 0}}' }
  ],
  context: {
    testResult: { code: 1 }
  }
});

// With conditional logic
await shellFlow({
  commands: [
    { parallel: ['server', 'watch'] },
    'run-tests',
    { exit: '{{isCI && testsFailed ? 1 : 0}}' }
  ],
  context: {
    isCI: true,
    testsFailed: false
  }
});

The exit command will:

1. Gracefully terminate all running processes
2. Wait for processes to clean up (with 5s timeout)
3. Exit with the specified code (0-127)

Exit Code Handling

The library always returns an exitCode property that reflects the final execution status. This is critical for shell orchestration and CI/CD pipelines.

Exit Code Scenarios

1. Successful Execution (exitCode: 0)

const result = await shellFlow({
  commands: [
    'echo "Test 1"',
    'echo "Test 2"'
  ]
});

console.log(result.exitCode);  // 0 (success)
console.log(result.$);          // Runtime context with builtin results

2. Command Failure with onError: "stop" (default)

const result = await shellFlow({
  onError: 'stop',  // Default policy
  commands: [
    'echo "Test 1"',
    'bad-command',      // ← Fails with exit code 127
    'echo "Test 2"'     // ← Never executes
  ]
});

console.log(result.exitCode);  // 127 (command not found)
console.log(result.error);      // { message, command, code: 127, onError: 'stop' }

Important: With onError: "stop", execution halts at the first error and the exit code is set to the failing command's exit code.

3. Command Failure with onError: "continue"

const result = await shellFlow({
  onError: 'continue',
  commands: [
    'echo "Test 1"',
    'bad-command',      // ← Fails but execution continues
    'echo "Test 2"'     // ← Still executes
  ]
});

console.log(result.exitCode);  // 0 (continue policy ignores errors)
console.log(result.errors);     // Array of all errors that occurred

Important: With onError: "continue", errors are collected but the exit code remains 0.

4. Manual Exit Code

const result = await shellFlow({
  commands: [
    'echo "Running tests..."',
    { exit: 42 },       // ← Manual exit code
    'echo "Never runs"'
  ]
});

console.log(result.exitCode);  // 42 (manual exit)

Important: The exit command accepts values 0-127 and immediately terminates execution after cleaning up all processes.

Using Exit Codes in Parent Processes

The exit code is designed to be used by parent processes (like Flownet CLI or CI/CD systems):

// In your orchestration tool or CI/CD pipeline
const result = await shellFlow({ commands: [...] });

// Set the process exit code based on shell-flow result
process.exitCode = result.exitCode;

// Or exit immediately
process.exit(result.exitCode);

Exit Code Summary

Scenario	Exit Code	Execution Behavior
All commands succeed	0	Normal completion
Command fails + onError: "stop"	Command's exit code (e.g., 127)	Stops at first error
Command fails + onError: "continue"	0	Continues, collects errors
Manual exit: N	N (0-127)	Immediate termination
onError: "throw"	N/A	Throws exception

Timeout

The timeout option sets a time limit (in seconds) on steps, parallel, or fork groups. If execution exceeds the limit, a ShellError is thrown with exit code 124 (consistent with GNU timeout).

// Timeout on sequential steps
await shellFlow({
  commands: [
    {
      timeout: 10,
      steps: [
        'curl https://api.example.com/data',
        'npm run process'
      ]
    }
  ]
});

// Timeout on parallel group
await shellFlow({
  commands: [
    {
      timeout: 30,
      parallel: [
        'curl https://api1.com',
        'curl https://api2.com'
      ]
    }
  ]
});

// Combined with retry — each attempt is time-limited
await shellFlow({
  commands: [
    {
      timeout: 5,
      retry: 3,
      steps: ['curl https://flaky-service.com/health']
    }
  ]
});

// Timeout with onError: continue — logs timeout, moves on
await shellFlow({
  commands: [
    {
      timeout: 2,
      steps: ['sleep 100'],
      onError: 'continue'
    },
    { echo: 'Continued after timeout' }
  ]
});

Behavior:

• Value is in seconds (consistent with sleep)
• timeout: 0 disables timeout (no time limit)
• On timeout: throws ShellError with code 124 and message "Command timed out after Ns"
• Respects onError policy: stop halts execution, continue moves to next step, throw re-throws
• Wraps retry when both are present: timeout applies to the full retry cycle
• Works with steps, parallel, and fork groups

Process Management & Signal Handling

shell-flow tracks every child process it spawns and cleans them up automatically on workflow completion, errors, or OS signals. This is especially important for fork (background) and parallel workflows that spawn long-running processes like servers.

Automatic Lifecycle

Every executeCommand call registers its child process with a ProcessManager. When the workflow completes (success or error) or receives SIGINT/SIGTERM, all tracked processes are terminated:

1. Send SIGTERM to all alive processes (graceful shutdown request)
2. Poll every 250ms, waiting up to gracefulTimeout ms for processes to exit
3. Any survivors are force-killed with SIGKILL
4. Uses tree-kill to terminate entire process trees (not just the direct child)

This means you can safely fork servers, CLI tools, or any long-running process — pressing CTRL+C will terminate them all, including child processes they spawned.

gracefulTimeout

Controls how long to wait for graceful shutdown before escalating to SIGKILL.

await shellFlow({
  gracefulTimeout: 5000,  // Wait up to 5s for SIGTERM to work (default: 1500ms)
  fork: [
    'node slow-shutdown-server.js'  // Needs time to close connections
  ]
});

Defaults:

• Process cleanup during normal flow: 1500ms
• Signal-triggered cleanup: gracefulTimeout (configurable)
• Workflow exit (dispose): 500ms
• Exit command cleanup: 2000ms (hardcoded for safety)

Signal Handling

The ProcessManager registers handlers for:

• SIGINT (CTRL+C) → graceful cleanup, then exit code 130
• SIGTERM → graceful cleanup, then exit code 143
• uncaughtException → emergency cleanup, exit 1
• unhandledRejection → emergency cleanup, exit 1

After cleanup, the original signal is re-sent to the process so parent processes (shells, orchestrators, CI runners) observe the expected exit behavior.

Fork with Signal Handling

await shellFlow({
  commands: [
    { echo: 'Starting development environment...' },
    {
      fork: [
        'node server.js',
        'npm run watch',
        'npm run dev-client'
      ]
    },
    { echo: 'All services started. Press CTRL+C to stop.' },
    { pause: true }
    // CTRL+C → all 3 forked processes terminated cleanly
  ]
});

External ProcessManager

For advanced scenarios (e.g., wrapping shell-flow in another orchestrator), you can pass your own ProcessManager instance. This lets multiple shellFlow invocations share a lifecycle:

import shellFlow, { ProcessManager } from '@fnet/shell-flow';

const pm = new ProcessManager({ gracefulTimeout: 3000 });

try {
  // Multiple invocations share the same process tracking
  await shellFlow({ commands: [...], processManager: pm });
  await shellFlow({ commands: [...], processManager: pm });
} finally {
  await pm.dispose();  // Cleanup at the end
}

Key differences when using external ProcessManager:

• Signal handlers are added by the external PM, not per-invocation
• Cleanup happens only when the caller calls dispose()
• Multiple invocations can share process tracking for coordinated shutdown

Zombie Process Prevention

The ProcessManager is designed to prevent orphaned processes. All test scenarios in tests/signal-*.fnet and tests/test-ctrl-c-*.fnet verify that:

• fork + CTRL+C terminates all background processes
• parallel + CTRL+C terminates all parallel tasks
• Nested steps + CTRL+C terminates every level
• Process trees (servers that spawn workers) are fully cleaned up
• Bad-citizen processes that ignore SIGTERM get SIGKILL

To verify no zombies after a test:

# Should return nothing if cleanup worked
ps aux | grep -E '(sleep|node|bun)' | grep -v grep

Conditional Execution (when)

The when key on a command object enables conditional step execution. If the condition evaluates to false, the entire step (steps/parallel/fork) is silently skipped.

String Conditions (truthy check)

await shellFlow({
  commands: [
    // Truthy string — runs
    { when: 'yes', steps: [{ echo: 'This runs' }] },

    // Falsy string — skipped
    { when: 'false', steps: [{ echo: 'This is skipped' }] },

    // Template variable — evaluated at runtime
    { when: '{{$.has_config}}', steps: [{ echo: 'Config found' }] }
  ]
});

Falsy strings: "false", "0", "", "null", "undefined" (case-insensitive).

Object Conditions

await shellFlow({
  commands: [
    // Equality
    { when: { equal: ['{{$.mode}}', 'production'] },
      steps: [{ echo: 'Production mode' }] },

    // Negation
    { when: { not: '{{$.debug}}' },
      steps: [{ echo: 'Debug is off' }] },

    // String contains
    { when: { contains: ['{{$.version}}', 'v'] },
      steps: [{ echo: 'Version starts with v' }] },

    // Numeric comparisons
    { when: { gt: ['{{$.count}}', 0] },
      steps: [{ echo: 'Count is positive' }] },

    // File existence
    { when: { exists: './config.json' },
      steps: [{ echo: 'Config file found' }] }
  ]
});

Supported Conditions:

Condition	Syntax	Description
Truthy	when: "value"	String/boolean truthy check
Equal	when: { equal: [a, b] }	String equality
Not	when: { not: value }	Negation / falsy check
Contains	when: { contains: [haystack, needle] }	String or array contains
Greater than	when: { gt: [a, b] }	Numeric a > b
Greater or equal	when: { gte: [a, b] }	Numeric a >= b
Less than	when: { lt: [a, b] }	Numeric a < b
Less or equal	when: { lte: [a, b] }	Numeric a <= b
File exists	when: { exists: "path" }	File existence check

Behavior:

• when is evaluated after template processing (so $ context values are resolved)
• If condition is false, the entire step is skipped silently (no error)
• No else block — use a second when with not: for the opposite branch
• Works with all step types: steps, parallel, fork
• Composable with timeout, retry, onError, env, wdir

Combining with Other Builtins

await shellFlow({
  commands: [
    // Check file, then conditionally process
    { 'file::exists::has_data': './data.json' },
    { when: '{{$.has_data}}',
      steps: [
        { 'file::read::data': './data.json' },
        { 'json::parse::parsed': '{{$.data}}' },
        { echo: 'Loaded {{$.parsed.items.length}} items' }
      ]
    },

    // Capture and conditionally assert
    { 'capture::health': 'curl -s http://localhost:3000/health' },
    { when: { contains: ['{{$.health.items.0.stdout}}', 'ok'] },
      steps: [
        { echo: 'Service is healthy' }
      ]
    }
  ]
});

Error Handling

try {
  await shellFlow({
    commands: ['invalid-command'],
    onError: 'throw'  // 'stop' | 'continue' | 'throw'
  });
} catch (error) {
  console.log(error.message);    // Error description
  console.log(error.command);    // Failed command
  console.log(error.code);       // Exit code
  console.log(error.onError);    // Active error policy
}

Error handling and retry examples

• onError policies:

  • stop: stop current sequence on first error in that scope (sets exit code)
  • continue: continue execution, collect errors (exit code 0)
  • throw: throw immediately

• retry options (global or per group):

  • attempts: number of tries (default 3)
  • delay: initial delay in ms (default 1000)
  • factor: backoff multiplier (default 2)
  • maxDelay: cap on delay (default 30000)
  • codes: exit codes to retry on (default [1])

// Global retry, with per-group override
await shellFlow({
  onError: 'stop',
  retry: { attempts: 3, delay: 1000, factor: 2, maxDelay: 30_000, codes: [1] },
  commands: [
    {
      steps: [
        'may-fail-once',
        'then-run-next'
      ]
    },
    {
      // Override retry for this group only
      steps: ['another-maybe-failing-command'],
      retry: { attempts: 5, delay: 500 }
    },
    {
      // Continue on error without throwing/logging
      steps: ['bad-command'],
      onError: 'continue'
    }
  ]
});

Best Practices

1. Always check the exit code - Use result.exitCode to determine execution success in parent processes
2. Use expression syntax for builtin operations - Leverage json::, http::, file::, txt::, etc. instead of shell commands
3. Access builtin results via $.name - Runtime context prevents naming collisions
4. Use appropriate error handling policy - Choose stop, continue, or throw based on your use case
5. Capture output when needed - Use capture::name for command output processing
6. Use script mode for shell-specific features - Enable useScript: true for complex shell scripts
7. Group related commands - Use steps to organize sequential operations
8. Set environment variables at the most specific scope - Apply env at command, group, or global level
9. Use working directory correctly - Set wdir to ensure proper command context
10. Use fork for background processes - Long-running services should run in background
11. Use parallel for concurrent tasks - Execute independent tasks simultaneously
12. Compose expressions for complex workflows - Nest retry::, capture::, and other expressions
13. Use template variables for dynamic values - Leverage {{variable}} syntax with context

Limitations

• Nested parallel/fork operations in script mode are not supported
• Output capture is limited to sequential commands
• Some shell features may have platform-specific behavior

Support

For issues and feature requests, please visit our repository at GitHub.