Args-tokenizer NPM

args-tokenizer

args-tokenizer is a lightweight JavaScript library for parsing shell commands with arguments into an argv array. This makes it easy to work with command-line tools and libraries that expect an array format for arguments, such as tinyexec.

Features

Simple and intuitive API.
Handles quoted strings and escapes correctly.
Supports multiline input.
Ideal for parsing human-readable shell commands, especially curl-style commands.

Made at Webstudio, open source website builder.

Installation

Install args-tokenizer:

npm install args-tokenizer

Usage

Here's how you can use args-tokenizer to parse shell commands:

import { tokenizeArgs } from "args-tokenizer";

const args = tokenizeArgs(`ls -la "./src"`);
console.log(args); // ["ls", "-la", "./src"]

Multiline Input Support

args-tokenizer also supports multiline commands, such as:

const args = tokenizeArgs(`
  curl \\
    -X POST \\
    "https://my-url.com"
`);
console.log(args); // ["curl", "-X", "POST", "https://my-url.com"]

Example with `tinyexec`

One common use case is passing more human-readable commands into the tinyexec library:

import { tokenizeArgs } from "args-tokenizer";
import { x } from "tinyexec";

const [command, ...args] = tokenizeArgs("ls -la");
const result = await x(command, args);
console.log(result.stdout);

API

`tokenizeArgs(command: string, options: Options): string[]`

Parses a shell command string into an array of arguments. Properly handles:

Quoted strings (e.g., '"./path/to/file"').
Escaped characters (e.g., \").
Multiline commands (e.g., lines ending with \\).

Options

loose: If true, the tokenizer will not throw an error when closing quotes are missing. Default is false.

Examples

// Without loose option (default behavior)
// This will throw an error due to the missing closing quote
tokenizeArgs('command "arg1 arg2');

// With loose option enabled
const args = tokenizeArgs('command "arg1 arg2', { loose: true });
// ['command', 'arg1 arg2']