@zorse/drop NPM

drop

Drop is a small toolbox of common tools in WebAssembly to provide a POSIX-like environment independent of the host system and platform capabilities.

Drop is designed primarily as a toolbox for file IO and offline scripting.
Zorse, TypeScript, TypeScriptX (React), and JavaScript syntax are supported.

$ npm install --global @zorse/drop

$ drop node test.ts
Hello From WebAssembly!

$ drop ls
webpack.std.mjs    src                lib                LICENSE
webpack.all.mjs    package.json       docs               Cargo.toml
tsconfig.json      package-lock.json  dist               Cargo.lock
test.zx.mjs        out                build.zx.mjs       CREDITS.md
test.ts            node_modules       build.rs           CMakeLists.txt
target             modules.tar.gz     build
std                modules            README.md

$ drop zip archive.zip file.txt

Goals

Isomorphism: Supporting runtime execution everywhere under a single API.
Portability: Staying entirely in WebAssembly and WASI for max portability.
Familiarity: Not introducing any new concepts. Faithfulness to NodeJS APIs.

Development

API Documentation: docs

To run the formatter and apply the style: npm run format -- --write.
To run the linter and apply the rules: npm run lint -- --apply.
To run the tests: npm test.
To build you need:

Rust (on PATH)
EMCC (on PATH currently)
CMake and C++ toolchain (on PATH) And then run npm run build.

Take a look at the Workflow file, it's the most up to date way to build the project and it's pretty minimal/easy to follow locally.

If you have emsdk installed at ~/emsdk, executing npm run build:dev will use its emsdk_env.sh without messing with your global shell.

Features

Drop is currently capable of running IstanbulJS and Babel internally.

Drop's JS runtime uses QuickJS and supports both CommonJS and ES Modules with a proper global require function and a module object, just like NodeJS.

Drop's JS runtime uses SWC to transpile TypeScript(X) to JavaScript on the fly.

Drop currently offers the following features in WebAssembly:

NodeJS with emulated native modules:
- assert • buffer: • crypto: • events: • fs • os • url • util • path • process • punycode • querystring • readline • stream • stringdecoder • timers • tty • zlib • uvu (test harness)
POSIX with emulated tooling:
- awk • base64 • basename • cat • chmod • chown • cksum • clear • cp • date • diff • dirname • echo • egrep • env • expr • false • fgrep • find • grep • hd • head • hexdump • link • ln • ls • md5sum • mkdir • mkfifo • mknod • mktemp • mv • nanozip • patch • printenv • printf • pwd • readlink • realpath • rev • rm • rmdir • sed • sha256sum • sleep • sort • split • stat • tail • tar • test • touch • true • uniq • unlink • unzip • usleep • whoami • xargs

Goal is to eventually go beyond Node and web browsers and consume these features cross-platform and polyglot.

Architecture

BusyBox is pretty simple to understand. This project compiles a BusyBox subset to WebAssembly with Emscripten. This enables the code to offer usual POSIX tools such as ls, cat, grep, sed, tar, zip, etc cross-platform.

Drop is written in Rust and depends on WASI for file IO and Clock access only. Drop does NodeJS emulation and is meant to be kept so it's easy to port to any other WASI host with minimal effort.

Drop uses a dual-loop model to run asynchronous logic outside its WASM guest.
The external loop is expected to offer extended native access such as fetch.
Drop does not depend on wasm-bindgen or any other WASI proposals.

Apart from WASI for file access and Clock, Drop expects the following imports to be available for the guest WASM module:

drop.host_loop_push(buf: *const u8, len: usize);
drop.host_loop_poll() -> usize;

And exposes the following functions to the host environment:

drop_loop_push(buf: *const u8, len: usize);
drop_loop_poll(); -> usize

In the dual-loop model, Asyncify is used to mark host_loop_push as an asynchronous function, so that once it's called by the guest WASM module, guest blocks until host processes the external loop and once it returns to the WASM side, drop_loop_push is used to retrieve the result of the external loop's poll. Calling to host_loop_poll is expected to trigger drop_loop_push synchronously, if there's any data available.
drop_loop_poll is called in a loop until it exits and is done processing.
xxxx_loop_poll returns an "exit hash" which is an unsigned number, that can be decoded to get the exit status and the exit code of either loops.

Hashed exit code allows either loops to cancel each other. Format of this hashed version is: [\d+][10]. If the first bit (rightmost digit) is 1, it indicates the loop has exited and no longer should be polled. In this scenario, remaining digits left-right are the exit code. For example 261 means the loop has exited with exit code 26. If the first bit is 0, it indicates the loop is still in progress and should be polled again. In this scenario, remaining digits left to right are the number of remaining jobs in the loop. For example 120 means the loop has 12 jobs remaining in its queue. 0 means a successful exit.

Format of the exchanged buf is JSON.

The dual-loop's execution model is shown in the following Mermaid diagram:

graph LR
  S((wasi.start)) --> A
  A[drop_loop_poll] --> |sync| B[host_loop_push]
  D[host_loop_poll]  --> |sync| C[drop_loop_push]
  B --> |async| Eg
  C --> |async| Eh
  Eg --> |no| D
  Eh --> |no| A
  Eg --> |yes| E
  Eh --> |yes| E
  Eg{guest exited?}
  Eh{host exited?}
  E((process.exit))