@znetstar/encode-tools v1.6.0

Encode Tools

This package aggregates different libraries for encoding, serializing, compressing, generating ids and hashing things, exposing a common interface.

Many other packages serve the same purpose, but our objective is to ensure a consistent experience in both node.js and the browser and standardize the api so functions work the same way across different underlying libraries.

Encode Tools also has a command line wrapper encode-cli.

Examples

Encoding a Buffer as base64url

  let enc = new EncodeTools();
  let buf = Buffer.from('hello world', 'utf8');
  let newBuf = enc.encodeBuffer(buf, BinaryEncoding.base64url);
  console.log(newBuf.toString('utf8'));

Hashing an object wth xxhash

let enc = new EncodeTools();
let obj = { foo: 'bar' };
let newBuf = await enc.hashObject(obj, HashAlgorithm.xxhash64);
console.log(newBuf.toString('utf8'));

Serializing an object wth msgpack

let enc = new EncodeTools();
let obj = { foo: 'bar' };
let newBuf = await enc.serializeObject(obj, SerializationFormat.msgpack);
console.log(newBuf.toString('base64'));

Generating a base64-encoded UUID v4

let enc = new EncodeTools();
let newBuf = await enc.uniqueId(IDFormat.uuidv4);
console.log(newBuf.toString('base64'));

Compressing a buffer with lzma

let enc = new EncodeTools();
let newBuf = await enc.compress(Buffer.from('hi', 'utf8'), CompressionFormat.lzma);
console.log(newBuf.toString('base64'));

Resizing a png image

let enc = new EncodeTools();
let imageBuf = await (await new Promise((resolve, reject) => {
  new (Jimp)(500, 500, '#FFFFFF', (err: unknown, image: any) => {
    if (err) reject(err);
    else resolve(image);
  });
})).getBufferAsync('image/png');

let myResizedPng = await enc.resizeImage(imageBuf, { width: 250 }, ImageFormat.png);

Structure

This project is divided into two packages, a core package with no native dependencies (@znetstar/encode-tools), and a full version with native modules as optional dependencies (@znetstar/encode-tools-native). The core package should work well in the browser via Webpack or Browserify, whereas the full version includes more formats and should have better performance.

Algorithms

Below are a list of supported algorithms, their backing library, and their support in the browser.

Binary Encoding

Hashing

ID Generation

Serialization

Compression

Image Manipulation

Requirements

Encode Tools runs in the browser and in node.js, with a few exceptions. The bson-ext, lzma-native, xxhash-addon and cbor-extract packages have native bindings, and so cannot run in the browser. For browser compatibility, the EncodeTools class uses the pure javascript bson, lzma, hash-wsam, and cbor-x packages, respectively, to provide equivalent support albeit at the cost of performance. Additionally, hash-wsam lacks support for xxhash3.

The EncodeToolsAuto class will use the native packages bson-ext, lzma-native and xxhash-addon (and any future native packages). bson-ext, lzma-native and xxhash-addon are listed as optional dependencies, and NPM will attempt to install them automatically.

The constructor of EncodeToolsAuto takes a second set of default EncodingOptions to use as a fallback if it cannot find the needed module.

const enc = new EncodeToolsAuto({ hashAlgorithm: HashAlgorithm.xxhash3 }, { hashAlgorithm: HashAlgorithm.xxhash64 });
if (enc.availableNativeModules.xxhashAddon)
    console.log('should be xxhash3', await enc.hashString('Test'));
else
    console.log('should be xxhash64', await enc.hashString('Test'));

The gif image format in EncodeToolsAuto requires libvips compiled with ImageMagick support (as described here). I haven't had time to re-build libvips on my machine, so there are no mocha tests for the gif format.

Usage

Please see the documentation located at https://znetstar.github.io/encode-tools/ and https://znetstar.github.io/encode-tools-native/

Webpack

For issues with Webpack, try adding all the native dependencies to the externals section.

{
  externals: {
      'xxhash-addon': 'commonjs xxhash-addon',
      'bson-ext': 'commonjs bson-ext',
      'shelljs': 'commonjs shelljs',
      'lzma-native': 'commonjs lzma-native',
      'sharp': 'commonjs sharp'
  },
  // For Webpack 5+ only, add `node-polyfill-webpack-plugin`
  plugins: [
    new (require('node-polyfill-webpack-plugin'))()
  ]
}

Next.js

For Next.js, you can insert into next.config.js

{
  webpack: (config, { isServer }) => {
    if (!isServer) {
      config.resolve.fallback = {
        fs: false
      }
      config.externals = {
        ...config.externals,
        'xxhash-addon': 'commonjs xxhash-addon',
        'bson-ext': 'commonjs bson-ext',
        'shelljs': 'commonjs shelljs',
        'lzma-native': 'commonjs lzma-native',
        'sharp': 'commonjs sharp'
      }
      config.plugins = [
        ...config.plugins,
        new (require('node-polyfill-webpack-plugin'))()
      ]
    }
    return config;
  }
}

Tests

Tests are written in Mocha, to run use npm test.

Attribution

This project is a fork of @etomon/encode-tools.

License

znetstar Encode Tools is licensed under the GNU LGPL-3.0, a copy of which can be found at https://www.gnu.org/licenses/.