@thi.ng/pixel v6.1.28

This project is part of the @thi.ng/umbrella monorepo.

• About
  • Integer pixel formats
  • Floating point pixel formats
  • Filtered image sampling and resizing
    • Filters
    • Wrap mode
  • Strided convolution & pooling
  • Normal map generation
  • Dominant color extraction
  • Status
  • Support packages
  • Related packages
• Installation
• Dependencies
• Usage examples
• API
• Authors
• License

About

Typedarray integer & float pixel buffers w/ customizable formats, blitting, drawing, convolution.

• Buffer creation from HTML image elements w/ opt resize & format conversion (browser only)
• Buffer-to-buffer blitting w/ automatic format conversion
• Buffer-to-canvas blitting
• Buffer-to-buffer blending w/ Porter-Duff operators
• Pre/post-multiply alpha
• Region / sub-image extraction
• Single-channel manipulation / extraction / replacement / conversion
• k-means based dominant color extraction (float buffers only)
• Accessors for normalized channel value
• Image sampling, resizing, pooling
  • Filters: nearest neighbor, bilinear, bicubic
  • Wrap behaviors: clamp, wrap, repeat
  • Pooling: mean/min/max
• Invert image
• Convolution w/ arbitrary shaped/sized kernels, pooling, striding
• Convolution kernel & pooling kernels presets
  • Higher order kernel generators (Gaussian, Lanczos)
• Image pyramid generation (w/ customizable kernels)
• Customizable normal map generation (i.e. X/Y gradients plus static Z component)
• XY full pixel & channel-only accessors
• 12 packed integer and 6 floating point preset formats (see table below)
• Declarative custom format & optimized code generation
• HTML canvas creation & ImageData utilities

Integer pixel formats

All integer formats use the canvas native ABGR 32bit format as common intermediate for conversions. During conversion to ABGR, channels with sizes smaller than 8 bits will be scaled appropriately to ensure an as full-range and as linear as possible mapping. E.g. a 4 bit channel will be scaled by 255 / 15 = 17.

Format specs can freely control channel layout within current limits:

• Channel sizes: 1 - 32 bits.
• Storage: 8, 16 or 32 bits per pixel

New formats can be defined via defIntFormat().

• ALPHA8 is mapped from/to ABGR alpha channel
• GRAY8/16, GRAY_ALPHA8/16 compute grayscale/luminance when converting from ABGR and in return produce grayscale ABGR
• In all built-in formats supporting it, the alpha channel always occupies the most-significant bits (up to format size)

Floating point pixel formats

Strided floating point format presets for use with floatBuffer(). New formats can be defined via defFloatFormat().

• All color channels are unclamped (but can be clamped via buf.clamp()). For conversion to packed int formats assumed to contain normalized data (i.e. 0..1 interval, with exception of FLOAT_NORMAL which uses -1..1 range)
• Conversion between float formats is currently unsupported

Filtered image sampling and resizing

Available (and optimized) for both integer & floating point formats, image samplers can be created with the following filters & wrap modes:

Filters

• "nearest" - nearest neighbor
• "linear" - bilinear interpolation
• "cubic" - bicubic interpolation

Wrap mode

• "clamp" - outside values return 0
• "wrap" - infinite tiling
• "repeat" - edge pixels are repeated

const src = intBuffer(4, 4, ABGR8888);

// fill w/ random colors
src.forEach((_,i) => 0xff000000 | Math.random() * 0xffffff);

// create bilinear sampler w/ repeated edge pixels
const sampler = defSampler(src, "linear", "repeat");

// sample at fractional positions (even outside image)
sampler(-1.1, 0.5).toString(16)
// 'ff79643a'

// resize image to 1024x256 using bicubic sampling
const img = src.resize(1024, 256, "cubic");

Filter
"nearest"
"linear"
"cubic"

Strided convolution & pooling

Floating point buffers can be processed using arbitrary convolution kernels. The following convolution kernel presets are provided for convenience:

Custom kernels can be defined (and code generated) using an array of coefficients and a given kernel size. See above presets and defKernel() for reference.

Furthermore, convolution supports striding (i.e. only processing & keeping every nth pixel column/row, aka downscaling) and pixel pooling (e.g. for ML applications). Available pooling kernel presets (kernel sizes must be configured independently):

Convolution can be applied to single, multiple or all channels of a FloatBuffer. See convolveChannel() and convolveImage().

See ConvolveOpts for config options.

// convolutions are only available for float buffers (for now)
src = floatBuffer(read("test.ppm"), FLOAT_RGB);

// apply horizontal Sobel kernel preset to all channels
// downscale image by factor 2 (must be integer)
// scale kernel result values by factor 4
convolveImage(src, { kernel: SOBEL_X, stride: 2, scale: 4 });

Normal map generation

Normal maps can be created via normalMap(). This function uses an adjustable convolution kernel size to control gradient smoothness & details. Result X/Y gradients can also be scaled (uniform or anisotropic) and the Z component can be customized to (default: 1.0). The resulting image is in FLOAT_NORMAL format, using signed channel values. This channel format is auto-translating these into unsigned values when the image is converted into an integer format.

Step	Scale = 1	Scale = 2	Scale = 4	Scale = 8
0
1
2
3

import { floatBuffer, normalMap, FLOAT_GRAY, RGB888 } from "@thi.ng/pixel";
import { asPPM, read } from "@thi.ng/pixel-io-netpbm";

// read source image into a single channel floating point buffer
const src = floatBuffer(read(readFileSync("noise.pgm")), FLOAT_GRAY);

// create normal map (w/ default options)
const nmap = normalMap(src, { step: 0, scale: 1 });

// pixel lookup (vectors are stored _un_normalized)
nmap.getAt(10, 10);
// Float32Array(3) [ -0.019607841968536377, -0.04313725233078003, 1 ]

// save as 24bit PBM, conversion to RGB int format first
writeFileSync("noise-normal.ppm", asPPM(nmap.as(RGB888)));

Dominant color extraction

The dominantColors() function applies k-means clustering to extract the dominant colors from the given image. The clustering can be configured. The function returns an array of { color, area } objects (sorted by descending area), where color is a cluster's dominant color (in the format of the source image) and area the normalized cluster size (number of selected pixels over total number of pixels in the image).

Picture credit: /u/kristophershinn

// read test PPM image and convert into float RGB format
const img = floatBuffer(read(readFileSync(`test.ppm`)), FLOAT_RGB);

// extract 5 dominant color clusters
const clusters = dominantColors(img, 5);

console.log(clusters);
// [
//   {
//     color: [ 0.4000000059604645, 0.30980393290519714, 0.21176470816135406 ],
//     area: 0.3141084558823529
//   },
//   {
//     color: [ 0.21960784494876862, 0.19607843458652496, 0.1411764770746231 ],
//     area: 0.2780330882352941
//   },
//   {
//     color: [ 0.4156862795352936, 0.4745098054409027, 0.5647059082984924 ],
//     area: 0.16620710784313725
//   },
//   {
//     color: [ 0.6666666865348816, 0.7568627595901489, 0.9254902005195618 ],
//     area: 0.12385110294117647
//   },
//   {
//     color: [ 0.7176470756530762, 0.4745098054409027, 0.12941177189350128 ],
//     area: 0.11780024509803921
//   }
// ]

Status

STABLE - used in production

Search or submit any issues for this package

Support packages

• @thi.ng/pixel-dither - Extensible image dithering w/ various algorithm presets
• @thi.ng/pixel-io-netpbm - Multi-format NetPBM reader & writer support for @thi.ng/pixel

Related packages

• @thi.ng/color - Array-based color types, CSS parsing, conversions, transformations, declarative theme generation, gradients, presets
• @thi.ng/porter-duff - Porter-Duff operators for packed ints & float-array alpha compositing
• @thi.ng/rasterize - 2D shape drawing & rasterization
• @thi.ng/shader-ast - DSL to define shader code in TypeScript and cross-compile to GLSL, JS and other targets
• @thi.ng/webgl - WebGL & GLSL abstraction layer

Installation

yarn add @thi.ng/pixel

ES module import:

<script type="module" src="https://cdn.skypack.dev/@thi.ng/pixel"></script>

Skypack documentation

For Node.js REPL:

# with flag only for < v16
node --experimental-repl-await

> const pixel = await import("@thi.ng/pixel");

Package sizes (gzipped, pre-treeshake): ESM: 8.91 KB

Dependencies

• @thi.ng/api
• @thi.ng/binary
• @thi.ng/checks
• @thi.ng/distance
• @thi.ng/errors
• @thi.ng/k-means
• @thi.ng/math
• @thi.ng/porter-duff

Usage examples

Several demos in this repo's /examples directory are using this package.

A selection:

API

Generated API docs

import * as pix from "@thi.ng/pixel";
import { SRC_OVER_I } from "@thi.ng/porter-duff";

import IMG from "../assets/haystack.jpg";
import LOGO from "../assets/logo-64.png";

Promise
    .all([IMG, LOGO].map(pix.imagePromise))
    .then(([img, logo]) => {

        // init 16 bit packed RGB pixel buffer from image (resized to 256x256)
        const buf = intBufferFromImage(img, pix.RGB565, 256, 256);

        // create grayscale buffer for logo and use Porter-Duff operator to
        // composite with main image. Since the logo has transparency, we need
        // to premultiply alpha first...
        pix.intBufferFromImage(logo, pix.GRAY_ALPHA88)
            .premultiply()
            .blend(SRC_OVER_I, buf, {
                dx: 10,
                dy: 10
            });

        // extract sub-image
        const region = buf.getRegion(32, 96, 128, 64);
        // copy region back at new position
        region.blit(buf, { dx: 96, dy: 32 });

        // or alternatively blit buf into itself
        // buf.blit(buf, { dx: 96, dy: 32, sx: 32, sy: 96, w: 128, h: 64 });

        // create html canvas
        // (returns obj of canvas & 2d context)
        const ctx = pix.canvas2d(buf.width, buf.height * 3, document.body);

        // write pixel buffer to canvas
        buf.blitCanvas(ctx.canvas);

        // manipulate single color channel (here red)
        const id = 0;
        // obtain channel & invert
        const ch = buf.getChannel(id).invert();
        // create dot pattern
        for (let y = 0; y < ch.height; y += 2) {
            for (let x = (y >> 1) & 1; x < ch.width; x += 2) {
                ch.setAt(x, y, 0xff);
            }
        }
        // replace original channel
        buf.setChannel(id, ch);

        // write pixel buffer to new position
        buf.blitCanvas(ctx.canvas, 0, buf.height);

        // create & write grayscale version
        buf.as(GRAY8).blitCanvas(ctx.canvas, 0, buf.height * 2);
});

TODO see examples & source comments for now

Authors

Karsten Schmidt

If this project contributes to an academic publication, please cite it as:

@misc{thing-pixel,
  title = "@thi.ng/pixel",
  author = "Karsten Schmidt",
  note = "https://thi.ng/pixel",
  year = 2019
}

License

© 2019 - 2021 Karsten Schmidt // Apache Software License 2.0