unicode-shaper-zig v0.5.1

unicode-shaper-zig

A Zig port of a subset of the functionality of International Components for Unicode (ICU). Supports right-to-left langauges like Arabic and Hebrew.

Purpose

This library is intended to be used in the browser, and can be compiled to WebAssembly. Mapbox uses emscripten but as you can see, the bundle size is 186.21KB 44.08KB (gzip). Because of it's size, it's imported as a separate module. Due to Zig's first class WASM support the bundle size is 9.762 kB wasm + 3.257 kB JS (example case). This is a 85.7% reduction in size.

Using unicode-shaper

unicode-shaper exposes three functions for modules

shapeArabic(input: []const u16, options: u32, allocator: std.mem.Allocator) ![]const u16

Takes an input string in "logical order" (i.e. characters in the order they are typed, not the order they will be displayed) and replaces Arabic characters with the "presentation form" of the character that represents the appropriate glyph based on the character's location within a word.

processBidiText(input: []const u16) ![]u16

Takes an input string with characters in "logical order", along with a set of chosen line break points, and applies the Unicode Bidirectional Algorithm to the string. Returns a new line in "visual order" (i.e. characters in the order they are displayed, left-to-right).

processText(input: []const u16): ![]const u16

Combines arabicShaping and processBidiText to process a string with both shaping and bidirectional processing. Takes an input string with characters in "logical order", and applies the Unicode Bidirectional Algorithm to the string. Returns a new line with characters in "visual order" (i.e. characters in the order they are displayed, left-to-right) and replaces Arabic characters with the "presentation form" of the character that represents the appropriate glyph based on the character's location within a word.

isRTL(input: u16) bool

Check if the unicode character is right to left.

isCJK(input: u16) bool

Check if the unicode character is Chinese, Japanese, or Korean. Useful if you want to draw text as a vertical line for CJK characters.

Example

    const WASM = require('rtl.ts')
    const wasm = new WASM()
    const text = 'سلام۳۹'
    const output = wasm.processString(input)
    console.log(output)
    // => '۳۹ﻡﻼﺳ'

CPP TESTS: Build

1 Download and build ICU

wget https://github.com/unicode-org/icu/releases/download/release-73-rc/icu4c-73rc-src.tgz
tar xzf icu4c-73rc-src.tgz
rm icu4c-73rc-src.tgz

cd icu/source

./runConfigureICU --help
CXXFLAGS=-std=c++20 ./runConfigureICU MacOSX/GCC --disable-renaming
# OR
CXXFLAGS=-std=c++20 ./runConfigureICU Linux/gcc --disable-renaming

make clean
make -j4
sudo make install

2 Run comparison experiments via test.cpp

g++ -stdlib=libc++ -std=c++20 -w -fPIC -I/usr/local/include -L/usr/local/lib -l icuuc test.cpp -o test