persian-to-pinglish

Persian to Pinglish/Finglish Converter

Converts persian text to its pinglish/finglish form, and to ascii url slugs.

It does it in its own way: it splits every word into all the hejas (syllables) it could be made of, scores each split against a list of word templates (وزن‌ها), and reads the word through the template that fits best.

Persian does not write its short vowels, so کتاب could be read kataab or ketaab and nothing in the spelling says which. Consonants and long vowels come out right; short vowels are a guess. On words the rules have never been aimed at they land exactly right about half the time, which is why a lexicon of common words ships with the library and why you can pass your own.

You can see it in action here.

Install

npm add persian-to-pinglish
# or
deno add jsr:@jd1378/persian-to-pinglish

Usage

import toPinglish, { toSlug } from 'persian-to-pinglish';

toPinglish('ضعیف'); // "zaeef"
toPinglish('علی به خانه رفت'); // "alee be khaane raft"

toSlug('نسخه ۲ از React'); // "naskhe-2-az-react"

Latin words and numbers are passed through untouched, and persian or arabic-indic digits are normalized to ascii.

toPinglish(str, options?)

option	default	what it does
join	' '	what to put between the converted words
splitBudget	4096	how many heja splits of one word may be generated

toSlug(str, options?)

Takes everything toPinglish takes, plus:

option	default	what it does
separator	'-'	what to put between the words
lowercase	true	lowercase the result
maxLength	0	cut at the last whole word that fits, 0 means no limit

The result only ever contains a-z, 0-9 and the separator, and never starts or ends with the separator.

Lexicon

Words the rules cannot guess are simply written down in src/lexicon.ts, and that is where to fix a word that comes out wrong. To override or extend it without touching the library:

toSlug('کتاب من', { lexicon: { کتاب: 'ketab' } }); // "ketab-man"
toPinglish('کتاب', { useDefaultLexicon: false }); // "kataab", straight from the rules

Performance

Splitting a word into hejas is exponential in its length, so a single word is capped at splitBudget splits and conversions are cached per word. That keeps long or hostile input from burning the cpu, but conversion is still real work: sanitize and bound your input like you would with any other user input.

Development

Deno is the only tool needed. The one dependency is jsr:@std/assert, used by the tests.

deno task test      # run the tests
deno task verify    # fmt, lint, type check, version check and test
deno task accuracy  # report how much of test/corpus.json is exactly right
deno task build     # write dist/ and the demo site/

Three sets of words guard conversion quality, and all three measure the rules with the lexicon turned off:

• test/words.json must convert exactly.
• test/corpus.json is the corpus the rules were tuned against, currently 75/110. deno task accuracy lists what is still wrong.
• test/holdout.json is words in neither the lexicon nor the corpus, currently 29/57. deno task accuracy -- --holdout reports it.

The holdout number is the honest one: the corpus flatters the rules because they were tuned against it. deno task accuracy -- --with-lexicon reports the end to end number, which mostly measures how much of the input the lexicon covers.

Releasing

The version lives in both deno.json (jsr) and package.json (npm), and deno task verify fails if they disagree. Bump both, then push a matching tag:

git tag v0.6.0 && git push origin v0.6.0

That runs the release workflow, which checks the tag against the version, verifies, builds, and publishes to both registries. Neither publish uses a token: jsr and npm both authenticate the workflow itself over OIDC, so there is no secret in the repository to leak or rotate, and npm attaches a provenance attestation on its own.

Both registries have to be told to trust this workflow once:

• jsr: create the @jd1378 scope and the package at jsr.io, and link it to the github repository.
• npm: on the package's settings page add a trusted publisher for this repository with the workflow filename release.yml. The repository.url in package.json has to match the github repository exactly, which is why it is written as git+https://github.com/....

Important changes

0.6.0

• Added a built in lexicon of common words, with lexicon and useDefaultLexicon options to extend, override or turn it off. Short vowels are not written in persian and cannot always be guessed, so the words the rules get wrong are written down instead.
• Migrated to Deno and TypeScript. The published npm package is unchanged in shape: dist/ still holds a cjs, an esm and a umd bundle, now with type declarations, and the browser global is still PersianToPinglish.
• Added toSlug.
• Conversion never throws anymore. A word that fits no template is read from its heja split, and a word that has no valid split at all is read letter by letter. Before, such words threw and took the whole sentence with them.
• Numbers and latin words are kept instead of being dropped.
• A word final ه after a consonant now reads as e, so خانه is khaane and not khaanh. ماه still keeps its h.
• Words glued together with a zwnj, and words built out of a known prefix or suffix, are now read part by part instead of failing.
• Splitting a word is bounded, and converted words are cached.
• getBestWordMatch now returns a tagged result (kind of template, word or heuristic) instead of an object with an optional heuristic flag.

0.5.0

Updated rollup config to produce a umd version that can be used in browser environment. main function renamed to toPinglish. this can be accessed from PersianToPinglish.toPinglish('yourstr'). this is a breaking change from last version.

0.4.0

Since this version the method used to generate the output is drastically changed. it generates all possible hejas then tries to find the best matching template from defined templates (available in source). then applies the template to get the output.

Want to support this project?

or want me full time on this project? Send me an email on hotmail: javad.mnjd