4.0.0 • Published 6d agoCLI

jschardet

Licence

0BSD

Version

4.0.0

Deps

Size

7.1 MB

Vulns

Weekly

Stars

742

Summary Dependency Versions

jschardet

jschardet is a character encoding detector for JavaScript. Runs in Node.js and browsers with zero runtime dependencies.

jschardet 4 is a ground-up TypeScript port of chardet 7. It's much faster and more accurate than jschardet 3 and keeps its API; see Upgrading from jschardet 3 for what behaves differently.

The API is detect() and detectAll(), returning encoding, confidence, language, and mimeType.

Features

99.4% accuracy on 3,138 test files, up from 43.1% in jschardet 3, with ~6× the throughput and ~10× lower peak memory. Language detection for every result. MIME type detection for binary files.

	jschardet 4.0.0	jschardet 3.1.4	chardet 7.6.0 (Python)
Accuracy (3,138 files)	99.4%	43.1%	99.5%
Speed	780 files/s	126 files/s	222 files/s
Language detection	93.4%	—	93.4%
Peak memory	85.5 MiB	831.8 MiB	37.6 MiB
Bundle size (min / gzip)	1,141 / 691 KiB	334 / 119 KiB	—
Cold start (import + first detect)	109.2 ms	29.9 ms	101.7 ms
Runs in browsers	yes	yes	—
MIME type detection	yes	no	yes
License	0BSD	LGPL	0BSD

Compared to jschardet 3, v4 has a larger bundle and a ~110 ms first-call cost. Both come from shipping a larger detection model; the model decompresses once on first detect() and stays in memory afterwards, so subsequent calls run at full speed.

See docs/performance.md for the full benchmark methodology and per-encoding accuracy.

Installation

npm

npm install jschardet

Both module systems are supported:

import { detect } from 'jschardet';         // ESM
const { detect } = require('jschardet');    // CommonJS

Browser

Copy and include jschardet.min.js in your page (attaches a global jschardet). For ESM, use jschardet.esm.min.js instead. Unminified builds and source maps are in dist/.

The library is also available via jsDelivr:

Format	URL
IIFE (`<script src>`)	`https://cdn.jsdelivr.net/npm/jschardet`
ESM (`<script type="module">`)	`https://cdn.jsdelivr.net/npm/jschardet/dist/jschardet.esm.min.js`

Classic script tag (after copying jschardet.min.js next to your HTML):

<script src="jschardet.min.js"></script>
<script>
  console.log(jschardet.detect("\xc3\xa0\xc3\xad\xc3\xa0\xc3\xa7\xc3\xa3"));
</script>

ESM (after copying jschardet.esm.min.js):

<script type="module">
  import { detect } from './jschardet.esm.min.js';
  console.log(detect("\xc3\xa0\xc3\xad\xc3\xa0\xc3\xa7\xc3\xa3"));
</script>

Quick start

import { detect, detectAll } from 'jschardet';

// string — ASCII
detect("Python is a great programming language for beginners and experts alike.")
// { encoding: 'ascii', confidence: 1, language: 'en', mimeType: 'text/plain' }

// Uint8Array — "The naïve approach doesn't always work in complex systems." in UTF-8
detect(new TextEncoder().encode("The naïve approach doesn't always work in complex systems."))
// { encoding: 'utf-8', confidence: 0.84, language: 'en', mimeType: 'text/plain' }

// Uint8Array — "日本語の文字コード検出テストです。" in EUC-JP
detect(new Uint8Array([
  0xc6, 0xfc, 0xcb, 0xdc, 0xb8, 0xec, 0xa4, 0xce, 0xca, 0xb8,
  0xbb, 0xfa, 0xa5, 0xb3, 0xa1, 0xbc, 0xa5, 0xc9, 0xb8, 0xa1,
  0xbd, 0xd0, 0xa5, 0xc6, 0xa5, 0xb9, 0xa5, 0xc8, 0xa4, 0xc7,
  0xa4, 0xb9, 0xa1, 0xa3,
]))
// { encoding: 'EUC-JP', confidence: 0.56, language: 'ja', mimeType: 'text/plain' }

// Buffer (Node.js) — "Le café est une boisson très populaire en France et dans le monde entier." in windows-1252
const results = detectAll(Buffer.from([
   76, 101,  32,  99,  97, 102, 233,  32, 101, 115, 116,  32, 117, 110, 101,
   32,  98, 111, 105, 115, 115, 111, 110,  32, 116, 114, 232, 115,  32, 112,
  111, 112, 117, 108,  97, 105, 114, 101,  32, 101, 110,  32,  70, 114,  97,
  110,  99, 101,  32, 101, 116,  32, 100,  97, 110, 115,  32, 108, 101,  32,
  109, 111, 110, 100, 101,  32, 101, 110, 116, 105, 101, 114,  46,
]))
for (const r of results.slice(0, 4)) {
  console.log(r.encoding, r.confidence.toFixed(2));
}
// Windows-1252 0.32
// iso8859-15 0.32
// ISO-8859-1 0.32
// MacRoman 0.31

API

`detect(buffer, options?)`

Accepts a string, Uint8Array, ArrayBuffer, or any ArrayBufferView (Node Buffer and DataView work too). Returns the best match as an object:

Field	Type	Description
`encoding`	`string \| null`	Detected encoding name, or `null` if unknown
`confidence`	`number`	Score from 0.0 to 1.0
`language`	`string \| null`	Language hint when available
`mimeType`	`string \| null`	MIME type hint when available

`detectAll(buffer, options?)`

Same input types as detect. Returns all candidates above the confidence threshold (default 0.20), sorted by confidence. At least one result is always returned when the built-in threshold applies.

Options

interface IOptionsMap {
  minimumThreshold?: number;      // override default 0.20 for detectAll filtering
  detectEncodings?: string[];     // allowlist of encoding names to consider
  excludeEncodings?: string[];    // blocklist of encoding names
}

`enableDebug()`

Logs full candidate lists to the console from detect / detectAll (useful when tuning thresholds or allowlists).

CLI

jschardet somefile.txt
# somefile.txt: utf-8 with confidence 1

jschardet --minimal somefile.txt
# utf-8

# Include detected language
jschardet -l somefile.txt
# somefile.txt: utf-8 en (English) with confidence 1

# Only consider specific encodings
jschardet -i utf-8,windows-1252 somefile.txt
# somefile.txt: utf-8 with confidence 1

# Pipe from stdin
cat somefile.txt | jschardet
# stdin: utf-8 with confidence 1

Supported encodings

Same encodings as chardet (aliases and encoding-era filters are documented there).

Modern Web

ascii, big5hkscs, cp874, cp932, cp949, euc-jis-2004, euc-kr, gb18030, hz-gb-2312, iso-2022-kr, iso2022-jp-2, iso2022-jp-2004, iso2022-jp-ext, koi8-r, koi8-u, shift_jis_2004, tis-620, utf-16, utf-16-be, utf-16-le, utf-32, utf-32-be, utf-32-le, utf-7, utf-8, utf-8-sig, windows-1250, windows-1251, windows-1252, windows-1253, windows-1254, windows-1255, windows-1256, windows-1257, windows-1258

import { chardet } from 'jschardet';
import { createReadStream } from 'node:fs';

const detector = new chardet.UniversalDetector();
for await (const chunk of createReadStream('unknown.txt')) {
  detector.feed(chunk);
  if (detector.done) break;
}
console.log(detector.close());
// { encoding: 'utf-8', confidence: 1, language: 'en', mimeType: 'text/plain' }

Upgrading from jschardet 3

detect(), detectAll(), enableDebug(), and the minimumThreshold and detectEncodings options keep their v3 signatures, and a string is still read as one byte per character. What changes:

Results. Many inputs detect differently; that is where the accuracy gain comes from. Encoding names follow chardet's spelling, e.g. utf-8 where v3 returned UTF-8, so compare names case-insensitively.
detectEncodings accepts v3 names except four encodings v4 does not detect, which throw: EUC-TW, ISO-2022-CN, X-ISO-10646-UCS-4-3412, and X-ISO-10646-UCS-4-2143. When the list leaves out windows-1252 and no listed encoding matches, the result is null and a warning is logged with console.warn.
Confidence scale. Correct answers often score far lower than in v3: the Big5 sample from v3's README scored 0.99 in v3 and scores 0.08 in v4. Re-tune any threshold chosen against v3 numbers.
detect() always returns its best guess. v3 returned { encoding: null, confidence: 0 } when the best guess scored below minimumThreshold (default 0.20); v4's detect() ignores minimumThreshold, so check confidence yourself. minimumThreshold filters detectAll() instead, which v3 did not filter.
Empty input returns utf-8 with confidence 0.1, where v3 returned { encoding: null, confidence: 0 }.
Result type. Results gain language and mimeType, and encoding is typed string | null (v3: string), so TypeScript code under strictNullChecks needs a null check.
UniversalDetector, exported but undocumented in v3, is now chardet.UniversalDetector (see chardet module). It has chardet's API: feed() takes a Uint8Array rather than a string, and the constructor takes chardet's options (includeEncodings rather than detectEncodings, no minimumThreshold).