# @tokey/core

> simple code like tokenizer

Latest version **2.0.0** (published 2025-11-04) · MIT license · 0 weekly downloads

## Install

```sh
npm install @tokey/core
pnpm add @tokey/core
yarn add @tokey/core
bun add @tokey/core
```

## Health

**Score 55/100 (C)** — status: stable.

Positive: esm support; no vulnerabilities; high maintenance score.

Warnings: low downloads; no types.

## Facts

| | |
|---|---|
| Version | 2.0.0 |
| Published | 2025-11-04 |
| First published | 2021-06-16 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | ESM |
| Dependencies | 0 |
| Unpacked size | 38.1 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 6 |
| Author | Dazl |
| Maintainers | tomrav, idoros, cijoe, avi.vahl |
| Keywords | parser, tokenizer, tokens, code, javascript |

## Links

- npm: https://www.npmjs.com/package/@tokey/core
- Repository: https://github.com/dazl-dev/tokey
- Homepage: https://github.com/dazl-dev/tokey#readme
- Issues: https://github.com/dazl-dev/tokey/issues
- npm.io page: https://npm.io/package/@tokey/core

## Alternatives

- [babylon](https://npm.io/package/babylon.md) — 5.1M weekly downloads
- [csscolorparser](https://npm.io/package/csscolorparser.md) — 3.7M weekly downloads
- [expr-eval-fork](https://npm.io/package/expr-eval-fork.md) — 1.5M weekly downloads
- [@leeoniya/ufuzzy](https://npm.io/package/@leeoniya/ufuzzy.md) — 247.7K weekly downloads
- [xml-parser](https://npm.io/package/xml-parser.md) — 78.4K weekly downloads

## Recent versions

- 2.0.0 (latest) — 2025-11-04
- 1.4.2 — 2025-11-04
- 1.4.1 — 2025-10-30
- 1.4.0 — 2023-06-14
- 1.3.0 — 2022-03-08
- 1.2.1 — 2021-08-16
- 1.2.0 — 2021-07-19
- 1.1.0 — 2021-06-28
- 1.0.0 — 2021-06-16

## README

# `@tokey/core`

## API

The core API is just one function.

```ts
type Descriptors =
  | "string"
  | "text"
  | "line-comment"
  | "multi-comment"
  | "unclosed-string"
  | "unclosed-comment"
  | "space";

interface Token<Types = Descriptors> {
  type: Types;
  start: number;
  end: number;
  value: string;
}

interface Options<T extends Token<unknown>> {
  shouldAddToken(type: T["type"], value: string): boolean;
  isStringDelimiter(char: string): boolean;
  isDelimiter(char: string): boolean;
  isWhitespace(char: string): boolean;
  createToken(value: string, type: T["type"], start: number, end: number): T;
  offset?: number;
}

function tokenize<T extends Token<unknown>>(
  source: string,
  {
    isDelimiter,
    isStringDelimiter,
    isWhitespace,
    shouldAddToken,
    createToken,
  }: Options<T>
): T[];
```

You can extend the tokenizer by providing options that match your use-case and extending the Token type.

```ts

type Delimiters = "(" | ")" | "," | ";" | ":";
type CSSCodeToken = Token<Descriptors | Delimiters>;
tokenize<CSSCodeToken>(source, {...})

```

## How it works

The main idea is looping over all the characters and splitting tokens via `isDelimiter`, `isWhitespace`, and `isStringDelimiter`.
After that, you can decide about the shape of the token with `createToken` and if it should be included with `shouldAddToken`

## What to do with the tokens

TBD Seeker

## Helpers

TBD helpers

## TODO

- better unclosed string ending detection

---
_Source: https://npm.io/package/@tokey/core · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._