# @clipperhouse/jargon

> A tokenizer and lemmatizer for canonical terms in text

Latest version **0.3.0** (published 2024-05-07) · MIT license · 20 weekly downloads

## Install

```sh
npm install @clipperhouse/jargon
pnpm add @clipperhouse/jargon
yarn add @clipperhouse/jargon
bun add @clipperhouse/jargon
```

## Health

**Score 25/100 (F)** — status: abandoned.

Positive: has types; no vulnerabilities; high quality score.

Warnings: low downloads; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.3.0 |
| Published | 2024-05-07 |
| First published | 2020-02-09 |
| Weekly downloads | 20 |
| License | MIT |
| TypeScript types | bundled |
| Module format | CommonJS |
| Dependencies | 0 |
| Unpacked size | 382.6 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 2 |
| Author | Matt Sherman |
| Maintainers | clipperhouse |
| Keywords | lemmatization, named entity recognition, tokenization, nlp |

## Links

- npm: https://www.npmjs.com/package/@clipperhouse/jargon
- Repository: https://github.com/clipperhouse/jargon-typescript
- Homepage: https://github.com/clipperhouse/jargon-typescript#readme
- Issues: https://github.com/clipperhouse/jargon-typescript/issues
- npm.io page: https://npm.io/package/@clipperhouse/jargon

## Recent versions

- 0.3.0 (latest) — 2024-05-07
- 0.2.0 — 2020-02-12
- 0.1.0 — 2020-02-10
- 0.0.10 — 2020-02-10
- 0.0.9 — 2020-02-10
- 0.0.8 — 2020-02-10
- 0.0.7 — 2020-02-10
- 0.0.6 — 2020-02-09
- 0.0.5 — 2020-02-09
- 0.0.4 — 2020-02-09
- 0.0.3 — 2020-02-09
- 0.0.2 — 2020-02-09
- 0.0.1 — 2020-02-09

## README

Jargon is a TypeScript/JavaScript library for tokenization and lemmatization. It finds variations on canonical terms and converts them to a single form.

For example, in tech, you might see 'node js' or 'NodeJS' or 'node.js' and want them understood as the same term. That’s lemmatization.

## Quick start

```
npm install "@clipperhouse/jargon@latest"
```

Then create a file, preferably TypeScript.

```ts
// demo.ts

import jargon from '@clipperhouse/jargon';	
import stackexchange from '@clipperhouse/jargon/stackexchange';	// a dictionary

const text = 'I ❤️ Ruby on Rails and vue';

const lemmatized = jargon.Lemmatize(text, stackexchange);

console.log(lemmatized.toString());

// I ❤️ ruby-on-rails and vue.js

```

```js
// demo.js

const jargon = require('@clipperhouse/jargon');
const stackexchange = require('@clipperhouse/jargon/stackexchange');

const text = 'I ❤️ Ruby on Rails and vue';

const lemmatized = jargon.Lemmatize(text, stackexchange);
console.log(lemmatized.toString());

// I ❤️ ruby-on-rails and vue.js
```

## What’s it doing?

`jargon` tokenizes the incoming text — it’s not search-and-replace or regex. It understands tech-ish terms as single words, such as Node.js and TCP/IP, and #hashtags and @handles, where other tokenizers might split those words. It does pretty well with email addresses and URLs.

Tokens go to the lemmatizer, with a `dictionary`. The lemmatizer passes over tokens, and asks the dictionary if it recognizes them. It greedily looks for multi-token phrases like 'Ruby on Rails', converting them a single `ruby-on-rails` token.

The lemmatizer returns a lazy iterable. You should consume tokens with `for..of`.

## Dictionaries

Two dictionaries are included.

The `stackexchange` dictionary looks for technology terms, using tags and synonyms from Stack Overflow. It is insensitive to spaces, hyphens, dots, slashes and case, so variations like ASPNET and asp.net are recognized as the same thing. It also understands synonyms such as ecmascript ↔ javascript.

Another example is the `contractions` dictionary. It splits tokens like `it'll` into two tokens `it` and `will`.

You can pass multiple dictionaries to the lemmatizer.

```ts
// demo.ts

import jargon from '@clipperhouse/jargon';
import stackexchange from '@clipperhouse/jargon/stackexchange';
import contractions from '@clipperhouse/jargon/contractions';

const text = 'She’ll use react js and type script';

const lemmatized = jargon.Lemmatize(text, stackexchange, contractions);

console.log(lemmatized.toString());

// She will use react.js and typescript

```

---
_Source: https://npm.io/package/@clipperhouse/jargon · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._