# @datagica/fast-index

> Fast Index

Latest version **0.1.0** (published 2018-09-27) · GPL-3.0 license · 0 weekly downloads

## Install

```sh
npm install @datagica/fast-index
pnpm add @datagica/fast-index
yarn add @datagica/fast-index
bun add @datagica/fast-index
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.1.0 |
| Published | 2018-09-27 |
| First published | 2016-05-02 |
| Weekly downloads | 0 |
| License | GPL-3.0 |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 0 |
| Unpacked size | 19.9 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 0 |
| Author | Julian Bilcke |
| Maintainers | datagica |

## Links

- npm: https://www.npmjs.com/package/@datagica/fast-index
- Repository: https://github.com/datagica/fast-index
- Homepage: https://github.com/datagica/fast-index#readme
- Issues: https://github.com/datagica/fast-index/issues
- npm.io page: https://npm.io/package/@datagica/fast-index

## Recent versions

- 0.1.0 (latest) — 2018-09-27
- 0.0.7 — 2018-09-27
- 0.0.6 — 2018-09-26
- 0.0.5 — 2016-05-13
- 0.0.4 — 2016-05-03
- 0.0.3 — 2016-05-02
- 0.0.2 — 2016-05-02
- 0.0.1 — 2016-05-02
- 0.0.0 — 2016-05-02

## README

# Datagica Fast-Index

A library to lookup if a word is inside an index, even if the spelling is a bit
different.

## Usage

### Installation

   $ npm install @datagica/fast-index --save

### Building the index

```javascript
import "FastIndex" from "@datagica/fast-index";

const index = new FastIndex({

  // fields to be indexed
  fields: [
    'label',
    'aliases'
  ],

  // a custom spelling generation function
  spellings: (map, word) => {
    // replace "le " or "el " by "the " with an arbitrary similarity score
   // of 0.5 (you can choose any value between 0 and 1)
    map.set(word.replace(/(?:le|el) /gi, 'the '), 0.5)
  }
})

// now we load some dataset
index.loadSync([
  { label: 'the chef', type: 'movie' },

  // duplicate entries are supported and will be returned in the results
  { label: 'the chef', type: 'book' },

  // duplicates inside an entry are simply skipped
  { label: 'el chef', aliases: [ 'el chef' ] }
]);


// side-note: here is the internal representation of the data inside the index:
[ [ 'the chef',
    [ { value: { label: 'the chef', type: 'movie' }, score: 1 },
      { value: { label: 'the chef', type: 'book' }, score: 1 },
      { value: { label: 'el chef', type: 'unknow', aliases: [ 'el chef' ] },
        score: 0.5 } ] ],
  [ 'el chef',
    [ { value: { label: 'el chef', type: 'unknow', aliases: [ 'el chef' ] },
        score: 1 } ] ] ]
```

### Querying the index

```javascript
const matches = index.get("le chef");

// this will output
[
  { value: { label: 'the chef', type: 'movie' },
    score: 0.5 },

  { value: { label: 'the chef', type: 'book' },
    score: 0.5 },

  // note how "el chef" has a lower score, although it would be closer using
  // a distance function. That's because we choose a naive spelling function
  // that converts everything into a single locale (english).
  // a better function would be more fine-tuned and store each locale
  // individually
  { value: { label: 'el chef',  type: 'unknow', aliases: [ 'el chef' ] },
    score: 0.25 }
]
```

## History

### Problem

The original algorithm used for fuzzy matching entities in all Datagica projects
(`@datagica/fuzzy-index`) was based on a lookup inside a tree of possible
spellings, using a [Finite State Levenshtein Transducer](http://www.aclweb.org/anthology/I08-2131).
This was nice because it allowed us to match a university name (for instance)
even if it was spelled a bit differently eg (*universidad* instead of *university*).

However for huge datasets this proved quite slow, incompatible with real-time
lookup, as it searched more alternative spellings than necessary.

### Solution

The new Fast Index simply converts its inputs into a simple representation of the
word, where punctuation, accents, useless spaces etc.. have been removed.

These transformations are quite opinionated, but some can be opted-out.

In addition to these default transforms, Fast-Index also gives you a way to
define custom alternative spellings.

For instance, you can tell Fast-Index to automatically convert:
 - ` de ` -> `of `
 -  `ité ` -> `ity `

Now, if the input word is `université de Poudlard` it will match `university of Poudlard`!

When using the spelling generator, you decide yourself which distance score
should be allocated. This helps the Fast-Index picks the best match it find
later.

---
_Source: https://npm.io/package/@datagica/fast-index · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._