# text-miner

> text mining utilities

Latest version **2.0.0** (published 2023-01-21) · MIT license · 0 weekly downloads

## Install

```sh
npm install text-miner
pnpm add text-miner
yarn add text-miner
bun add text-miner
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 2.0.0 |
| Published | 2023-01-21 |
| First published | 2014-08-15 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 11 |
| Unpacked size | 68 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 142 |
| Author | Philipp Burckhardt |
| Maintainers | planeshifter |
| Keywords | text-mining, text-analytics, document-term-matrix, dtm, text-analysis |

## Links

- npm: https://www.npmjs.com/package/text-miner
- Repository: https://github.com/Planeshifter/text-miner
- Homepage: https://github.com/Planeshifter/text-miner#readme
- Issues: https://github.com/Planeshifter/text-miner/issues
- npm.io page: https://npm.io/package/text-miner

## Dependencies (11)

- [stemmer](https://npm.io/package/stemmer.md) ^1.0.5
- [underscore](https://npm.io/package/underscore.md) ~1.13.6
- [lancaster-stemmer](https://npm.io/package/lancaster-stemmer.md) ^1.0.5
- [underscore.string](https://npm.io/package/underscore.string.md) ^3.3.6
- [validate.io-object](https://npm.io/package/validate.io-object.md) ^1.0.4
- [validate.io-array-like](https://npm.io/package/validate.io-array-like.md) ^1.0.2
- [validate.io-object-array](https://npm.io/package/validate.io-object-array.md) ^1.0.0
- [validate.io-string-array](https://npm.io/package/validate.io-string-array.md) ^1.0.0
- [validate.io-string-primitive](https://npm.io/package/validate.io-string-primitive.md) ^1.0.1
- [validate.io-boolean-primitive](https://npm.io/package/validate.io-boolean-primitive.md) ^1.0.0
- [utils-define-read-only-property](https://npm.io/package/utils-define-read-only-property.md) ^1.0.0

## Alternatives

- [@cantoo/pdf-lib](https://npm.io/package/@cantoo/pdf-lib.md) — 297.9K weekly downloads
- [datatables.net-buttons](https://npm.io/package/datatables.net-buttons.md) — 200.1K weekly downloads
- [@ckeditor/ckeditor5-export-pdf](https://npm.io/package/@ckeditor/ckeditor5-export-pdf.md) — 167.0K weekly downloads
- [scanbot-web-sdk](https://npm.io/package/scanbot-web-sdk.md) — 15.0K weekly downloads
- [@syncfusion/ej2-angular-pdfviewer](https://npm.io/package/@syncfusion/ej2-angular-pdfviewer.md) — 8.8K weekly downloads

## Recent versions

- 2.0.0 (latest) — 2023-01-21
- 1.3.0 — 2019-07-23
- 1.2.0 — 2019-06-13
- 1.0.6 — 2018-05-06
- 1.0.5 — 2015-09-19
- 1.0.4 — 2015-09-19
- 1.0.3 — 2015-05-22
- 1.0.2 — 2015-05-21
- 1.0.1 — 2015-05-21
- 1.0.0 — 2015-04-09
- 0.2.0 — 2015-03-23
- 0.1.17 — 2015-03-17
- 0.1.16 — 2015-03-17
- 0.1.15 — 2015-03-17
- 0.1.14 — 2015-01-29
- … 14 more at https://npm.io/package/text-miner/versions

## README

[![NPM version][npm-image]][npm-url]
[![Build Status][travis-image]][travis-url]
[![Coverage Status][codecov-image]][codecov-url]

text-miner
==========

> text mining utilities for node.js

# Introduction

The text-miner package can be easily installed via npm:

``` bash
npm install text-miner
```

To require the module in a project, we can use the expression

``` javascript
var tm = require( 'text-miner' );
```

## Corpus

The fundamental data type in the `text-miner` module is the *Corpus*. An instance of this class wraps a collection of documents
and provides several methods to interact with this collection and perform post-processing tasks such as stemming,
stopword removal etc.

A new corpus is created by calling the constructor

``` javascript
var my_corpus = new tm.Corpus([]);
```

where `[]` is an array of text documents which form the data of the corpus. The class supports method chaining, such that mutliple methods can be invoked after each other, e.g.

``` javascript
my_corpus
	.trim()
	.toLower()
```

The following methods and properties are part of the Corpus class:

### Methods

#### `.addDoc(doc)`

Add a single document to the corpus. Has to be a string.

#### `.addDocs(docs)`

Adds a collection of documents (in form of an array of strings) to the corpus.

#### `.clean()`
Strips extra whitespace from all documents, leaving only at most one whitespace between any two other characters.

#### `.map(fun)`
Applies the function supplied to `fun` to each document in the corpus and maps each document to the result of its respective
function call.

#### `.removeInterpunctuation()`
Removes interpunctuation characters (! ? . , ; -) from all documents.

#### `.removeNewlines()`
Removes newline characters (\n) from all documents.

#### `.removeWords(words[, case_insensitive])`
Removes all words in the supplied `words` array from all documents. This function is usually invoked to remove stopwords. For convenience,
the *text-miner* package ships with a list of stopwords for different languages. These are stored in the
`STOPWORDS` object of the module.

Currently, stopwords for the following languages are included:

``` javascript
STOPWORDS.DE
STOPWORDS.EN
STOPWORDS.ES
STOPWORDS.IT
```

As a concrete example, we could remove all english stopwords from corpus `my_corpus` as follows:

``` javascript
my_corpus.removeWords( tm.STOPWORDS.EN )
```

The second (optional) parameter of the function `case_insensitive` expects a Boolean indicating whether to ignore cases or not.
The default value is `false`.

#### `.removeDigits()`

Removes any digits occuring in the texts.

#### `.removeInvalidCharacters()`

Removes all characters which are unknown or unrepresentable in Unicode.

#### `.stem(type)`
Performs stemming of the words in each document. Two stemmers are supported: Porter and Lancaster. The former is the default
option. Passing "Lancaster" to the `type` parameter of the function ensured that the latter one is used.

#### `.toLower()`
Converts all characters in the documents to lower-case.

#### `.toUpper()`
Converts all characters in the documents to upper-case.

#### `.trim()`
Strips off whitespace at the beginning and end of each document.

## DocumentTermMatrix / TermDocumentMatrix

We can pass a corpus to the constructor `DocumentTermMatrix` in order to create a document-term-matrix or a term-document matrix. Objects derived from either share the same methods, but differ in how the underlying matrix is represented: A `DocumentTermMatrix` has documents on its rows and columns corresponding to words, whereas a `TermDocumentMatrix` has rows corresponding to words and columns to documents.

``` javascript
var terms = new tm.DocumentTermMatrix( my_corpus );
```

An instance of either `DocumentTermMatrix` or `TermDocumentMatrix` has the following properties:

### Properties

#### `.vocabulary`
An array holding all the words occuring in the corpus, in order corresponding to the column entries of the document-term matrix.

#### `.data`
The document-term or term-document matrix, implemented as a nested array in JavaScript. Rows correspond to individual documents, while each column index corresponds to the respective word in `vocabulary`. Each entry of `data` holds the number of counts the word appears in the respective documents. The array is sparse, such that each entry which is undefined corresponds to a value of zero.

#### `.nDocs`
The number of documents in the term matrix

#### `.nTerms`
The number of distinct words appearing in the documents

### Methods

#### `.findFreqTerms( n )`

Returns all terms in alphabetical ordering which appear `n` or more times in the corpus. The return value is an array of objects of the form
`{word: "<word>", count: <number>}`.

#### `.removeSparseTerms( percent )`

Remove all words from the document-term matrix which appear in less than `percent` of the documents.

#### `.weighting( fun )`

Apply a weighting scheme to the entries of the document-term matrix. The `weighting` method expects a function as its argument, which is then applied to each entry of the document-term matrix. Currently, the function `weightTfIdf`, which calculates the term-frequency inverse-document-frequency (TfIdf) for each word, is the only built-in weighting function.  

#### `.fill_zeros()`

Turn the document-term matrix `dtm` into a non-sparse matrix by replacing each value which is `undefined` by zero and save the result.

## Utils

The module exports several other utility functions.

#### `.expandContractions( str )`

Replaces all occuring English contractions by their expanded equivalents, e.g. "don't" is changed to
"do not". The resulting string is returned.

#### `.weightTfIdf( terms )`

Weights document-term or term-document matrix `terms` by term frequency - inverse document frequency. *Mutates* the input `DocumentTermMatrix` or `TermDocumentMatrix` object.

## Data

#### .STOPWORDS

An object with four keys: `DE`, `EN`, `ES` and `IT`, each of which is an `array` of stopwords for the German, English, Spanish and Italian language, respectively.

``` javascript
{
	"EN": [
		"a",
		"a's",
		"able",
		"about",
		"above",
		// (...)  
	],
	"DE": [
		// (...)
	],
	// (...)
}
```

#### .CONTRACTIONS

The keys of the `CONTRACTIONS` object are the contracted expressions and the corresponding values are `arrays` of the possible expansions.

``` javascript
{
	"ain't": ["am not", "are not", "is not", "has not","have not"],
	"aren't": ["are no", "am not"],
	"can't": ["cannot"],
	// (...)
}
```

## Unit Tests

Run tests via the command `npm test`

---
## License

[MIT license](http://opensource.org/licenses/MIT).

[npm-image]: https://badge.fury.io/js/text-miner.svg
[npm-url]: http://badge.fury.io/js/text-miner

[travis-image]: https://travis-ci.org/Planeshifter/text-miner.svg
[travis-url]: https://travis-ci.org/Planeshifter/text-miner

[codecov-image]: https://img.shields.io/codecov/c/github/Planeshifter/text-miner/master.svg
[codecov-url]: https://codecov.io/github/Planeshifter/text-miner?branch=master

---
_Source: https://npm.io/package/text-miner · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._