# mnl-ws-norm

> Light-weight tool for normalizing whitespace, splitting lines, and accurately tokenizing words (no regex). Multiple natural languages supported.

Latest version **1.0.3** (published 2021-10-29) · Apache-2.0 license · 0 weekly downloads

## Install

```sh
npm install mnl-ws-norm
pnpm add mnl-ws-norm
yarn add mnl-ws-norm
bun add mnl-ws-norm
```

## Health

**Score 20/100 (F)** — status: abandoned.

Positive: esm support; no vulnerabilities.

Warnings: low downloads; no types.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.0.3 |
| Published | 2021-10-29 |
| First published | 2021-10-23 |
| Weekly downloads | 0 |
| License | Apache-2.0 |
| TypeScript types | none |
| Module format | ESM + CommonJS |
| Dependencies | 0 |
| Unpacked size | 10.7 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 0 |
| Author | Rairye |
| Maintainers | rairye-nlp |
| Keywords | whitespace, normalization, line, splitting, noregex |

## Links

- npm: https://www.npmjs.com/package/mnl-ws-norm
- Repository: https://github.com/Rairye/js-mnl-ws-norm
- Issues: https://github.com/Rairye/js-mnl-ws-norm/issues
- npm.io page: https://npm.io/package/mnl-ws-norm

## Alternatives

- [localforage](https://npm.io/package/localforage.md) — 6.2M weekly downloads
- [localforage-observable](https://npm.io/package/localforage-observable.md) — 30.8K weekly downloads
- [@y/y](https://npm.io/package/@y/y.md) — 30.1K weekly downloads
- [@metaobjectsdev/render](https://npm.io/package/@metaobjectsdev/render.md) — 3.5K weekly downloads
- [@ledgerhq/coin-algorand](https://npm.io/package/@ledgerhq/coin-algorand.md) — 1.1K weekly downloads

## Recent versions

- 1.0.3 (latest) — 2021-10-29
- 1.0.2 — 2021-10-29
- 1.0.1 — 2021-10-23

## README

Light-weight tool for normalizing whitespace and accurately tokenizing words (no regex). Multiple natural languages supported. Useful for scrapping, machine learning, and data analysis.

## Installation

npm install mnl-ws-norm

## function isWhiteSpace(char)

returns true if char is a whitespace character.

char must be passed as a string with a length of 1.

```javascript
import {isWhiteSpace} from 'mnl-ws-norm';

console.log("Half-width space isWhiteSpace: " + isWhiteSpace(" "));
console.log("Tab is white space: " + isWhiteSpace("	"));
console.log("'A' is white space: " + isWhiteSpace("A"));
console.log("'\\n' is white space: " + isWhiteSpace("\n"));
```

## function isLineBreak(char)

returns true if char is a line break.

char must be passed as a string with a length of 1.

```javascript
import {isLineBreak} from 'mnl-ws-norm';

console.log("'\\n' is line break: " + isLineBreak("\n"));
console.log("Half-width space is line break: " + isLineBreak(" "));
```

## function splitBySpaces(inputStr)

inputStr is the string from which words are to be tokenized. 

inputStr must be passed as a string.

Note: This function splits inputStr by all whitespace characters (spaces, line breaks, etc.).

```javascript
import {splitBySpaces} from 'mnl-ws-norm';

//Source string 1 with half-width spaces (Unicode: U+0020) and a tab (Unicode: U+0009).
var sourceStr1 = "Hey, everybody,  how are you doing?";

//Source string 2 with half-width spaces and a \n character (Unicode: U+000A).
var sourceStr2 = "Hey, everybody\nhow are you doing?";

//Source string 3 with half-width spaces and a full-width space (Unicode: U+3000).
var sourceStr3 = "Hey, everybody,	how are you doing?";

//The join method is used in this example to separate the elements in the returned array.

console.log("sourceStr1: " + splitBySpaces(sourceStr1).join("-"));
console.log("sourceStr2: " + splitBySpaces(sourceStr2).join("-"));
console.log("sourceStr3: " + splitBySpaces(sourceStr3).join("-"));
```

## function splitByLines(inputStr, removeExtraSpaces = false)

Required parameter -> inputStr

inputStr is the string from which lines are to be tokenized.

inputStr must be passed as a string.

Optional parameter -> removeExtraSpaces

By default, leading/trailing spaces are not removed from lines. Specifying removeExtraSpace as true removes leading/trailing spaces.

```javascript
import {splitBySpaces, splitByLines} from 'mnl-ws-norm';

var sourceStr = "Hey, everybody.\nHow are you doing?\rI am alright.";

var lines = splitByLines(sourceStr);

//The join method is used in this example to separate the elements in the returned array.

for (var i = 0; i < lines.length; i++) {
	console.log("Line " + i.toString() + " : " + (splitBySpaces(lines[i])).join("-"));
}
```

## function normSpaces(inputStr, spaceType, removeExtraSpaces = false)

Required parameters -> inputStr, spaceType

inputStr is the string in which the whitespace characters are to be replaced.

inputStr must be passed as a string.

spaceType is the string used to replace all whitespace characters in inputStr.

spaceType must be passed as a string.

Optional parameter -> removeExtraSpaces

By default, extra whitespace characters are not removed from inputStr.

Specifying removeExtraSpaces as true removes extra whitespace characters from inputStr.

Note: Regardless of the value of removeExtraSpaces, the returned string may have leading/trailing whitespace characters, so you may want to use the trim() method as necessary.

```javascript
import {normSpaces} from 'mnl-ws-norm';

//Source string with consecutive half-width spaces (Unicode: U+0020) and a tab (Unicode: U+0009).
var sourceStr = "  Hey,  everybody, 	how  are  you  doing?  ";

//Spaces in sourceStr are replaced with a half-width space, while extra spaces are ignored.
console.log(normSpaces(sourceStr, " "));

//Spaces in sourceStr are replaced with a half-width space, and extra spaces are removed.
console.log(normSpaces(sourceStr, " ", true));

//Spaces in source_str are replaced with a full-width space (Unicode: U+3000), and extra spaces are removed.
console.log(normSpaces(sourceStr, "　", true));
```

# Other languages

1. Python -> https://github.com/Rairye/mnl-ws-norm

---
_Source: https://npm.io/package/mnl-ws-norm · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._