text-processor v1.0.3
TextProcessor
A JavaScript library for profanity filtering and sentiment analysis.
Features
- Combines the functionalities of bad words filtering and AFINN-based sentiment analysis.
- Filters unwanted words and phrases from text.
- Analyzes the sentiment of text using the AFINN-165 wordlist and Emoji Sentiment Ranking.
Installation
npm install text-processor --save
Usage
Toxicity Analysis
var TextProcessor = require('text-processor');
const textProcessor = new TextProcessor();
const API_KEY = 'your_api_key_here'; // Replace with your actual API key
textProcessor.analyzeToxicity("Your text to analyze", API_KEY).then(result => {
console.log(result);
}).catch(err => {
console.error(err);
});
Sentiment Analysis
var result = textProcessor.analyze('Cats are stupid.');
console.dir(result); // Score: -2, Comparative: -0.666
Registering New Language for Sentiment Analysis
var frLanguage = {
labels: { 'stupide': -2 }
};
textProcessor.registerLanguage('fr', frLanguage);
var result = textProcessor.analyze('Le chat est stupide.', { language: 'fr' });
console.dir(result); // Score: -2, Comparative: -0.5
Profanity Filtering
var TextProcessor = require('text-processor'),
textProcessor = new TextProcessor();
console.log(textProcessor.clean("Don't be an ash0le")); // Don't be an ******
Placeholder Overrides for Filtering
var customTextProcessor = new TextProcessor({ placeHolder: 'x'});
customTextProcessor.clean("Don't be an ash0le"); // Don't be an xxxxxx
Adding Words to the Blacklist
textProcessor.addWords('some', 'bad', 'word');
textProcessor.clean("some bad word!") // **** *** ****!
Remove words from the blacklist
textProcessor.removeWords('hells', 'sadist');
textProcessor.clean("some hells word!"); //some hells word!
API
Table of Contents
- constructor
- isProfane
- replaceWord
- clean
- addWords
- removeWords
- registerLanguage
- analyze
- analyzeToxicity
- tokenize
- addLanguage
- getLanguage
- getLabels
- applyScoringStrategy
constructor
TextProcessor constructor. Combines functionalities of word filtering and sentiment analysis.
Parameters
options
Object TextProcessor instance options. (optional, default{}
)options.emptyList
boolean Instantiate filter with no blacklist. (optional, defaultfalse
)options.list
array Instantiate filter with custom list. (optional, default[]
)options.placeHolder
string Character used to replace profane words. (optional, default'*'
)options.regex
string Regular expression used to sanitize words before comparing them to blacklist. (optional, default/[^a-zA-Z0-9|\$|\@]|\^/g
)options.replaceRegex
string Regular expression used to replace profane words with placeHolder. (optional, default/\w/g
)options.splitRegex
string Regular expression used to split a string into words. (optional, default/\b/
)options.sentimentOptions
Object Options for sentiment analysis. (optional, default{}
)
isProfane
Determine if a string contains profane language.
Parameters
string
string String to evaluate for profanity.
replaceWord
Replace a word with placeHolder characters.
Parameters
string
string String to replace.
clean
Evaluate a string for profanity and return an edited version.
Parameters
string
string Sentence to filter.
addWords
Add word(s) to blacklist filter / remove words from whitelist filter.
Parameters
words
...anyword
...string Word(s) to add to blacklist.
removeWords
Add words to whitelist filter.
Parameters
words
...anyword
...string Word(s) to add to whitelist.
registerLanguage
Registers the specified language.
Parameters
languageCode
String Two-digit code for the language to register.language
Object The language module to register.
analyze
Performs sentiment analysis on the provided input 'phrase'.
Parameters
phrase
String Input phrase.opts
Object Options. (optional, default{}
)callback
function Optional callback.
Returns Object
analyzeToxicity
Analyzes the toxicity of a given text using the Perspective API.
Parameters
Returns Promise A promise that resolves with the analysis result.
tokenize
Remove special characters and return an array of tokens (words).
Parameters
input
string Input string
Returns array Array of tokens
addLanguage
Registers the specified language
Parameters
languageCode
String Two-digit code for the language to registerlanguage
Object The language module to register
getLanguage
Retrieves a language object from the cache, or tries to load it from the set of supported languages
Parameters
languageCode
String Two-digit code for the language to fetch
getLabels
Returns AFINN-165 weighted labels for the specified language
Parameters
languageCode
String Two-digit language code
Returns Object
applyScoringStrategy
Applies a scoring strategy for the current token
Parameters
languageCode
String Two-digit language codetokens
Array Tokens of the phrase to analyzecursor
int Cursor of the current token being analyzedtokenScore
int The score of the current token being analyzed
How it works
AFINN
AFINN is a list of words rated for valence with an integer between minus five (negative) and plus five (positive). Sentiment analysis is performed by cross-checking the string tokens (words, emojis) with the AFINN list and getting their respective scores. The comparative score is simply: sum of each token / number of tokens
. So for example let's take the following:
I love cats, but I am allergic to them.
That string results in the following:
{
score: 1,
comparative: 0.1111111111111111,
calculation: [ { allergic: -2 }, { love: 3 } ],
tokens: [
'i',
'love',
'cats',
'but',
'i',
'am',
'allergic',
'to',
'them'
],
words: [
'allergic',
'love'
],
positive: [
'love'
],
negative: [
'allergic'
]
}
- Returned Objects
- Score: Score calculated by adding the sentiment values of recognized words.
- Comparative: Comparative score of the input string.
- Calculation: An array of words that have a negative or positive valence with their respective AFINN score.
- Token: All the tokens like words or emojis found in the input string.
- Words: List of words from input string that were found in AFINN list.
- Positive: List of positive words in input string that were found in AFINN list.
- Negative: List of negative words in input string that were found in AFINN list.
In this case, love has a value of 3, allergic has a value of -2, and the remaining tokens are neutral with a value of 0. Because the string has 9 tokens the resulting comparative score looks like:
(3 + -2) / 9 = 0.111111111
This approach leaves you with a mid-point of 0 and the upper and lower bounds are constrained to positive and negative 5 respectively (the same as each token! 😸). For example, let's imagine an incredibly "positive" string with 200 tokens and where each token has an AFINN score of 5. Our resulting comparative score would look like this:
(max positive score * number of tokens) / number of tokens
(5 * 200) / 200 = 5
Tokenization
Tokenization works by splitting the lines of input string, then removing the special characters, and finally splitting it using spaces. This is used to get list of words in the string.
Benchmarks
A primary motivation for designing sentiment
was performance. As such, it includes a benchmark script within the test directory that compares it against the Sentimental module which provides a nearly equivalent interface and approach. Based on these benchmarks, running on a MacBook Pro with Node v6.9.1, sentiment
is nearly twice as fast as alternative implementations:
sentiment (Latest) x 861,312 ops/sec ±0.87% (89 runs sampled)
Sentimental (1.0.1) x 451,066 ops/sec ±0.99% (92 runs sampled)
To run the benchmarks yourself:
npm run test:benchmark
Validation
While the accuracy provided by AFINN is quite good considering it's computational performance (see above) there is always room for improvement. Therefore the sentiment
module is open to accepting PRs which modify or amend the AFINN / Emoji datasets or implementation given that they improve accuracy and maintain similar performance characteristics. In order to establish this, we test the sentiment
module against three labelled datasets provided by UCI.
To run the validation tests yourself:
npm run test:validate
Rand Accuracy
Amazon: 0.726
IMDB: 0.765
Yelp: 0.696
Testing
npm test
License
The MIT License (MIT)
Copyright (c) 2013 Michael Price
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.