tokenize-file v0.1.7

tokenize-file

Read a file, tokenize it, and spit out a handy JSON.

Installation

npm i tokenize-file -S

Example

var tokenizeFile = require("tokenize-file");

tokenizeFile("path/to/file.txt", tokens => {
  console.log(tokens.filter(d => !d.stop_word && d.pos !== "N"));
});

API

# tokenizeFile(path/to/file_name, callback)

Read a file, tokenize it, and spit out the JSON of the tokens. The tokenized data is passed as an array of objects to the callback function. In the array, each token is an object, represented as:

{
  value: "String", // the token
  count: Number, // the number of times it appears in the file
  pos: "String" // the token's Penn Treebank POS tag,
  stop_word: Boolean // whether the token value is a stop word, which can be filtered out in some analyses
}

tokenizeFile can read any type of file supported by textract:

• HTML, HTM
• ATOM, RSS
• Markdown
• XML, XSL
• PDF
• DOC, DOCX
• ODT, OTT (experimental, feedback needed!)
• RTF
• XLS, XLSX, XLSB, XLSM, XLTX
• CSV
• ODS, OTS
• PPTX, POTX
• ODP, OTP
• ODG, OTG
• PNG, JPG, GIF
• DXF
• application/javascript
• All text/* mime-types.

The POS tags are: