9.0.5 • Published 1 year ago

html-to-text v9.0.5

Weekly downloads
435,986
License
MIT
Repository
github
Last release
1 year ago

html-to-text

lint status test status License: MIT npm npm

Advanced converter that parses HTML and returns beautiful text.

Features

  • Inline and block-level tags.
  • Tables with colspans and rowspans.
  • Links with both text and href.
  • Word wrapping.
  • Unicode support.
  • Plenty of customization options.

Changelog

Available here: CHANGELOG.md

Version 6 contains a ton of changes, so it worth to take a look at the full changelog.

Version 7 contains an important change for custom formatters.

Version 8 brings the selectors support to greatly increase the flexibility but that also changes some things introduced in version 6. Base element(s) selection also got important changes.

Version 9 drops a lot of previously deprecated options, introduces some new formatters and new capabilities for custom formatters. Now a dual-mode package (cjs and esm). CLI is moved to a separate package.

Installation

npm install html-to-text

Usage

Convert a single document:

const { convert } = require('html-to-text');
// There is also an alias to `convert` called `htmlToText`.

const options = {
  wordwrap: 130,
  // ...
};
const html = '<div>Hello World</div>';
const text = convert(html, options);
console.log(text); // Hello World

Configure html-to-text once to convert many documents with the same options (recommended for good performance when processing big batches of documents):

const { compile } = require('html-to-text');

const options = {
  wordwrap: 130,
  // ...
};
const compiledConvert = compile(options); // options passed here

const htmls = [
  '<div>Hello World!</div>',
  '<div>こんにちは世界!</div>',
  '<div>Привіт Світ!</div>'
];
const texts = htmls.map(compiledConvert);
console.log(texts.join('\n'));
// Hello World!
// こんにちは世界!
// Привіт Світ!

Both convert and compiledConvert can take one more optional argument - metadata object that can be used by formatters.

Options

General options

OptionDefaultDescription
baseElementsDescribes which parts of the input document have to be converted and present in the output text, and in what order.
baseElements.selectors['body']Elements matching any of provided selectors will be processed and included in the output text, with all inner content.Refer to Supported selectors section below.
baseElements.orderBy'selectors''selectors' - arrange base elements in the same order as baseElements.selectors array;'occurrence' - arrange base elements in the order they are found in the input document.
baseElements.returnDomByDefaulttrueConvert the entire document if none of provided selectors match.
decodeEntitiestrueDecode HTML entities found in the input HTML if true. Otherwise preserve in output text.
encodeCharacters{}A dictionary with characters that should be replaced in the output text and corresponding escape sequences.
formatters{}An object with custom formatting functions for specific elements (see Override formatting section below).
limitsDescribes how to limit the output text in case of large HTML documents.
limits.ellipsis'...'A string to insert in place of skipped content.
limits.maxBaseElementsundefinedStop looking for more base elements after reaching this amount. Unlimited if undefined.
limits.maxChildNodesundefinedMaximum number of child nodes of a single node to be added to the output. Unlimited if undefined.
limits.maxDepthundefinedStop looking for nodes to add to the output below this depth in the DOM tree. Unlimited if undefined.
limits.maxInputLength16_777_216If the input string is longer than this value - it will be truncated and a message will be sent to stderr. Ellipsis is not used in this case. Unlimited if undefined.
longWordSplitDescribes how to wrap long words.
longWordSplit.wrapCharacters[]An array containing the characters that may be wrapped on. Checked in order, search stops once line length requirement can be met.
longWordSplit.forceWrapOnLimitfalseBreak long words at the line length limit in case no better wrap opportunities found.
preserveNewlinesfalseBy default, any newlines \n from the input HTML are collapsed into space as any other HTML whitespace characters. If true, these newlines will be preserved in the output. This is only useful when input HTML carries some plain text formatting instead of proper tags.
selectors[]Describes how different HTML elements should be formatted. See Selectors section below.
whitespaceCharacters' \t\r\n\f\u200b'A string of characters that are recognized as HTML whitespace. Default value uses the set of characters defined in HTML4 standard. (It includes Zero-width space compared to living standard.)
wordwrap80After how many chars a line break should follow.Set to null or false to disable word-wrapping.

Deprecated or removed options

Old optionDepr.Rem.Instead use
baseElement8.0baseElements: { selectors: [ 'body' ] }
decodeOptions9.0Entity decoding is now handled by htmlparser2 itself and entities internally. No user-configurable parts compared to he besides boolean decodeEntities.
format6.0The way formatters are written has changed completely. New formatters have to be added to the formatters option, old ones can not be reused without rewrite. See new instructions below.
hideLinkHrefIfSameAsText6.09.0selectors: [ { selector: 'a', options: { hideLinkHrefIfSameAsText: true } } ]
ignoreHref6.09.0selectors: [ { selector: 'a', options: { ignoreHref: true } } ]
ignoreImage6.09.0selectors: [ { selector: 'img', format: 'skip' } ]
linkHrefBaseUrl6.09.0selectors: [{ selector: 'a', options: { baseUrl: 'https://example.com' } },{ selector: 'img', options: { baseUrl: 'https://example.com' } }]
noAnchorUrl6.09.0selectors: [ { selector: 'a', options: { noAnchorUrl: true } } ]
noLinkBrackets6.09.0selectors: [ { selector: 'a', options: { linkBrackets: false } } ]
returnDomByDefault8.0baseElements: { returnDomByDefault: true }
singleNewLineParagraphs6.09.0selectors: [{ selector: 'p', options: { leadingLineBreaks: 1, trailingLineBreaks: 1 } },{ selector: 'pre', options: { leadingLineBreaks: 1, trailingLineBreaks: 1 } }]
tables8.0selectors: [ { selector: 'table.class#id', format: 'dataTable' } ]
tags8.0See Selectors section below.
unorderedListItemPrefix6.09.0selectors: [ { selector: 'ul', options: { itemPrefix: ' * ' } } ]
uppercaseHeadings6.09.0selectors: [{ selector: 'h1', options: { uppercase: false } },...{ selector: 'table', options: { uppercaseHeaderCells: false } }]

Other things removed:

  • fromString method - use convert or htmlToText instead;
  • positional arguments in BlockTextBuilder methods - pass option objects instead.

Selectors

Some example:

const { convert } = require('html-to-text');

const html = '<a href="/page.html">Page</a><a href="!#" class="button">Action</a>';
const text = convert(html, {
  selectors: [
    { selector: 'a', options: { baseUrl: 'https://example.com' } },
    { selector: 'a.button', format: 'skip' }
  ]
});
console.log(text); // Page [https://example.com/page.html]

Selectors array is our loose approximation of a stylesheet.

  • highest specificity selector is used when there are multiple matches;
  • the last selector is used when there are multiple matches of equal specificity;
  • all entries with the same selector value are merged (recursively) at the compile stage, in such way so the last defined properties a kept and the relative order of unique selectors is kept;
  • user-defined entries are appended after predefined entries;
  • Every unique selector must have format value specified (at least once);
  • unlike in CSS, values from different matched selectors are NOT merged at the convert stage. Single best match is used instead (that is the last one of those with highest specificity).

To achieve the best performance when checking each DOM element against provided selectors, they are compiled into a decision tree. But it is also important how you choose selectors. For example, div#id is much better than #id - the former will only check divs for the id while the latter has to check every element in the DOM.

Supported selectors

html-to-text relies on parseley and selderee packages for selectors support.

Following selectors can be used in any combinations:

  • * - universal selector;
  • div - tag name;
  • .foo - class name;
  • #bar - id;
  • [baz] - attribute presence;
  • [baz=buzz] - attribute value (with any operators and also quotes and case sensitivity modifiers - syntax);
  • + and > combinators (other combinators are not supported).

You can match <p style="...; display:INLINE; ...">...</p> with p[style*="display:inline"i] for example.

Predefined formatters

Following selectors have a formatter specified as a part of the default configuration. Everything can be overridden, but you don't have to repeat the format or options that you don't want to override. (But keep in mind this is only true for the same selector. There is no connection between different selectors.)

SelectorDefault formatNotes
*inlineUniversal selector.
aanchor
articleblock
asideblock
blockquoteblockquote
brlineBreak
divblock
footerblock
formblock
h1heading
h2heading
h3heading
h4heading
h5heading
h6heading
headerblock
hrhorizontalLine
imgimage
mainblock
navblock
olorderedList
pparagraph
prepre
tabletableEquivalent to block. Use dataTable instead for tabular data.
ulunorderedList
wbrwbr

More formatters also available for use:

FormatDescription
dataTableFor visually-accurate tables. Note that this might be not search-friendly (output text will look like gibberish to a machine when there is any wrapped cell contents) and also better to be avoided for tables used as a page layout tool.
skipSkips the given tag with it's contents without printing anything.
blockStringInsert a block with the given string literal (formatOptions.string) instead of the tag.
blockTagRender an element as HTML block bag, convert it's contents to text.
blockHtmlRender an element with all it's children as HTML block.
inlineStringInsert the given string literal (formatOptions.string) inline instead of the tag.
inlineSurroundRender inline element wrapped with given strings (formatOptions.prefix and formatOptions.suffix).
inlineTagRender an element as inline HTML tag, convert it's contents to text.
inlineHtmlRender an element with all it's children as inline HTML.
Format options

Following options are available for built-in formatters.

OptionDefaultApplies toDescription
leadingLineBreaks1, 2 or 3all block-level formattersNumber of line breaks to separate previous block from this one.Note that N+1 line breaks are needed to make N empty lines.
trailingLineBreaks1 or 2all block-level formattersNumber of line breaks to separate this block from the next one.Note that N+1 line breaks are needed to make N empty lines.
baseUrlnullanchor, imageServer host for link href attributes and image src attributes relative to the root (the ones that start with /).For example, with baseUrl = 'http://asdf.com' and <a href='/dir/subdir'>...</a> the link in the text will be http://asdf.com/dir/subdir.
linkBrackets['[', ']']anchor, imageSurround links with these brackets.Set to false or ['', ''] to disable.
pathRewriteundefinedanchor, imageA function to rewrite link href attributes and image src attributes. Optional second argument is the metadata object.Applied before baseUrl.
hideLinkHrefIfSameAsTextfalseanchorBy default links are translated in the following way:<a href='link'>text</a> => becomes => text [link].If this option is set to true and link and text are the same, [link] will be omitted and only text will be present.
ignoreHreffalseanchorIgnore all links. Only process internal text of anchor tags.
noAnchorUrltrueanchorIgnore anchor links (where href='#...').
itemPrefix' * 'unorderedListString prefix for each list item.
uppercasetrueheadingBy default, headings (<h1>, <h2>, etc) are uppercased.Set this to false to leave headings as they are.
lengthundefinedhorizontalLineLength of the line. If undefined then wordwrap value is used. Falls back to 40 if that's also disabled.
trimEmptyLinestrueblockquoteTrim empty lines from blockquote.While empty lines should be preserved in HTML, space-saving behavior is chosen as default for convenience.
uppercaseHeaderCellstruedataTableBy default, heading cells (<th>) are uppercased.Set this to false to leave heading cells as they are.
maxColumnWidth60dataTableData table cell content will be wrapped to fit this width instead of global wordwrap limit.Set this to undefined in order to fall back to wordwrap limit.
colSpacing3dataTableNumber of spaces between data table columns.
rowSpacing0dataTableNumber of empty lines between data table rows.
string''blockString, inlineStringA string to be inserted in place of a tag.
prefix''inlineSurroundString prefix to be inserted before inline tag contents.
suffix''inlineSurroundString suffix to be inserted after inline tag contents.
Deprecated format options
Old optionApplies toDepr.Rem.Instead use
noLinkBracketsanchor8.1linkBrackets: false

Override formatting

formatters option is an object that holds formatting functions. They can be assigned to format different elements in the selectors array.

Each formatter is a function of four arguments that returns nothing. Arguments are:

  • elem - the HTML element to be processed by this formatter;
  • walk - recursive function to process the children of this element. Called as walk(elem.children, builder);
  • builder - BlockTextBuilder object. Manipulate this object state to build the output text;
  • formatOptions - options that are specified for a tag, along with this formatter (Note: if you need general html-to-text options - they are accessible via builder.options).

Custom formatter example:

const { convert } = require('html-to-text');

const html = '<foo>Hello World</foo>';
const text = convert(html, {
  formatters: {
    // Create a formatter.
    'fooBlockFormatter': function (elem, walk, builder, formatOptions) {
      builder.openBlock({ leadingLineBreaks: formatOptions.leadingLineBreaks || 1 });
      walk(elem.children, builder);
      builder.addInline('!');
      builder.closeBlock({ trailingLineBreaks: formatOptions.trailingLineBreaks || 1 });
    }
  },
  selectors: [
    // Assign it to `foo` tags.
    {
      selector: 'foo',
      format: 'fooBlockFormatter',
      options: { leadingLineBreaks: 1, trailingLineBreaks: 1 }
    }
  ]
});
console.log(text); // Hello World!

New in version 9: metadata object can be provided as the last optional argument of the convert function (or the function returned by compile function). It can be accessed by formatters as builder.metadata.

Refer to generic formatters of the base package and text formatters of this package for more examples. The easiest way to write your own is to pick an existing one and customize.

Refer to BlockTextBuilder for available functions and arguments.

Custom metadata

If you need to supply extra information about your HTML documents to use in custom formatters - it can be done with the help of metadata object.

It is supplied as an extra argument to the convert function:

import { compile, convert } from 'html-to-text';

// for batch use:
const compiledConvert = compile(options);
let text = compiledConvert(html, metadata);

// for single use:
let text = convert(html, options, metadata);

And it can be accessed within formatter functions as builder.metadata.

Call other formatters from a custom formatter

Most of the times this is not what you actually need. Most practical problems can be solved with selectors.

If you really need to inspect the node internals, not just attributes, then you can do it like this:

const options = {
  // ...
  formatters: {
    filterBlockFormatter: function (elem, walk, builder, formatOptions) {
      // all built-in and custom formatters available by name
      const blockFormatter = builder.options.formatters['block'];
      if (blockFormatter && elem.children.some(/* predicate */)) {
        blockFormatter(elem, walk, builder, formatOptions);
      }
    }
  },
  selectors: [
    {
      selector: 'div.questionable',
      format: 'filterBlockFormatter',
      options: { leadingLineBreaks: 1, trailingLineBreaks: 1 }
    }
  ],
  // ...
}

Example

Contributors

License

MIT License

@me2day/rest@modern-js/doc-corecoveo-interface-editornexcocuplan-mailer@bongnv/gridsome-transformer-remarkrectified-iocer-notifications-apisuhie-connectewokin.rssthecoder08-googlerbcgj-mind-coreaws-assistantaws-ses-helperkreta-clicontent-ingest-images-v1ratbird@pranabkalita/express.mvchubot-basecampbrand-techmutasi-bcaglue-js-documentation-builder@ourharvest/email-templatesnodereactorenglish-level-analyser@godeep-dev/svkit-docs-tools@jkoenig134/matrix-bot-sdk@mkiloyan/core@mkiloyan/plugins@mkiloyan/renderer-react@useherald/react-widgetpomment-backend@infinitebrahmanuniverse/nolb-html-t@lcnc/commonprocessing-rncpthing-it-serverjob-web-appesante-api@everything-registry/sub-chunk-1868hexo-aurora-auto-excerpt-abcddhexo-aurora-auto-excerpt-js@tepez/jasmine-misc-matchers@andrewcturing/gmail@andrewcturing/gmail_custom_oauth@andris/emailengine@actionsflow/trigger-activitypub@addev-be/blabels-integration@addev-be/blabels-types@anchorjetti-shared/shared@app-masters/node-lib@armada-inc/gatsby-wordpress-theme-libre@alkuip/dhis2@almedso/cosmea-core@a_kawashiro/jendeley@3wks/gae-node-nestjs@autorest/go@axah/postie-server@cats-cradle/email-service@cats-cradle/message-broker@coderspirit/lyra-astro@datarailsshared/ai-service-types@darekkay/11ty@data-fair/processing-rncp-rs@dada513/wikipedia-search@dcl/docs-site@density/postcard@bongnv/transformer-remark@brightslides/api-core@blog-cms/common@bradford/svelte-mail@calipsa/mailparser@gestya/common-components@ggabella-photo-share/commonrss-to-jekyllrunbox-mailparserrutracker-suckersanechainsearch-gptsendingnetwork-bot-sdkscrivitoses-template-mailerxenforo-dlvelitquozc-ui-mobileul-apiultrasesvnc-libraryvscode-azureextensionuivlabs-buildexpvue-mailervue-satoritpl-emailstext-extractorsunireadvuepress-theme-vtwa-chat-server-facebookwalter-backendwaveorb-mailervuepress-theme-grootworshiptogether
9.0.5

1 year ago

9.0.4

1 year ago

9.0.3

1 year ago

9.0.0-preview2

1 year ago

9.0.0-preview1

1 year ago

9.0.2

1 year ago

9.0.1

1 year ago

9.0.0

1 year ago

7.1.3

2 years ago

7.1.2

2 years ago

8.2.1

2 years ago

8.2.0

2 years ago

8.1.1

2 years ago

8.1.0

2 years ago

8.0.0

3 years ago

7.1.1

3 years ago

7.1.0

3 years ago

7.0.0

3 years ago

6.0.0

3 years ago

5.1.1

5 years ago

5.1.0

5 years ago

5.0.0

5 years ago

4.0.0

6 years ago

3.3.0

7 years ago

3.2.0

7 years ago

3.1.0

7 years ago

3.0.0

7 years ago

2.1.3

8 years ago

2.1.2

8 years ago

2.1.1

8 years ago

2.1.0

8 years ago

2.0.0

8 years ago

1.6.2

8 years ago

1.6.1

8 years ago

1.6.0

8 years ago

1.5.1

8 years ago

1.5.0

8 years ago

1.4.0

8 years ago

1.3.2

9 years ago

1.3.1

9 years ago

1.3.0

9 years ago

1.2.1

9 years ago

1.2.0

9 years ago

1.1.1

9 years ago

1.1.0

9 years ago

1.0.0

9 years ago

0.1.0

10 years ago

0.0.8

10 years ago

0.0.7

10 years ago

0.0.6

11 years ago

0.0.5

12 years ago

0.0.4

12 years ago

0.0.3

12 years ago

0.0.2

12 years ago

0.0.1

12 years ago