rehype-extract-content v0.0.0

rehype-extract-content

Rehype plugin to extract essential contents from HTML documents, like articles, blogs, ebooks, docs, and wikis.

Install

npm install rehype-extract-content

Use

import { got } from 'got'
import { unified } from 'unified'
import rehypeParse from 'rehype-parse'
import rehypeExtractContent from '../index.js'

const html = await got.get('https://github.com/gorango/rehype-extract-content').then(res => res.body)
const file = { value: html }
const tree = unified().use(rehypeParse).parse(file)
const value = await unified().use(rehypeExtractContent).run(tree, file)

console.log(value)

Running the above will return sanitized HTML containing only the salient content from the original document:

{
  type: 'root',
  children: [
    { tagName: 'h1', children: [Array], /* ... */ },
    { tagName: 'p', children: [Array], /* ... */ },
    { tagName: 'h2', children: [Array], /* ... */ },
    { tagName: 'pre', children: [Array], /* ... */ },
    { tagName: 'h2', children: [Array], /* ... */ },
    { tagName: 'pre', children: [Array], /* ... */ },
    { tagName: 'p', children: [Array], /* ... */ },
    { tagName: 'pre', children: [Array], /* ... */ },
    /* ... */
  ]
}

Content will always be returned unnested from any unsemantic tags (like divs and sections).

By default, most classes and props get stripped from the original tree. Props with relative URLs get updated with absolute URLs (like src and href). When semantic tags are missing (like <p>s and <figure>s), those nodes will get wrapped.

See the test/ folder for examples and edge-cases, as well as options for modifying various stages of the plugin.

API

This package exports a single plugin function.

unified().use(rehypeExtractContent[, options])

The plugin executes a series of traversals to find, sanitize, and format the content. Each stage can be modified using the following options:

options

Configuration (optional).

options.container

To skip the search traversal for finding the container, you can provide your own selector:

• String selector supported by hast-util-select.
• Function that receives a hast Node evaluated for truthiness.

.use(rehypeExtractContent, {
  // container: 'article.post',
  container: (node) => node.tagName === 'article'
      && node.properties?.className?.includes('post'),
})

options.exclude

Exclude specific nodes with one or more selectors containing either:

• String selector supported by hast-util-select.
• Function that receives a hast Node evaluated for truthiness.

.use(rehypeExtractContent, {
  exclude: [
    '.ad-container',
    (node) => node.properties.className?.includes('ad-container'),
  ],
})

options.schema

Schema for hast-util-sanitize, merged with defaults.

If you want to preserve code highlights from the original document, for example:

.use(rehypeExtractContent, {
  schema: {
    attributes: {
      'pre': ['className'],
      'code': ['className'],
      'span': ['className'],
    },
    tagNames: ['span'],
  },
})

options.host

Prepend the host string to all internal URLs on the page. Applies to nodes with href and src props (like a, img, video).

If rehypeExtractMeta plugin is used before rehypeExtractContent, the url value from file.data.meta will be used as a fallback. If not, the plugin will try to infer the host from from the meta tags in the tree.

options.target

Add a target prop to all anchor tags. Defaults to "_blank". Setting to null will exclude the prop.

options.ref

Add a ref prop to all anchor tags. Defaults to null.

options.format

A function that executes on each hast Node in the final format stage.

.use(rehypeExtractContent, {
  format: (node) => {
    if (node.tagName.startsWith('h'))
      node.value = node.value.toUpperCase()
  },
})