rehype-extract-content v1.7.0
rehype-extract-content
Rehype plugin to extract essential contents from HTML documents, like articles, blogs, ebooks, docs, and wikis.
Install
npm install rehype-extract-content
Use
import { got } from 'got'
import { unified } from 'unified'
import rehypeParse from 'rehype-parse'
import rehypeExtractContent from '../index.js'
const html = await got.get('https://github.com/gorango/rehype-extract-content').then(res => res.body)
const file = { value: html }
const tree = unified().use(rehypeParse).parse(file)
const value = await unified().use(rehypeExtractContent).run(tree, file)
console.log(value)
Running the above will return sanitized HTML containing only the salient content from the original document:
{
type: 'root',
children: [
{ tagName: 'h1', children: [Array], /* ... */ },
{ tagName: 'p', children: [Array], /* ... */ },
{ tagName: 'h2', children: [Array], /* ... */ },
{ tagName: 'pre', children: [Array], /* ... */ },
{ tagName: 'h2', children: [Array], /* ... */ },
{ tagName: 'pre', children: [Array], /* ... */ },
{ tagName: 'p', children: [Array], /* ... */ },
{ tagName: 'pre', children: [Array], /* ... */ },
/* ... */
]
}
Content will always be returned unnested from any unsemantic tags (like div
s and section
s).
By default, most classes and props get stripped from the original tree. Props with relative URLs get updated with absolute URLs (like src
and href
). When semantic tags are missing (like <p>
s and <figure>
s), those nodes will get wrapped.
See the test/
folder for examples and edge-cases, as well as options
for modifying various stages of the plugin.
API
This package exports a single plugin function.
unified().use(rehypeExtractContent[, options])
The plugin executes a series of traversals to find, sanitize, and format the content. Each stage can be modified using the following options:
options
Configuration (optional).
options.container
To skip the search traversal for finding the container, you can provide your own selector:
String
selector supported by hast-util-select.Function
that receives a hastNode
evaluated for truthiness.
.use(rehypeExtractContent, {
// container: 'article.post',
container: (node) => node.tagName === 'article'
&& node.properties?.className?.includes('post'),
})
options.exclude
Exclude specific nodes with one or more selectors containing either:
String
selector supported by hast-util-select.Function
that receives a hastNode
evaluated for truthiness.
.use(rehypeExtractContent, {
exclude: [
'.ad-container',
(node) => node.properties.className?.includes('ad-container'),
],
})
options.schema
Schema for hast-util-sanitize, merged with defaults
.
If you want to preserve code highlights from the original document, for example:
.use(rehypeExtractContent, {
schema: {
attributes: {
'pre': ['className'],
'code': ['className'],
'span': ['className'],
},
tagNames: ['span'],
},
})
options.host
Prepend the host string to all internal URLs on the page. Applies to nodes with href
and src
props (like a
, img
, video
).
If rehypeExtractMeta
plugin is used before rehypeExtractContent
, the url
value from file.data.meta
will be used as a fallback. If not, the plugin will try to infer the host from from the meta
tags in the tree.
options.target
Add a target
prop to all anchor tags. Defaults to "_blank"
. Setting to null
will exclude the prop.
options.ref
Add a ref
prop to all anchor tags. Defaults to null
.
options.format
A function that executes on each hast Node
in the final format
stage.
.use(rehypeExtractContent, {
format: (node) => {
if (node.tagName.startsWith('h'))
node.value = node.value.toUpperCase()
},
})