@disqada/scraper v2.0.3
Scraper
Badges
Table of Contents
About
This tool uses its own parsing of the HTML into JSON, then this package scrolls through it to find the desired information.
Read this for more information. Or visit this website for the API types
License
Copyright © 2024 DisQada
This tool is licensed under the Apache 2.0 License.
See the LICENSE file for more information.
Getting Started
Basic information
The package breaks down the HTML into JSON nodes, following the concept that an HTML tag is represented as a JSON object/node.
Each node represents an HTML tag with it's tag name, attributes and children nodes. The children nodes may contain text as string values
Note that the parser completely ignores HTML comments
(<!--example-->)
Usage
First and most importantly, we need to have our HTML string ready in a variable.
If you're going to use the same HTML string for multiple uses, it's better to parse it yourself and then pass the JSON output to the functions (so the HTML string will be parsed only once). The following example shows how to do this:
import { parse } from '@disqada/scraper'
// Use another package to fetch the HTML from the web
const html = `
<html>
<head>
<title>Test Page</title>
</head>
<body>
<h1 id="title">Hello, world!</h1>
<p class="content">This is a test paragraph.</p>
</body>
</html>
`
const nodes = parse(html)
// Rest of the code ...A full node
import { findNode } from '@disqada/scraper'
const node = findNode(nodes, {
tag: 'h1',
attr: {
key: 'id',
value: 'title'
}
})
// node = {
// type: 'element',
// tagName: 'h1'
// attributes: [{key: 'id', value: 'title' }],
// children: []
// }A text value
import { grabText } from '@disqada/scraper'
const text = grabText(nodes, {
tag: 'p'
})
// text = 'This is a test paragraph.'TextOptions
The function grabText can be given a TextOptions object that specifies some configurations for the search process. Note that it's optional.
Click on the blue highlighted
TextOptionsto read more.
An attribute value
import { grabAttr } from '@disqada/scraper'
const attr = grabAttr(nodes, { tag: 'p' }, 'class')
// attr = 'content'CLI commands
download
You can download an HTML file and its parsed JSON file under the scrap folder in the root path of your project outside runtime by calling the download command CLI.
Arguments
Note that either
--urlor--pathmust be given
| Arg name | required | Description |
|---|---|---|
--file | true | Name of the downloaded HTML and parsed JSON file |
--url | false | Link of the web page |
--path | false | Path of a local HTML file (the HTML file will be copied to the scrap folder) |
Examples
scraper --url='https://example.com/sample' --file='sample'scraper --path='./samples/v1/index.html' --file='sample1'