# crawl-links > takes an array of urls and crawls the urls for links Latest version **1.0.10** (published 2023-06-27) · MIT license · 0 weekly downloads ## Install ```sh npm install crawl-links pnpm add crawl-links yarn add crawl-links bun add crawl-links ``` ## Health **Score 15/100 (F)** — status: abandoned. Positive: no vulnerabilities. Warnings: low downloads; no types; no esm support. Negative: abandoned; low maintenance score. ## Facts | | | |---|---| | Version | 1.0.10 | | Published | 2023-06-27 | | First published | 2023-06-07 | | Weekly downloads | 0 | | License | MIT | | TypeScript types | none | | Module format | CommonJS | | Dependencies | 2 | | Unpacked size | 32.3 KB | | Known vulnerabilities | 0 | | Install scripts | no | | GitHub stars | 1 | | Author | Wisdom Oparaocha | | Maintainers | wisdomcsharp | | Keywords | crawl, domain, crawl, links, website, scraper | ## Links - npm: https://www.npmjs.com/package/crawl-links - Repository: https://github.com/wisdomcsharp/crawl-links - Homepage: https://github.com/wisdomcsharp/crawl-links#readme - Issues: https://github.com/wisdomcsharp/crawl-links/issues - npm.io page: https://npm.io/package/crawl-links ## Dependencies (2) - [axios](https://npm.io/package/axios.md) ^1.4.0 - [cheerio](https://npm.io/package/cheerio.md) ^1.0.0-rc.12 ## Alternatives - [@tsparticles/shape-image](https://npm.io/package/@tsparticles/shape-image.md) — 303.7K weekly downloads - [@tsparticles/shape-line](https://npm.io/package/@tsparticles/shape-line.md) — 233.7K weekly downloads - [stringify-attributes](https://npm.io/package/stringify-attributes.md) — 58.6K weekly downloads - [mobile-drag-drop](https://npm.io/package/mobile-drag-drop.md) — 46.3K weekly downloads - [@comunica/actor-rdf-parse-html](https://npm.io/package/@comunica/actor-rdf-parse-html.md) — 29.2K weekly downloads ## Recent versions - 1.0.10 (latest) — 2023-06-27 - 1.0.9 — 2023-06-07 - 1.0.8 — 2023-06-07 - 1.0.7 — 2023-06-07 - 1.0.6 — 2023-06-07 - 1.0.5 — 2023-06-07 - 1.0.4 — 2023-06-07 - 1.0.3 — 2023-06-07 - 1.0.2 — 2023-06-07 - 1.0.1 — 2023-06-07 - 1.0.0 — 2023-06-07 ## README # Crawl Links Crawl Links is a library that allows you to recursively crawl and scrape URLs from a website up to a specified depth. It provides flexibility for both Node.js and web browser environments. developed by Wisdom Oparaocha build and use dist/index.js — for web browsers [Repository](https://github.com/wisdomcsharp/crawl-links) ## Compatibility - Node.js: Version 12 or above - Web Browsers: All modern browsers that support ES6 ## Installation You can install Crawl Links using npm: ```shell npm install crawl-links ``` ## Configuration To use Crawl Links, follow these steps: Import the library into your project: ```js // For Node.js const crawlLinks = require('crawl-links'); // For Web Browser (with module bundler) import crawlLinks from 'crawl-links'; ``` ## Define the configuration options The crawl-links package provides configuration options to customize the crawling behavior. These options should be passed as properties of an options object when calling the crawlLinks function. ```js const options = { urls: ['https://example.com'], maxDepth: 2, maxConcurrentRequests: 5, sameDomain: true }; const result = await crawlLinks(options); console.log(result); ``` - `urls`: An array of URLs to crawl. The crawler will start from these URLs and recursively follow links. - `maxDepth`: The maximum depth level to crawl. The crawler will stop at this depth level and not follow further links. - `maxConcurrentRequests`: The maximum number of concurrent requests to make at a time. - `sameDomain`: If sameDomain is set to true, only links from the same domain will be crawled. If sameDomain is set to false or not provided, links from any domain will be crawled. In the example above, we define an `options` object with the `urls`, `maxDepth`, and `maxConcurrentRequests` properties. We pass this object to the crawlLinks function to initiate the crawling process. The crawler will start from the specified URLs and crawl up to a depth of 2, making a maximum of 10 concurrent requests. ## Call the crawlLinks function ```js crawlLinks(options) .then((result) => { // Process the result console.log('Scraped Links:', result.scrapedLinks); console.log('Ignored Links:', result.ignoreLinks); }) .catch((error) => { console.error('Error:', error); }); ``` ## Usage - To start crawling, provide an array of URLs in the urls option. - The crawler will visit each URL and extract all the links found on the page. - It will then follow the links to the specified depth level, scraping URLs along the way. - Concurrent requests are limited to the specified maxConcurrentRequests value. - The result object contains the scrapedLinks and ignoreLinks arrays, which can be processed accordingly. ## Examples Here are some examples of how to use Crawl Links: ### Example 1: Single-Page Scraping ```js const options = { urls: ['https://example.com'], maxDepth: 0, maxConcurrentRequests: 5 }; crawlLinks(options) .then((result) => { console.log('Scraped Links:', result.scrapedLinks); }) .catch((error) => { console.error('Error:', error); }); ``` This example performs scraping on a single page (maxDepth: 0), extracting all the links found on that page. ## Example 2: Multi-Page Scraping ```js const options = { urls: ['https://example.com'], maxDepth: 2, maxConcurrentRequests: 10, }; crawlLinks(options) .then((result) => { console.log('Scraped Links:', result.scrapedLinks); console.log('Ignored Links:', result.ignoreLinks); }) .catch((error) => { console.error('Error:', error); }); ``` This example performs scraping on a website up to a depth of 2, with a maximum of 10 concurrent requests. ## URL Normalization The crawl-links script performs URL normalization to ensure consistent and uniform URLs across different types. Here's how it handles normalization for various URL formats: ## Absolute URLs Absolute URLs, such as `https://example.com/path/to/page`, are already complete URLs that point to a specific web page. The script does not modify absolute URLs during the normalization process. They are used as-is to retrieve the webpage content. ## Relative URLs Relative URLs, such as `/path/to/page` or `../path/to/page`, are URLs that are relative to the current page's URL. The script converts relative URLs to absolute URLs by combining them with the base URL of the current page. For example, if the base URL is `https://example.com`, a relative URL of /path/to/page will be normalized to `https://example.com/path/to/page`. ## Protocol-less URLs Protocol-less URLs, such as `//example.com/path/to/page`, do not specify a protocol (e.g., `http` or `https`). The script automatically adds the appropriate protocol based on the page's URL. For example, if the current page URL is `https://example.com`, a protocol-less URL of `//example.com/path/to/page` will be normalized to `https://example.com/path/to/page`. ## Fragment URLs Fragment URLs, such as `https://example.com/page#section`, include a fragment identifier starting with a # symbol. The script removes the fragment part during normalization to avoid duplicate URLs. For example, `https://example.com/page#section` will be normalized to `https://example.com/page`. ## Trailing Slashes The script removes trailing slashes from URLs to ensure consistency. For example, `https://example.com/path/` will be normalized to `https://example.com/path`. By performing these normalization techniques, the crawl-links script ensures that URLs are consistent, avoids duplicates caused by different URL variations, and facilitates proper navigation through the website's structure. ## Contribution Contributions, issues, and feature requests are welcome! Feel free to check the GitHub repository and contribute to make Crawl Links even better. License This project is licensed under the [MIT License](https://opensource.org/license/mit/) --- _Source: https://npm.io/package/crawl-links · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._