# anansi-js

> Node.js web spider for site search

Latest version **0.0.5** (published 2024-04-13) · MIT license · 0 weekly downloads

## Install

```sh
npm install anansi-js
pnpm add anansi-js
yarn add anansi-js
bun add anansi-js
```

## Health

**Score 25/100 (F)** — status: abandoned.

Positive: has types; esm support; no vulnerabilities.

Warnings: low downloads; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.0.5 |
| Published | 2024-04-13 |
| First published | 2023-07-26 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Dependencies | 0 |
| Unpacked size | 198.2 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 1 |
| Author | giladbeer |
| Maintainers | giladbeer |

## Links

- npm: https://www.npmjs.com/package/anansi-js
- Repository: https://github.com/anansi-js/anansi
- Homepage: https://github.com/anansi-js/anansi#readme
- Issues: https://github.com/anansi-js/anansi/issues
- npm.io page: https://npm.io/package/anansi-js

## Recent versions

- 0.0.5 (latest) — 2024-04-13
- 0.0.4 — 2024-04-02
- 0.0.3 — 2023-07-26
- 0.0.2 — 2023-07-26
- 0.0.1 — 2023-07-26

## README

![github-banner-new](assets/github-banner-new.jpeg)

[Visit docs website](https://anansi-docs.netlify.app/)


[![Build Status](https://github.com/anansi-js/anansi/actions/workflows/quality-checks.yml/badge.svg)](https://github.com/anansi-js/anansi/actions/workflows/quality-checks.yml)
[![npm](https://img.shields.io/npm/v/anansi-js)](https://www.npmjs.com/package/anansi-js)
[![Known Vulnerabilities](https://snyk.io/test/github/anansi-js/anansi/badge.svg)](https://snyk.io/test/github/anansi-js/anansi)
[![MIT License](https://img.shields.io/npm/l/anansi-js.svg)](#license)


A Node.js web spider for site search. Inspired by the deprecated https://github.com/algolia/docsearch-scraper
NOTE: The project is in a very early stage.

## overview

Anansi lets you crawl your website, scrape content that matches html selectors you specified in a config file, then index them in a search engine (currently supports [Algolia](https://www.algolia.com/) as well as your own custom search plugin) to serve your site search features.

Under the hood, the project uses [puppeteer-cluster](https://github.com/thomasdondorf/puppeteer-cluster), which in turn uses [puppeteer](https://github.com/puppeteer/puppeteer)

## getting started

### installation

npm
```sh
npm install --save puppeteer # the project uses puppeteer-cluster and puppeteer under the hood
npm install --save puppeteer-cluster # the project uses puppeteer-cluster and puppeteer under the hood
npm install --save anansi-js
```

yarn
```sh
yarn add puppeteer puppeteer-cluster anansi-js
```

### usage

```js
import { crawlSite } from 'anansi-js';

const letsStartCrawling = async () => {
  await crawlSite({
      configFilePath: 'path/to/your/config.json',
      searchEngineOpts: {
        algolia: {
          apiKey: '<your algolia API key>',
          appId: '<your algolia app ID>',
          indexName: '<your algolia index name>'
        }
      },
      diagnostics: true,
      logLevel: 'debug',
      maxIndexedRecords: 300
    });
}

letsStartCrawling().then(() => {
  process.exit(0);
})
```

## API docs (WIP)

- [crawlSite](#crawlSite)

### crawlSite

instantiates a Spider object, initializing it based on your config file and settings, then invoking its `crawl` method.

#### crawlSite options:

| Property | Required | Type | Description |
| --- | --- | --- | --- |
| `configFilePath` | N | string | the path to your config json file (see sample config: https://github.com/anansi-js/anansi/blob/main/config.sample.json or [reference](#crawlsiteoptionscrawlerconfig)) |
| `config` | N | [CrawlSiteOptionsCrawlerConfig](#crawlsiteoptionscrawlerconfig) | alternatively to passing a config file path, can pass the config file's properties here |
| `searchEngineOpts` | N | SearchEngineOpts | search engine settings |
| `logLevel` | N | `"debug"` / `"warn"` / `"error"` | log level |
| `diagnostics` | N | boolean | whether or not to output diagnostics |
| `diagnosticsFilePath` | N | string | path to the file where diagnostics will be written to |
| `timeout` | N | number | timeout in ms |
| `maxIndexedRecords` | N | number | maximum number of records to index. If reached, the crawling jobs will terminate |
| `maxIndexedPages` | N | number | maximum number of pages to index. If reached, the crawling jobs will terminate |

#### CrawlSiteOptionsCrawlerConfig
| Property | Required | Type | Description |
| --- | --- | --- | --- |
| `startUrls` | Y | string / string[] | list of urls that the crawler will start from |
| `scraperSettings` | Y | [ScraperSettings](#scrapersettings) | html selectors for telling the crawler which content to scrape for indexing |
| `allowedDomains` | N | string / string[] | list of allowed domains. When not specified, defaults to the domains of your startUrls |
| `ignoreUrls` | N | string / string[] | list of url patterns to ignore |
| `maxConcurrency` | N | number | maximum concurrent puppeteer clusters to run |

#### ScraperSettings
all of the scraper settings groups (each group except the default ties to a specific URL pattern)
| Property | Required | Type | Description |
| --- | --- | --- | --- |
| `default` | Y | [ScraperPageSettings](#scraperpagesettings) | default scraper page settings - will be applied when the scraped url doesn't match any other scraper page settings group |
| `[your scraper page-level settings group name]` | N | [ScraperPageSettings](#scraperpagesettings) | page-level settings group. Can add as many as you want. Each group will be applied to a given url pattern. During crawling, the settings for each page will be chosen based on which group's  `urlPatten` field matches the page url. The default one will be chosen if no match was found |
| `shared` | Y | [ScraperPageSettings](#scraperpagesettings) | shared scraper settings - settings defined here will be applied for all pages unless there is an overriding setting in the default or the specific settings group that is matches the current page |

#### ScraperPageSettings
A group of a scraper settings - mostly hierarchy and metadata selectors, grouped by a specific URL pattern
| Property | Required | Type | Description |
| --- | --- | --- | --- |
| `hierarchySelectors` | Y | [HierarchySelectors](#hierarchyselectors) | selectors hierarchy (see below) |
| `metadataSelectors` | Y | Record<string, string> | metadata selectors. Mapping from html selectors to custom additional fields in the index, e.g. can scrape meta tags of a certain content pattern and store under a custom field |
| `urlPattern` | Y | string | URL pattern. During crawling, the settings group for each page will be chosen based on which group's `urlPatten` field matches the page url. The default one will be chosen if no match was found |
| `pageRank` | N | number | custom ranking for the matched pages. Defaults to 0 |
| `respectRobotsMeta` | N | boolean | whether or not the crawler should respect `noindex` meta tag. Defaults to false |
| `excludeSelectors` | N | string[] | list of html selectors to exclude from being scraped |
| `userAgent` | N | string | custom user agent to set when running puppeteer |
| `headers` | N | Record<string, string> | request headers to include when crawling the site |
| `basicAuth` | N | { user: string; password: string } | basic auth credentials |



#### HierarchySelectors
hierarchy selectors. Essentially a mapping from html selectors to indexed hierarchy levels
| Property | Required | Type | Description |
| --- | --- | --- | --- |
| `l0` | N | string | HTML selectors for matching l0, e.g. "span[class='myclass'], .myclass2" |
| `l1` | N | string | HTML selectors for matching l1 |
| `l2` | N | string | HTML selectors for matching l2|
| `l3` | N | string | HTML selectors for matching l3 |
| `l4` | N | string | HTML selectors for matching l4 |
| `content` | N | string | HTML selectors for matching content |

## Under the hood
For detailed information about the way the spider works, visit:
https://github.com/anansi-js/anansi/blob/main/docs/under-the-hood.md

---
_Source: https://npm.io/package/anansi-js · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._