# spiderable-middleware

> Boost organic traffic from SERPs, AI Chat Bots, and GPT

Latest version **2.4.0** (published 2026-11-19) · BSD-3-Clause license · 0 weekly downloads

## Install

```sh
npm install spiderable-middleware
pnpm add spiderable-middleware
yarn add spiderable-middleware
bun add spiderable-middleware
```

## Health

**Score 70/100 (B)** — status: active.

Positive: has types; esm support; no vulnerabilities; recently updated; high maintenance score; high quality score.

Warnings: low downloads.

## Facts

| | |
|---|---|
| Version | 2.4.0 |
| Published | 2026-11-19 |
| First published | 2016-03-12 |
| Weekly downloads | 0 |
| License | BSD-3-Clause |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Node | >=10.9.0 |
| Dependencies | 0 |
| Unpacked size | 59.7 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 43 |
| Author | dr.dimitru |
| Maintainers | dr.dimitru |
| Keywords | _escaped_fragment_, crawl, SEO, middleware, spiderable, crawlble, prerender, pre-rendering, prerendering, ssr, server side rendering, ajax, seo, angular, backbone, emberjs, meteor, meteorjs, express, connect, http, https, ostr.io |

## Links

- npm: https://www.npmjs.com/package/spiderable-middleware
- Repository: https://github.com/veliovgroup/spiderable-middleware
- Issues: https://github.com/veliovgroup/spiderable-middleware/issues
- npm.io page: https://npm.io/package/spiderable-middleware

## Alternatives

- [launchdarkly-js-client-sdk](https://npm.io/package/launchdarkly-js-client-sdk.md) — 2.5M weekly downloads
- [@elastic/elasticsearch](https://npm.io/package/@elastic/elasticsearch.md) — 2.1M weekly downloads
- [@c8y/client](https://npm.io/package/@c8y/client.md) — 15.3K weekly downloads
- [@signaldb/maverickjs](https://npm.io/package/@signaldb/maverickjs.md) — 1.7K weekly downloads
- [@bbc/http-transport-cache](https://npm.io/package/@bbc/http-transport-cache.md) — 1.2K weekly downloads

## Recent versions

- 2.4.0 (latest) — 2026-11-19
- 2.3.1 — 2025-12-02
- 2.3.0 — 2025-07-08
- 2.2.0 — 2025-02-14
- 2.1.0 — 2025-02-11
- 2.0.0 — 2024-02-09
- 1.6.6 — 2022-06-09
- 1.6.5 — 2022-05-29
- 1.6.4 — 2022-05-02
- 1.6.3 — 2021-09-11
- 1.6.2 — 2020-11-02
- 1.6.1 — 2020-08-17
- 1.6.0 — 2020-05-26
- 1.5.1 — 2020-05-04
- 1.5.0 — 2020-03-09
- … 23 more at https://npm.io/package/spiderable-middleware/versions

## README

# Spiderable middleware

__Spiderable Middleware__ is a lightweight Node.js package designed for modern JavaScript web apps — including those built with React, Preact, Vue, Svelte, Angular, Ember, Backbone, Meteor, and others. It ensures that search engine crawlers and social media bots receive fully-rendered HTML, not just an empty skeleton. This bridges the gap between client-rendered apps and the indexing and preview requirements of platforms like Google, Facebook, and Twitter.

When paired with the global CDN and smart caching layer from [`ostr.io`](https://ostr.io), `spiderable-middleware` significantly improves SEO scores, link previews, and performance metrics like TTFB and Lighthouse scores — all without requiring changes to existing codebase. It intelligently reroutes bot traffic to `ostr.io` rendering endpoints, minimizing server load, reducing database queries, and lowering infrastructure costs.

## Why Pre-render?

- 🕸 Execute JavaScript, — get rendered HTML page and its content;
- 🏎️ Improve delivery for dynamic and static pages via our advanced CDN and caching;
- 🏃‍♂️ Boost response rate and decrease response time with caching;
- 🚀 Optimized HTML markup for the best SEO score;
- 🎛️ Improve TTFB, LCP, INP, CLS, and other Web Vitals and LightHouse metrics positively enhancing overall SEO score;
- 🖥 Supports PWAs and SPAs;
- 📱 Supports mobile-like crawlers;
- 💅 Supports [`styled-components`](https://styled-components.com);
- ⚡️ Supports [AMP (Accelerated Mobile Pages)](https://www.ampproject.org);
- 🤓 Works with `Content-Security-Policy` and other complex front-end security rules;
- 📦 This package shipped with [types](https://github.com/veliovgroup/spiderable-middleware/blob/master/types/index.d.ts) and [TS examples](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/);
- ❤️ Search engines and social network crawlers love straightforward and pre-rendered pages;
- 📱 Consistent link previews in messaging apps, like iMessage, Messages, Facebook, Slack, Telegram, WhatsApp, Viber, VK, Twitter, and other apps;
- 💻 Image, title, and description previews for links posted at social networks, like Facebook, X/Twitter, Instagram, and other social networks.

## Related packages and docs

- __Related Packages__
  - [Next.js Integration](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/nextjs-prerendering.md)
  - [Meteor.js Integration via Atmosphere Package](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/meteor-atmosphere.md)
- __Cloud Integrations__
  - [CloudFlare Worker Integration](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/cloudflare-worker.md)
  - [Netlify Integration](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/netlify-prerendering.md)
- __Web Server Integrations__
  - [Nginx Integration](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/nginx.md)
  - [Apache Integration](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/apache.md)

## ToC

- [Installation](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#installation)
- [Basic usage](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#usage)
- [Meteor.js usage](https://github.com/veliovgroup/spiderable-middleware/blob/master/docs/meteor.md)
- [__API__](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#api)
  - [Constructor `new Spiderable()`](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#constructor)
  - [Middleware](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#handle)
  - [TS Types](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#types)
- [Return genuine status code](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#return-genuine-status-code)
- [Speed-up rendering](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/optimization.md)
- [Detect request from Prerendering engine during runtime](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#detect-request-from-pre-rendering-engine-during-runtime)
- [Detect type of Prerendering engine](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#detect-type-of-the-pre-rendering-engine)
- [JavaScript redirects](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#javascript-redirects)
- [AMP Support](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#amp-support)
- [Rendering Endpoints](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#rendering-endpoints)
- [Debugging](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#debugging)
- [Running Tests](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#running-tests)

## About Package

This package works as Express-style middleware, intercepting requests from crawlers and social media bots to your Node.js application. It seamlessly proxies those requests to a pre-rendering service, which returns fully-rendered static HTML — optimized for indexing and rich previews.

Built with developers in mind, `spiderable-middleware` is lightweight, well-structured, and easy to customize. Whether you're scaling a production app or prototyping, it's designed to be hackable and flexible enough to fit any project. By offloading bot traffic to the pre-rendering engine, it helps reduce backend load and improve server performance effortlessly.

> [!NOTE]
> *This package proxies real HTTP headers and response code, to reduce overwhelming requests, try to avoid HTTP-redirect headers, like* `Location` *. Read how to [return genuine status code](https://github.com/veliovgroup/spiderable-middleware#return-genuine-status-code) and [handle JS-redirects](https://github.com/veliovgroup/spiderable-middleware#javascript-redirects)*


> [!IMPORTANT]
> This is __server only package__. This package should be imported/initialized only within server codebase

This middleware was tested and works like a charm with:

- [express](https://www.npmjs.com/package/express): [`example.js`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/express.middleware.js), [`example.ts`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/express.middleware.ts)
- [connect](https://www.npmjs.com/package/connect): [`example.js`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/connect.middleware.js), [`example.ts`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/connect.middleware.ts)
- [vanilla http(s) server](https://nodejs.org/api/http.html): [`example.js`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/http.middleware.js), [`example.ts`](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/http.middleware.ts)
- [Nginx](https://nginx.org/en/docs/): [example](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/nginx)
- [Apache](https://httpd.apache.org/): [example](https://github.com/veliovgroup/spiderable-middleware/blob/master/examples/apache.htaccess)
- See [all examples](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples)

*All other frameworks that follows Node/Express middleware convention - will work too.*

> [!TIP]
> This package was originally developed for [ostr.io](https://ostr.io) service. But it's not limited to, and can proxy-pass requests to any other rendering-endpoint.

## Installation

Install [`spiderable-middleware` package from NPM](https://www.npmjs.com/package/spiderable-middleware):

```sh
# using npmjs
npm install spiderable-middleware --save

# using yarn
yarn add spiderable-middleware
```

## Usage

Setup pre-rendering middleware in few lines of code

### Basic usage

Start with adding `fragment` meta-tag to HTML template, head, or page:

> [!TIP]
> See [usage examples in `.js` and `.ts`](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples) for quick copy-paste experience

```html
<html>
  <head>
    <meta name="fragment" content="!">
    <!-- ... -->
  </head>
  <body>
    <!-- ... -->
  </body>
</html>
```

Import or require `spiderable-middleware` package:

```js
// ES6 import
import Spiderable from 'spiderable-middleware';

// or CommonJS require
const Spiderable = require('spiderable-middleware');
```

Register middleware handle:

```js
import express from 'express';
import Spiderable from 'spiderable-middleware';

const spiderable = new Spiderable({
  rootURL: 'http://example.com',
  auth: 'test:test',
});

const app = express();
// ensure this is the most top registered handle
// to reduce response time and server load
app.use(spiderable).get('/', (req, res) => {
  res.send('Hello World');
});

app.listen(3000);
```

> [!NOTE]
> `Spiderable` instance is a middleware function, pass it to `.use()` directly. `spiderable.handle` and `spiderable.handler` are kept as aliases.

> [!TIP]
> We provide various options for `serviceURL` as "[Rendering Endpoints](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/rendering-endpoints.md)", each endpoint has its own features to fit different project needs

## Return genuine status code

To pass expected status code of a response from front-end JavaScript framework to browser/crawlers use specially formatted HTML-comment. This comment __can be placed in any part of HTML-page__. `head` or `body` tag is the best place for it.

### Format

__html__:

```html
<!-- response:status-code=404 -->
```

__jade__:

```jade
// response:status-code=404
```

This package support __any__ standard and custom status codes:

- `201` - `<!-- response:status-code=201 -->`
- `401` - `<!-- response:status-code=401 -->`
- `403` - `<!-- response:status-code=403 -->`
- `500` - `<!-- response:status-code=500 -->`
- `514` - `<!-- response:status-code=514 -->` (*non-standard*)

## Speed-up rendering

To speed-up rendering, JS-runtime __should__ tell to the Spiderable engine when the page is ready. Set `window.IS_RENDERED` to `false`, and once the page is ready set this variable to `true`. Example:

```html
<html>
  <head>
    <meta name="fragment" content="!">
    <script>
      window.IS_RENDERED = false;
    </script>
  </head>
  <body>
    <!-- ... -->
    <script type="text/javascript">
      //Somewhere deep in app-code:
      window.IS_RENDERED = true;
    </script>
  </body>
</html>
```

## Detect request from Pre-rendering engine during runtime

Pre-rendering engine will set `window.IS_PRERENDERING` global variable to `true`. Detecting requests from pre-rendering engine are as easy as:

```js
if (window.IS_PRERENDERING) {
  // This request is coming from Pre-rendering engine
}
```

> [!NOTE]
> `window.IS_PRERENDERING` can be `undefined` on initial page load, and may change during runtime. That's why we recommend to pre-define a setter for `IS_PRERENDERING`:

```js
let isPrerendering = false;
Object.defineProperty(window, 'IS_PRERENDERING', {
  set(val) {
    isPrerendering = val;
    if (isPrerendering === true) {
      // This request is coming from Pre-rendering engine
    }
  },
  get() {
    return isPrerendering;
  }
});
```

## Detect type of the Pre-rendering engine

Like browsers, — crawlers and bots may request page as "mobile" (small screen touch-devices) or as "desktop" (large screens without touch-events) the pre-rendering engine supports these two types. For cases when content needs to get optimized for different screens pre-rendering engine will set `window.IS_PRERENDERING_TYPE` global variable to `desktop` or `mobile`

```js
if (window.IS_PRERENDERING_TYPE === 'mobile') {
  // This request is coming from "mobile" web crawler and "mobile" pre-rendering engine
} else if (window.IS_PRERENDERING_TYPE === 'desktop') {
  // This request is coming from "desktop" web crawler and "desktop" pre-rendering engine
} else {
  // This request is coming from user
}
```

## JavaScript redirects

Redirect browser/crawler inside application when needed while a page is loading (*imitate navigation*), use any of classic JS-redirects can be used, including framework's navigation, or `History.pushState()`

```js
window.location.href = 'http://example.com/another/page';
window.location.replace('http://example.com/another/page');

Router.go('/another/page'); // framework's navigation !pseudo code
```

> [!IMPORTANT]
> *Only 4 redirects are allowed during one request after 4 redirects session will be terminated.*

## API

Create new instance and pass middleware to server's routes chain;

### *Constructor*

```ts
new Spiderable(opts?: SpiderableOptions);
```

- `opts` {*SpiderableOptions?*} - [Optional] Configuration options
- `opts.serviceURL` {*string*} - Valid `http://` or `https://` URL to Spiderable endpoint (local or foreign). Default: `https://render.ostr.io`. Can be set via environment variables: `SPIDERABLE_SERVICE_URL` or `PRERENDER_SERVICE_URL`
- `opts.rootURL` {*string*} - Valid root URL of a website. Can be set via an environment variable: `ROOT_URL`
- `opts.auth` {*string*} - Auth string in next format: `user:pass`. Can be set via an environment variables: `SPIDERABLE_SERVICE_AUTH`, `PRERENDER_SERVICE_AUTH`, or `OSTR_AUTH`. Logs a warning at startup when empty
- `opts.sanitizeUrls` {*boolean*} - Sanitize URLs in order to "fix" badly composed URLs. Default `false`
- `opts.botsUA` {*string[]*} - An array of strings (case insensitive) with additional User-Agent names of crawlers that needs to get intercepted. See default [bot's names](https://github.com/veliovgroup/spiderable-middleware/blob/master/lib/index.js#L198). Set to `['.*']` to match all browsers and robots, to serve static pages to all users/visitors
- `opts.ignoredHeaders` {*string[]*} - An array of strings (case insensitive) with HTTP header names to exclude from response. See default [list of ignored headers](https://github.com/veliovgroup/spiderable-middleware/blob/master/lib/index.js#L206). Set to `['.*']` to ignore all headers
- `opts.ignore` {*string[]*} - An array of strings (case __sensitive__) with ignored routes. Note: it's based on first match, so route `/users` will cause ignoring of `/part/users/part`, `/users/_id` and `/list/of/users`, but not `/user/_id` or `/list/of/blocked-users`. Default `null`
- `opts.only` {*(String|RegExp)[]*} - An array of strings (case __sensitive__) or regular expressions (*could be mixed*). Define __exclusive__ route rules for pre-rendering. Could be used with `opts.onlyRE` rules. __Note:__ To define "safe" rules as {*RegExp*} it should start with `^` and end with `$` symbols, examples: `[/^\/articles\/?$/, /^\/article\/[A-z0-9]{16}\/?$/]`
- `opts.onlyRE` {*RegExp*} - Regular Expression with __exclusive__ route rules for pre-rendering. Could be used with `opts.only` rules
- `opts.timeout` {*number*} - Number, proxy-request timeout to rendering endpoint in milliseconds. Default: `180000`
- `opts.retries` {*number*} - Extra attempts when the connection to rendering endpoint fails before it answered, for example when it closed a keep-alive socket. Never retries after the response started. Default: `1`
- `opts.requestOptions` {*RequestOptions*} - Options for request module (like: `timeout`, `lookup`, `insecureHTTPParser`), for all available options see [`http` API docs](https://nodejs.org/docs/latest-v14.x/api/http.html#http_http_request_url_options_callback). Pass `agent` to replace the default keep-alive agent, or `agent: false` to disable connection reuse
- `opts.staticExt` {*RegExp*} - Regular Expression matching paths of static files, these requests are never pre-rendered. See default [list of extensions](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/shared/static-extensions-regex.md)
- `opts.debug` {*boolean*} - [Optional] Enable debug and extra logging, default: `false`
- `opts.logger` {*Logger*} - [Optional] Object with `info` and `warn` methods, receives warnings and debug messages. Default: `console`

> [!IMPORTANT]
> *Setting* `.onlyRE` *and/or* `.only` *rules are highly recommended. Otherwise, all routes, including randomly generated by bots will be subject of Pre-rendering and may cause unexpectedly higher usage.*

> [!NOTE]
> Requests for static files (`opts.staticExt`), paths under `/.well-known/`, and requests to `ostr.io` hosts always pass to the app without pre-rendering.

When the rendering endpoint fails, crawlers never get a broken page cached:

- Connection error before the endpoint answered: the request is retried `opts.retries` times, then passed to the app with `next()`.
- No answer within `opts.timeout`: the request is passed to the app with `next()`.
- Failure after the response started, a timeout or a dropped connection mid-body: the connection to the crawler is destroyed instead of ending a partial page. Crawlers retry such requests later.

```js
// CommonJS
// const Spiderable = require('spiderable-middleware');

// ES6 import
// import Spiderable from 'spiderable-middleware';

const spiderable = new Spiderable({
  rootURL: 'http://example.com',
  auth: 'test:test'
});

// More complex setup (recommended):
const spiderable = new Spiderable({
  rootURL: 'http://example.com',
  serviceURL: 'https://render.ostr.io',
  auth: 'test:test',
  only: [
    /\/?/, // Root of the website
    /^\/posts\/?$/, // "/posts" path with(out) trailing slash
    /^\/post\/[A-z0-9]{16}\/?$/ // "/post/:id" path with(out) trailing slash
  ],
  // [Optionally] force ignore for secret paths:
  ignore: [
    '/account/', // Ignore all routes under "/account/*" path
    '/billing/' // Ignore all routes under "/billing/*" path
  ]
});
```

#### Configuration via env.vars

Same configuration can get achieved via setting up environment variables:

```sh
ROOT_URL='http://example.com'
SPIDERABLE_SERVICE_URL='https://render.ostr.io'
SPIDERABLE_SERVICE_AUTH='APIUser:APIPass'
```

alternatively, when migrating from other pre-rendering service — keep using existing variables, we support the next ones for compatibility:

```sh
ROOT_URL='http://example.com'
PRERENDER_SERVICE_URL='https://render.ostr.io'
PRERENDER_SERVICE_AUTH='APIUser:APIPass'
```

### handle

Middleware handle. The instance itself is callable with the same arguments

```ts
const spiderable = new Spiderable();
spiderable(req: IncomingMessage, res: ServerResponse, next: NextFunction): void;
spiderable.handle(req: IncomingMessage, res: ServerResponse, next: NextFunction): void;

// Alias that returns {boolean} 
// true — if prerendering takes over the request
spiderable.handler(req: IncomingMessage, res: ServerResponse, next: NextFunction): boolean;
```

Example using `connect` and `express` package:

```js
import { createServer } from 'node:http';
import Spiderable from 'spiderable-middleware';

const app = express();
// const app = connect();
const spiderable = new Spiderable();

app.use(spiderable).use((_req, res) => {
  res.end('Hello from Connect!\n');
});

createServer(app).listen(3000);
```

Example using node.js `http` server:

```js
import { createServer } from 'node:http';
import Spiderable from 'spiderable-middleware';

// HTTP(s) Server
http.createServer((req, res) => {
  spiderable.handle(req, res, () => {
    // Callback, triggered if this request
    // is not a subject of spiderable pre-rendering
    res.writeHead(200, {'Content-Type': 'text/plain; charset=UTF-8'});
    res.end('Hello vanilla NodeJS!');
    // Or do something else ...
  });
}).listen(3000);
```

### Types

Import types right from NPM package

```ts
import Spiderable from 'spiderable-middleware';
import type { SpiderableOptions, NextFunction } from 'spiderable-middleware';

const options: SpiderableOptions = {
  rootURL: 'http://example.com',
  auth: 'test:test',
  debug: false,
  /* ..and other options.. */
};
expectType<SpiderableOptions>(options);

const spiderable = new Spiderable(options);
expectType<Spiderable>(spiderable);

const next: NextFunction = (_err?: unknown): void => {};
expectType<void>(spiderable.handle(req, res, next));
```

## AMP Support

To properly serve pages for [Accelerated Mobile Pages](https://www.ampproject.org) (AMP) we support following URI schemes:

```sh
# Regular URIs:
https://example.com/index.html
https://example.com/articles/article-title.html
https://example.com/articles/article-uniq-id/article-slug

# AMP optimized URIs (prefix):
https://example.com/amp/index.html
https://example.com/amp/articles/article-title.html
https://example.com/amp/articles/article-uniq-id/article-slug

# AMP optimized URIs (extension):
https://example.com/amp/index.amp.html
https://example.com/amp/articles/article-title.amp.html
```

> [!IMPORTANT]
> All URLs with `.amp.` extension and `/amp/` prefix will be optimized for AMP.

## Rendering Endpoints

- __render (*default*)__ - `https://render.ostr.io` - This endpoint has "optimal" settings, and should fit 98% cases. This endpoint respects cache headers of Crawler and origin server
- __render-bypass (*devel/debug*)__ - `https://render-bypass.ostr.io` - This endpoint will bypass caching mechanisms. Use it when experiencing an issue, or during development, to make sure responses are not cached. It's safe to use this endpoint in production, but it may result in higher usage and response time
- __render-cache (*under attack*)__ - `https://render-cache.ostr.io` - This endpoint has the most aggressive caching mechanism. Use it to achieve the shortest response time, and when outdated pages (*for 6-12 hours*) are acceptable

To change default endpoint, grab [integration examples code](https://github.com/veliovgroup/spiderable-middleware/tree/master/examples) and replace `render.ostr.io`, with endpoint from the list above. For NPM integration change value of [`serviceURL`](https://github.com/veliovgroup/spiderable-middleware?tab=readme-ov-file#constructor) option.

__Note:__ Described differences in caching behavior related to intermediate proxy caching, `Cache-Control` header will be always set to the value defined in "Cache TTL". Cached results at the "Pre-rendering Engine" end can be [purged at any time](https://github.com/ostr-io/ostrio-docs/blob/master/docs/prerendering/cache-purge.md).

## Convert dynamic website to static

`spiderable-middleware` package can get used to convert dynamic websites to rendered, cached, and lightweight static pages. Simply set `botsUA` to `['.*']` to achieve this behavior

```js
import Spiderable from 'spiderable-middleware';

const spiderable = new Spiderable({
  botsUA: ['.*']
  /* ... other options ...*/
});
```

## Debugging

Pass `{ debug: true }` or set `DEBUG=true` environment variable to enable debugging mode.

> [!TIP]
> To make sure a server can reach a rendering endpoint run `cURL` command or send request via Node.js to (*replace example.com with your domain name*):

```shell
# cURL example:
curl -v "https://test:test@render-bypass.ostr.io/?url=http://example.com"
```

In this example we're using `render-bypass.ostr.io` endpoint to avoid any possible cached results, [read more about rendering endpoints](https://github.com/veliovgroup/spiderable-middleware#rendering-endpoints). As API credentials we're using `test:test`, this part of URL can be replaced with `auth` option from Node.js example.

> [!TIP]
> The API credentials and instructions can be found at the very bottom of [Pre-rendering Panel](https://ostr.io/service/prerender), — click on the name of your website, then on <kbd>Integration Guide</kbd> at the bottom of the page

```js
// Node.js example:
const https = require('https');

https.get('https://test:test@render-bypass.ostr.io/?url=http://example.com', (resp) => {
  let data = '';

  resp.on('data', (chunk) => {
    data += chunk.toString('utf8');
  });

  resp.on('end', () => {
    console.log(data);
  });
}).on('error', (error) => {
  console.error(error);
});
```

## Running Tests

1. Clone this package
2. In Terminal (*Console*) go to directory where package was cloned
3. Then run:

### Node.js/Mocha

Unit tests run offline against a local stub of the rendering service. Integration tests call the public ostr.io test endpoint and need network access. Both need Node.js@>=20.19, the package itself runs on Node.js@>=10.9.

```sh
# Install development dependencies
npm ci

# Unit tests
npm test

# Unit tests with coverage report, fails below 95%
npm run test:coverage

# Integration tests
# ROOT_URL port and ports +1, +5, 3010, 3011, 3015 must be free
ROOT_URL=http://127.0.0.1:3003 npm run test:integration

# Same tests with extra logging
DEBUG=true ROOT_URL=http://127.0.0.1:3003 npm run test:integration

# TypeScript declarations
npm run build
npm run test:types
```

Dependency-free smoke test, runs on every Node.js release down to 10.9:

```sh
npm run test:compat
```

GitHub Actions runs all of the above on every push, plus unit tests on Node.js 12, 14, 16, and 18, and the smoke test on Node.js 10.9 to 26.

---
_Source: https://npm.io/package/spiderable-middleware · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._
