# request-all-pages

> Requests all pages of paginated data and emits them into a stream or aggregates them into an array.

Latest version **0.3.1** (published 2013-09-03) · MIT license · 0 weekly downloads

## Install

```sh
npm install request-all-pages
pnpm add request-all-pages
yarn add request-all-pages
bun add request-all-pages
```

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support; pre 1.0.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 0.3.1 |
| Published | 2013-09-03 |
| First published | 2013-06-07 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 4 |
| Known vulnerabilities | 0 (+1 in 1 direct dependencies) |
| Install scripts | no |
| GitHub stars | 7 |
| Author | Thorsten Lorenz |
| Maintainers | thlorenz |
| Keywords | pagination, page, pages, paging, request, api, follow |

## Links

- npm: https://www.npmjs.com/package/request-all-pages
- Repository: https://github.com/thlorenz/request-all-pages
- Issues: https://github.com/thlorenz/request-all-pages/issues
- npm.io page: https://npm.io/package/request-all-pages

## Dependencies (4)

- [through](https://npm.io/package/through.md) ~2.3.4
- [extend-url](https://npm.io/package/extend-url.md) ~0.1.0
- [hyperquest](https://npm.io/package/hyperquest.md) ~0.1.7
- [parse-link-header](https://npm.io/package/parse-link-header.md) ~0.1.0

## Alternatives

- [launchdarkly-js-client-sdk](https://npm.io/package/launchdarkly-js-client-sdk.md) — 2.5M weekly downloads
- [@elastic/elasticsearch](https://npm.io/package/@elastic/elasticsearch.md) — 2.1M weekly downloads
- [@c8y/client](https://npm.io/package/@c8y/client.md) — 15.3K weekly downloads
- [@signaldb/maverickjs](https://npm.io/package/@signaldb/maverickjs.md) — 1.7K weekly downloads
- [@bbc/http-transport-cache](https://npm.io/package/@bbc/http-transport-cache.md) — 1.2K weekly downloads

## Recent versions

- 0.3.1 (latest) — 2013-09-03
- 0.3.0 — 2013-09-02
- 0.2.3 — 2013-06-26
- 0.2.2 — 2013-06-26
- 0.2.1 — 2013-06-25
- 0.2.0 — 2013-06-24
- 0.1.4 — 2013-06-18
- 0.1.3 — 2013-06-08
- 0.1.2 — 2013-06-08
- 0.1.1 — 2013-06-07
- 0.1.0 — 2013-06-07

## README

# request-all-pages [![build status](https://secure.travis-ci.org/thlorenz/request-all-pages.png)](http://travis-ci.org/thlorenz/request-all-pages)

Requests all pages of paginated data and emits them into a stream or aggregates them into an array.

Follows the [link headers](http://tools.ietf.org/html/rfc5988) until it reaches the last page. As an example see [github
api pagination](http://developer.github.com/v3/#pagination)

```js
var requestAllPages = require('request-all-pages'); 

var requestOpts = {
    uri: 'https://api.github.com/users/substack/repos'
  , json: true
  , body: {}
  , headers: { 'user-agent': 'request-all-pages' } 
  };

requestAllPages(requestOpts, { startPage: 1, pagesPer: 100 }, function (err, pages) {
  if (err) return console.error(err);  
  var names = pages
    .reduce(
      function (acc, page) {
        acc = acc.concat(page.body.map(function (repo) { return repo.name; }))
        return acc;
      }
    , []);

  console.log('%s\nTotal: %s', names.join(', '), names.length);
});
```

```
airport, airport-cluster-example, amok-copter, astw, .... 
```

### Default opts

```js
// startPage defaults to 1 and pagesPer defaults to 50
requestAllPages(requestOpts, function (err, pages) {
  [..]
});
```

### Streaming Interface

```js
requestAllPages(requestOpts, { startPage: 1, pagesPer: 100 })
  .on('error', console.error) 
  .pipe(through(
    function (data) {
      var page = JSON.parse(data)
        , names = page.body.map(function (repo) { return repo.name; });
      this.queue(names.join(', '));
    }
  ))
  .pipe(process.stdout);
```

### Limit option

```js
// aborts immediately if last page > maxPages
requestAllPages(
      requestOpts
    , { pagesPer: 100, limit: { maxPages: 2, abort: true }  }
  )
  .pipe([...]);
```

```js
// gets only the first 2 pages even if there are more
requestAllPages(
      requestOpts
    , { pagesPer: 100, limit: { maxPages: 2, abort: false }  } 
  )
  .pipe([...]);
```

[Complete versions of these examples](https://github.com/thlorenz/request-all-pages/tree/master/examples).

## Installation

    npm install request-all-pages 

## API

***requestAllPages(requestOpts : Object[, opts: Object, callback : Function]) : Stream***

- **requestOpts**: options passed to [request](https://github.com/mikeal/request) after the `uri` was modified to
  include paging information. The same opts will be used for all paging requests.
- **opts**: optional configuration (see example below)
  - **startPage**: the page to start at (default: 1)
  - **perPage**: how many pages to ask for per request -- the smaller this number, the more requests have to be made to get
    all data (default: 50)

  - **limit**: object with following properties
      - **maxPages**: the maximum number of pages to fetch
      - **abort**: 
          - if `true` aborts immediately and returns [empty response](#empty-response) with `aborted: true` if last page exceeds `maxPages`
          - if `false` it fetches and returns data until `maxPages` is reached

- **callback**: `function (err, pages) {..}` if supplied, it will be called with an error or an array containing all
  pages each with the following structure: 

### response structure

```js
[ { headers      // response headers 
  , statusCode   // response statusCode 
  , body      }, // response body 
  { .. },
  .. ]
```

### empty response

```js
[ { statusCode: xxx
  , body: []
  , headers: { ... }
  , aborted: true } ]
```


If **no callback** is supplied, a `stream` is returned instead which emits `data` for each page and `error` if one
occurs.

---
_Source: https://npm.io/package/request-all-pages · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._