@hoplin/puppeteer-pool NPM

Puppeteer Pool Manager

Puppeteer-Pool is a lightweight and efficient library for managing multiple Puppeteer contexts with ease. Designed for scalable web scraping and automation, it helps you handle concurrency and resource management without the hassle.

Package Installation

npm
```
npm i puppeteer @hoplin/puppeteer-pool
```

yarn

yarn add puppeteer @hoplin/puppeteer-pool

pnpm

pnpm install puppeteer @hoplin/puppeteer-pool

Release 2.1.0

Garbage collection is executed at the end of each context task
Client's method are now all 'static'. You don't need to create instance of client and need migration if you are using previous version(< 2.1.0).
- getPoolMetrics
- runTask
- stop
- checkInstanceInitalized

Add Priority Queue for task queue

taskQueueType option is added to PuppeteerPoolStartOptions

Priorty can be specified in the second factor of 'PuppeteerPool.runTask'. The larger the number, the higher the priority, and the priority is ignored in the Default Queue.
Enum: QueueMode

Example

import {
  ContextMode,
  PuppeteerPool,
  QueueMode,
} from '@hoplin/puppeteer-pool';

const priority = 10;

await PuppeteerPool.start({
  taskQueueType: QueueMode.PRIORITY,
});

PuppeteerPool.runTask(async (page) => {
  await page.goto(url);
  const title = await page.title();
  return title;
}, priority);

Client API

PuppeteePool
- PuppeteerPool is singleton class. You can use PuppeteerPool.start to initialize pool manager.

PuppeteerPool.start

Static Method
Description: Initialize pool manager. You need to call this function to start puppeteer pool. Even if you invoke this function multiple times with differenct arguments, it will return the first initialized instance.

Args: PuppeteerPoolStartOptions

type PuppeteerPoolStartOptions = {
  /**
   * Number of concurrency,
   * Default is 3
   */
  concurrencyLevel: number;
  /**
   * Context mode
   * Default is ContextMode.SHARED
   */
  contextMode: ContextMode;
  /**
   * Puppeteer launch options
   * Default is {}
   */
  options?: puppeteer.LaunchOptions;
  /**
   * Custom config path
   */
  customConfigPath?: string;
  /**
   * Enable log
   * Default is true
   */
  enableLog?: boolean;
  /**
   * Log level
   * Default is LogLevel.DEBUG
   */
  logLevel?: LogLevel;
};

Return
- Promise<PuppeteerPool>
- Returns PuppeteerPool Instance.

PuppeteerPool.stop
- Static Method
- Description: Stop pool manager. It will close all sessions and terminate pool manager.
- Return
  - Promise<void>
PuppeteerPool.runTask
- Static Method
- Description: Run task in pool manager. It will return result of task.
- Args
- task
  - Required
  - Function
- Return
- Promise<any>
- Returns result of task(Same return type with task callback return type)

PuppeteerPool.getPoolMetrics

Static Method
Description: Get pool metrics. It will return metrics of pool manager.

Return

    {
        memoryUsageValue: (Memory Usage in MegaBytes),
        memoryUsagePercentage: (Memory Usage with percentage),
        cpuUsage: (CPU Usage with percentage)
    }

Simple Demo

import { ContextMode, PuppeteerPool, QueueMode } from '@hoplin/puppeteer-pool';

async function main() {
  await PuppeteerPool.start({
    concurrencyLevel: 6,
    contextMode: ContextMode.ISOLATED,
    customConfigPath: `./puppeteer-pool-config.json`,
    taskQueueType: QueueMode.PRIORITY,
  });

  const baseUrls = [
    'https://www.google.com',
    'https://www.bing.com',
    'https://github.com',
    'https://www.naver.com',
    'https://www.daum.net',
    'https://www.youtube.com',
    'https://www.amazon.com',
    'https://www.netflix.com',
  ];

  const urls = Array.from({ length: 50 }, (_, index) => {
    const baseUrl = baseUrls[index % baseUrls.length];
    const priority = Math.floor(Math.random() * 10) + 1;
    return { url: baseUrl, priority: priority };
  });

  const promises = urls.map(({ url, priority }) => {
    console.log(`Enqueue task: ${url}`);
    return PuppeteerPool.runTask(async (page) => {
      console.log(`Process task for ${url} with priority ${priority}`);
      await page.goto(url);
      const title = await page.title();
      return title;
    }, priority);
  });

  const titles = await Promise.all(promises);
  console.log('[ Result length ] :', titles.length);
  console.log('[ Expected Tasks ] :', urls.length);
  console.log('[ Metrics ] :', await PuppeteerPool.getPoolMetrics());
}

main();

Support

Pool Managing
Config
- Support config customize
Threshold Watcher
- Memory
- Support safe pool instance reset in runtime
Metrics
- Support Metric by pool
  - CPU usage of pool
  - Memory usage of pool
  - Managing session count in runtime

Puppeteer Pool Manager Config

Default config should be puppeteer-pool-config.json in root directory path.

Default config setting

If config file are not given or invalid path, manager will use default defined configurations. Or if you want to pass config path, you can pass path to start() function as parameter.

{
  "session_pool": {
    "width": 1080,
    "height": 1024
  },
  "context": {
    "timeout": 10
  },
  "threshold": {
    "activate": true,
    "interval": 5,
    "memory": 2048
  }
}

`session_pool`

width: Width of session pool
height: Height of session pool
- Inteager Validation
  - width should be larger or equal than 50
  - height should be larger or equal than 50
  - width should be integer
  - height should be integer

`context`

timeout: Timeout of context.
- Inteager Validation
  - timeout should be at least 1
  - timeout should be integer

`threshold`

activate: Activate threshold watcher
interval: Interval of threshold watcher
memory: Memory threshold value
- Inteager Validation
  - interval should be at least 1
  - interval should be integer
  - memory should be at least 1
  - memory should be integer