@mjyocca/async-parallel-limit v1.0.2

async-parallel-limit

Limits the amount of asynchronous operations running in parallel at a given time with a provided callback interface.

Inspired by async's asyncTimesLimit.

Instead of queueing up an array of asynchronous tasks to run in parallel such as await Promise.all([...]);,

&&

Instead of queueing those promises in a form of a batch and waiting for all promises in the batch to resolve before processing the next batch, async-parallel-limit, will queue up the next promise once one in the existing processing queue resolves.

Visualized

Example => 12 tasks, limit of 3 in parallel

Install via npm

npm i --save @mjyocca/async-parallel-limit

import asyncParallel from '@mjyocca/async-parallel-limit';

// Process 20 total async tasks with a limit of 5 at time
await asyncParallel(20, 5, async (n, next) => {
    // processing
    next();
})

Module Support

ESM

import asyncTimesLimit from '@mjyocca/async-parallel-limit';

CommonJs

const asyncTimesLimit = require('@mjyocca/async-parallel-limit');

Example 1

import asyncTimesLimit from '@mjyocca/async-parallel-limit'

(async () => {

    // array of objects
    const dataToProcess = getData();
    // will process the entire collection limiting only to running 10 in parallel
    // once each promise resolves, the next promise is added to the queue
    const apiProcessedIds = await asyncTimesLimit(
        dataToProcess, 
        10, 
        // async callback function to process the data
        async (n, data, next) => {
            const apiRes = await getFromAPI(data);
            // call next to resolve async task
            // and can pass data to collect once all promises resolve
            next(apiRes.id);
        }
    )
    // Array of Id's [,,,,]
    console.log(apiProcessedIds)

})();

Example 2

import asyncTimesLimit from '@mjyocca/async-parallel-limit'

(async () => {

    // Will process 50 times with a max of 10 concurrently/parallel
    const apiProcessedIds = await asyncTimesLimit(50, 10, 
        async (n, next) => {
            const apiRes = await getFromAPI(data);
            // call next to resolve async task
            // and can pass data to collect once all promises resolve
            next(apiRes.id);
        }
    )
    // Array of Id's [,,,,]
    console.log(apiProcessedIds)

})();