scarlet-task v2.1.0

Scarlet Task

A task queue module for node.js. You can set several children-queue for one task queue.

Why named Scarlet? ๛ก(ｰ̀ωｰ́ก)

At first, I wrote this module is for searching one song in 萌否收音機. And last I found that song named <the Embodiment of Scarlet Devil>.

For rembembering this and for my favorite Flandre Scarlet, I named this module Scarlet Task.

Usage

For one situation, once you want to crawl one website. If you use primitive node.js, it will like you're DDOSing that website.

So you need a task queue to help you. It will process tasks in queue one by one.

What's more, you can set that one queue has several children-queue to work concurrently.

And you can use it at any other situation that suitable.

Installation

$ npm install --save scarlet-task

If you're using it in browser, you may need to install events as well.

$ npm install --save events

Tutorials

Require the module at first and instantiate an object.

var { Scarlet } = require("scarlet-task");
var taskQueue = new TaskQueue(10);

The parameter for constructor means number of children-queue. Pass no parameter for default 1 children-queue.

Define a processor function for one task. In fact, you can pass an anonymous function.

function processor(taskObject) {
	// get task object
  var task = taskObject.task;
  
  // Do something...
  // blahblah...
  
  taskObject.done(); // You can call `taskQueue.taskDone(taskObject);` either

  console.log(taskQueue.numberOfProcessed());
};

Notice: In the processor function, you should call taskObject.done() or taskQueue.taskDone(taskObject) when you think this task is done. And then the taskQueue will process next task. The parameter taskObject is a parameter that taskQueue passed to you.

You can push task(s) at anytime.

The task object can be any type - string, number, json, etc.

var task = "it may be a url, or an object that process can do something with this task object.";
taskQueue.push(task, processor);

See more reference at test/touhou.js.

What's more, if you want to see the queue status for debuging, you can pass true to taskDone and push.

eg.

taskObject.done(true); // or `taskQueue.taskDone(taskObject, true);`
taskQueue.push(task, processor, true);

You can reset the number of processed tasks as well:

taskQueue.resetNumberOfProcessed();

And you can set an after-finish function so that Scarlet will call it after a certain number of tasks finished.

taskQueue.afterFinish(20, done, false);
// this will call done() after 20 tasks done without loop (means only once unless you reset number of processed).

taskQueue.clearAfterFinish();
// You can clear after finish processor

See more reference at examples/hackernews.js.

Migrate From v1.x to v2.x

scarlet-task v1.x exports Scarlet directly:

const Scarlet = require('scarlet-task');

While v2.x exports like this:

const { Scarlet } = require('scarlet-task');

Just replace the requirement.

Contribute

You're welcome to make pull requests!

「雖然我覺得不怎麼可能有人會關注我」