lvr_bulk_downloader v0.5.8

LVR Bulk Downloader

Easy downloading...

Tested with 17000 files in input, about 17GB downloaded without an error.

Features

• Single file or bulk download;

• Retry download if failed;

• Output directory flat or tree directories;

• Sync or Async download;

• Output response for all files downloaded;

Installation

   npm i lvr_bulk_downloader --save

Import

   import { Downloader, DownloaderTypes } from 'lvr_bulk_downloader';

Simple bulk downalod use

    const files =  [ 
        new DownloaderTypes.InputFile('http://host.domain/dir/file.ext'), 
        new DownloaderTypes.InputFile('http://host.domain/dir/file2.ext')
        ];
    const downloader = new Downloader();
    downloader.run(files);

Complex bulk download use

    const files =  [ 
        new DownloaderTypes.InputFile('http://host.domain/dir/file.ext', './download/file.ext'), 
        new DownloaderTypes.InputFile('http://host.domain/dir/file2.ext', './download/dir/file2.ext'), 
        new DownloaderTypes.InputFile('http://host.domain/dir/file3.ext', './download/dir/file3.ext')
        ];
    
    const options = new DownloaderTypes.Options();
    options.output_directory = './download/';
    options.debug_mode = DownloaderTypes.DebugMode.DEBUG;
    options.retry_times = 3;

    const downloader = new Downloader(options);
    downloader.run(files, function(output: DownloaderTypes.OutputFile[]){
        const downloadedCount = output.filter(x=>x.response?.status == DownloaderTypes.Status.OK).length;
        const errorCount = output.filter(x=>x.response?.status == DownloaderTypes.Status.KO).length;
    });

For sync download adding await to run function

    const files =  [ 
        new DownloaderTypes.InputFile('http://host.domain/dir/file.ext'), 
        new DownloaderTypes.InputFile('http://host.domain/dir/file2.ext')
        ];
    const downloader = new Downloader();
    await downloader.run(files);

Single file download use

    var file = new DownloaderTypes.InputFile('http://host.domain/dir/file.ext');
    var downloader = new Downloader();
    downloader.run(file);

Types definition

Types are contained in module DownloaderTypes

InputFile

url: string

output_path: string | null //This attribute overwrite output_directory defined on options only for this file.

OutputFile

url: string

path: string | null

retry_times: number

response: Response | null

Options

retry_times: number = 3

debug_mode: DebugMode = DebugMode.LOG

output_directory: string = './download/'

Response

status: Status;

message: any;

enum Status{
    OK,
    KO
}

enum DebugMode{
    NONE,
    DEBUG,
    LOG
}