# node-backup-loader

> Utility for copying public data from public databases (mostly, packages, from npm, bower, etc). Multiple threads, easy to use.

Latest version **1.4.4** (published 2019-09-17) · MIT license · 0 weekly downloads

## Install

```sh
npm install node-backup-loader
pnpm add node-backup-loader
yarn add node-backup-loader
bun add node-backup-loader
```

Provides the command `node-backup`.

## Health

**Score 15/100 (F)** — status: abandoned.

Positive: no vulnerabilities.

Warnings: low downloads; no types; no esm support.

Negative: abandoned; low maintenance score.

## Facts

| | |
|---|---|
| Version | 1.4.4 |
| Published | 2019-09-17 |
| First published | 2019-09-15 |
| Weekly downloads | 0 |
| License | MIT |
| TypeScript types | none |
| Module format | CommonJS |
| Dependencies | 9 |
| Unpacked size | 64.5 KB |
| Known vulnerabilities | 0 (+6 in 1 direct dependencies) |
| Install scripts | no |
| Author | Roman Yakimchuk |
| Maintainers | yakimchuk |
| Keywords | backup, npm, modules, download, parser, tool, cli, copy |

## Links

- npm: https://www.npmjs.com/package/node-backup-loader
- Repository: https://github.com/yakimchuk/node-backup
- Homepage: https://github.com/yakimchuk/node-backup#readme
- Issues: https://github.com/yakimchuk/node-backup/issues
- npm.io page: https://npm.io/package/node-backup-loader

## Dependencies (9)

- [lodash](https://npm.io/package/lodash.md) 4.17.15
- [ts-node](https://npm.io/package/ts-node.md) 8.3.0
- [optimist](https://npm.io/package/optimist.md) 0.6.1
- [typescript](https://npm.io/package/typescript.md) 3.6.2
- [@types/node](https://npm.io/package/@types/node.md) 12.7.4
- [glob-promise](https://npm.io/package/glob-promise.md) 3.4.0
- [node-libcurl](https://npm.io/package/node-libcurl.md) 2.0.1
- [all-the-package-names](https://npm.io/package/all-the-package-names.md) latest
- [child-process-promise](https://npm.io/package/child-process-promise.md) 2.2.1

## Alternatives

- [babylon](https://npm.io/package/babylon.md) — 5.1M weekly downloads
- [csscolorparser](https://npm.io/package/csscolorparser.md) — 3.7M weekly downloads
- [expr-eval-fork](https://npm.io/package/expr-eval-fork.md) — 1.5M weekly downloads
- [@leeoniya/ufuzzy](https://npm.io/package/@leeoniya/ufuzzy.md) — 247.7K weekly downloads
- [xml-parser](https://npm.io/package/xml-parser.md) — 78.4K weekly downloads

## Recent versions

- 1.4.4 (latest) — 2019-09-17
- 1.4.3 — 2019-09-17
- 1.4.2 — 2019-09-16
- 1.4.1 — 2019-09-16
- 1.4.0 — 2019-09-16
- 1.3.5 — 2019-09-16
- 1.3.4 — 2019-09-16
- 1.3.3 — 2019-09-16
- 1.3.2 — 2019-09-16
- 1.3.1 — 2019-09-15

## README

# Node Backup (NPM, etc)
CLI for downloading public data (for now it is NPM packages meta or archives). Multithread.

## Installation
`npm install node-backup-loader -g`

## Usage

There is an important note. NPM has the only way to retrieve their packages, it is public 
search, therefore we must do about 1-2k requests to obtain information about all their packages, BUT,
I found out it has much less packages in search, than actually can be found and installed
directly (using `https://www.npmjs.com/package/<package_name>`)!

I found the repository called [all-the-package-names](https://github.com/nice-registry/all-the-package-names) 
which updated NPM packages list every day, and it has all of them (even deleted or deprecated).

So, there are two algorythms implemented to download NPM packages:

1. Fast and reliable, **not all packages**, search-way. Just get meta from NPM database using search, and 
download them. Bulk downloading (200 packages for each thread).

2. Slow, but you will receive **all packages**, package-api-way. You will request each packages one by 
one (they are not available on search API), so you will send about 1mm request, and it will produce huge
load on NPM servers, your network connection, and, possibly, you hard drive (if you want to download archives).

### Download packages meta
Meta data will be downloaded as JSON, one file for each package. At first, I implemented the way to download one 
huge JSON, but it become too large (about 5-10 Gb), so it was not possible to debug it or even open it. So the 
only way to download the data is to split it by files. Now all downloaded meta will be stored in the directory 
you specify. Total count of files is about 1mm, so if you are using `fat32` table partition, create an issue 
then, you just will not be able to store all files in the same folder, restriction of `fat32` systems is 
about 65k files per folder.

Commands:
* `node-backup --command meta --registry npm --threads <threads_number> <directory>` (fast-download, not all packages)
* `node-backup --command meta --registry npm --npm-force-package-api --threads <threads_number> <directory>` (slow download, all packages)

Examples:
* `node-backup --command meta --registry npm --threads 25 ../my-folder/to/store/packages/meta/files/`
* `node-backup --command meta --registry npm --npm-force-package-api --threads 50 ../my-folder/to/store/packages/meta/files/`

### Download packages archives
Useful when you want to download all packages from NPM.

Commands:
* `node-backup --command copy --registry npm [--preserve-downloaded-files] --threads <threads_number> <directory>` (fast-download, not all packages)
* `node-backup --command copy --registry npm [--preserve-downloaded-files] --npm-force-package-api --threads <threads_number> <directory>` (slow download, all packages)

Examples:
* `node-backup --command copy --registry npm --threads 1 ../my-folder/to/store/packages/archives/`
* `node-backup --command copy --registry npm --npm-force-package-api --threads 10 ../my-folder/to/store/packages/archives/`
* `node-backup --command copy --registry npm --preserve-downloaded-files --threads 1 ../my-folder/to/store/packages/archives/`
* `node-backup --command copy --registry npm --preserve-downloaded-files --npm-force-package-api --threads 10 ../my-folder/to/store/packages/archives/`

I would not recommend you to set huge threads number (like, 100-200). For my case at any threads number more than 10-20, I could 
not get the downloading speed more than 5-15 packages per second due to `npm pack` command delay and my network connection speed.

The option `--preserve-downloaded-files` will force tool to not override already downloaded files in case if you 
accidentally closed your terminal. This options use the module `glob`, but as is there is too much files, it may cause 
performance issues, so it disabled by default.

## Screenshots

How archives will be stored:

![](doc/packages-list.png)

Downloading process:

![](doc/downloading-process.png)

## Contribution
Look at the file `src/modules/registry.ts` and see what is registry. Every loaded data is abstracted from algorithm how it 
used, so you can create any new registry (like, `bower`, or any other database, from news blog to `archive.org`) and add new 
commands in this doc.

**Be free to send pull requests.**

---
_Source: https://npm.io/package/node-backup-loader · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._