Deduplication Packages

util for yarn.lock

command line lead deduplication in js

Fast multi-threaded content-dependent chunking deduplication for Buffers in C++ with a reference implementation in Javascript. Ships with extensive tests, a fuzz test and a benchmark.

A standalone in-memory tracker for deduplication, deferral, and change detection of events.

A package that provides helper methods for common and not-so-common use cases

Checks the similarity of 2 marc records using neural network.

command line lead deduplication in js

Merges two arrays into one and removes duplicates

Deduplicates items in an array

Remove consecutive duplicated values.

Copy elements to a new generic array after removing consecutive duplicated values.

Create an iterator which removes consecutive duplicated values.

Create an iterator which removes consecutive values that resolve to the same value according to a provided function.

A powerful file deduplication tool that quickly identifies and manages duplicate files. Features both CLI and web interfaces, detailed visual reporting, automatic cleanup options, and smart space-saving statistics. Reclaim your storage space with ease.

Find and manage duplicate files

A file management system for node.js application making managements of files easier, all in one code, you can delete, move, etc, files all in one code, making ability to manage a system eg a application requiring transfering of files in folders easier

Astrohelm file system watcher

A TypeScript library for enhancing array manipulation with asynchronous operations such as mapping, filtering, and deduplication.

Variable sized rolling chunks. Good for deduplication.

Automatic matching of Blue Button JSON data (detection of new, duplicate and partial match entries).