2.0.0 • Published 4 years ago
@kingotten/remove-duplicates v2.0.0
rmdupes
Rmdupes is a command used to remove duplicate files from folders The way it works is, we hash the contents of a file, and compare the hashes with eachother, to check if the files contain the same content If they have the same content, we delete the file.
You may want to use this command recursively as well, checking all the subfolders for duplicate files with other folders.
Be careful running this on extremely large data sets, you might overload your memory. (saving 50k hashes to memory isnt always a good idea)
Install
npm i @kingotten/remove-duplicates
Example code
const { RemoveDuplicates } = require("@kingotten/remove-duplicates");
const settings = {
dry_run: false, // run without deleting files
recursive: true, // run in subfolders (does not compare to subfolders tho)
depth: 2, // check 2 folders deep
quiet: true, // run without logging information about the command
filter: "", // regex filter performed on each filename
};
// use with single string
const path = "path/to/folder";
await RemoveDuplicates(path, settings);
// use with array
const paths = ["PathA", "PathB", "PathC"];
await RemoveDuplicates(paths, settings);
Global install
npm i -g @kingotten/remove-duplicates
Global Use
rmdupes -h
rmdupes .
rmdupes FolderA
rmdupes FolderA FolderB
rmdupes FolderA -d -r -D=2
rmdupes E:/Absolute/Path/To/Folder