@iebh/dedupe-sweep NPM

IEBH/Dedupe-Sweep

Deduplicate reference libraries using the sweep method.

This library is intended to be used with Reflib compatible references.

// Simple example with an array of references
var Dedupe = require('@iebh/dedupe-sweep');

(new Dedupe())
	.set('strategy', 'doiOnly')
	.run([
		{doi: 'https://doi.org/10.1000/182'},
		{doi: '10.1000/182'},
	])
	.then(deduped => { /* ... */ })

// More complex example reading in a reference library with RefLib, deduping it and saving as another file
var Dedupe = require('@iebh/dedupe-sweep');
var reflib = require('reflib');

// Read in the library
var refs = await reflib.promises.parseFile('my-large-reference-library.xml');

// Dedupe
var deduper = new Dedupe()
deduper.set('strategy', 'clark')
var dedupedRefs = await deduper.run(refs);

// Save the deduped library
await reflib.promises.outputFile('my-large-reference-library-deduped.xml', dedupedRefs);

Testing

The various strategies within this project are tested using the Systematic Reviews Data Sets for Testing Automation Tools by Beller et. al and are available in the test/data directory.

Tests can be run via npm test or mocha. See the test directory for more information on specifics.

Testing statistics are based on the methodology from Evaluating automated deduplication tools: protocol by Hair et. al.

API

Constructor: Dedupe(options)

Returns a Dedupe class which extends a basic EventEmitter.

Dedupe.settings

Object storing all local settings for the class.

Setting	Type	Default	Description
stratergy	string	`'clark'`	The stratergy to use on the next `run()`
validateStratergy	boolean	`true`	Validate the strategy before beginning, only disable this if you are sure the strategy is valid
action	string	'`0`'	The action to take when detecting a duplicate. ENUM: ACTIONS
actionField	string	`'dedupe'`	The field to use with actions
threshold	number	`0.1`	Floating value (between 0 and 1) when marking or deleting refs automatically
markOk	string / function	`'OK'`	String value to set the action field to when `actionField=='mark'` and the ref is a non-dupe, if a function it is called as `(ref)`
markDupe	string / function	`'DUPE'`	String value to set the action field to when `actionField=='mark'` and the ref is a dupe, if a function it is called as `(ref)`
dupeRef	string	`0`	How to refer to other refs when `actionfield=='stats'`. ENUM: DUPEREF
fieldWeight	number	`0`	How to calculate duplication score. ENUM: FIELDWEIGHT
markOriginal	boolean	`false`	Whether to mark the original as a duplicate or not

Static: Dedupe.ACTIONS

Actions to take when detecting duplicates

Value	Setting	Description
`0`	`'STATS'`	Add the field field in `Dedupe.settings.actionField` with the deduplicate chance to the input
`1`	`'MARK'`	Set the field in `Dedupe.settings.actionField` to `Dedupe.settings.mark{Ok,Dupe}` depending on duplicate status but leave input unchanged
`2`	`'DELETE'`	Remove duplicates from input and return sliced output

Static: Dedupe.DUPEREF

How to refer to other references.

Value	Setting	Description
`0`	`'INDEX'`	Refer to other references by their offset in the input array
`1`	`'RECNUMBER'`	Refer to other references by their `recnumber` field

Static: Dedupe.FIELDWEIGHT

How to refer to other references.

Value	Setting	Description
`0`	`'MINIMUM'`	Calculate duplication score based on minumum field score
`1`	`'AVERAGE'`	Calculate duplication score based on average field score

Dedupe.comparisons

A lookup object of comparison functions used within strategies.

Each comparison is made up of:

Setting	Type	Description
key	string	Internal short name of the comparison in camelCase
`title`	string	Human friendly title of the comparison
`description`	string	Longer description of what the comparison does
`handler`	function	Function, called as `(a, b)` for fields which is expected to return a floating value of duplicate-ness

Dedupe.mutators

A lookup object of field mutators used within strategies.

Each mutator is made up of:

Setting	Type	Description
key	string	Internal short name of the mutator in camelCase
`title`	string	Human friendly title of the mutator
`description`	string	Longer description of what the mutator does
`handler`	function	Function, called as `(value)` which is expected to return the mutated input

Static: Dedupe.strategies

A lookup object of strategies.

Each strategy is made up of:

Setting	Type	Description
key	string	Internal short name of the strategy in camelCase
`title`	string	Human friendly title of the strategy
`description`	string	Longer description of the strategy
`mutators`	object	List of fields which will be mutated and how, prior to the strategy being run
`steps`	array	Array of steps to take when running the strategy

Dedupe.set(option, value)

Convenience function to quickly set a single option, or merge an object of options. Returns the original Dedupe instance.

Dedupe.run(input)

Takes an array of input references applying the action specified in Dedupe.settings.action. Returns a promise.

Strategies

This module includes a selection of deduplication strategies which are basic JavaScript objects which detail steps to take to detect reference duplication.

Each strategy should include a title, description, optional mutations and a collection of steps to perform.

A simple example of the DOI only strategy:

module.exports = {
	title: 'DOI only',
	description: 'Compare references against DOI fields only',
	mutations: {
		doi: 'doiRewrite',
	},
	steps: [
		{
			fields: ['doi'],
			comparison: 'exact',
		},
	],
};

Strategy format:

Path	Type	Default	Description
`title`	`string`		The short human-readable title of the strategy
`description`	`string`		A longer, HTML compatible description of the strategy
`mutators`	`object`		An object of the reference properties to mutate prior to processing, each value should be a known mutator
`steps`	`array`		A collection of steps for the deduplication process
`steps.skipOmitted`	`boolean`	`true`	Skip field comparison where either side is not specified
`steps[].fields`	`array`		An array of strings, each value should correspond to a known reference field
`steps[].comparison`	`string`		The comparison method to use in this step, should correspond to a known comparison method