# @flatfile/plugin-dedupe

> Dedupe records in a sheet via a sheet level custom action.

Latest version **1.5.1** (published 2024-12-19) · ISC license · 0 weekly downloads

## Install

```sh
npm install @flatfile/plugin-dedupe
pnpm add @flatfile/plugin-dedupe
yarn add @flatfile/plugin-dedupe
bun add @flatfile/plugin-dedupe
```

## Health

**Score 50/100 (C)** — status: stable.

Positive: has types; esm support; no vulnerabilities; high quality score.

Warnings: low downloads.

Negative: stale.

## Facts

| | |
|---|---|
| Version | 1.5.1 |
| Published | 2024-12-19 |
| First published | 2023-08-08 |
| Weekly downloads | 0 |
| License | ISC |
| TypeScript types | bundled |
| Module format | ESM + CommonJS |
| Node | >= 18 |
| Dependencies | 2 |
| Unpacked size | 51.7 KB |
| Known vulnerabilities | 0 |
| Install scripts | no |
| GitHub stars | 7 |
| Author | Flatfile, Inc. |
| Maintainers | sarocu, dboskovic, nate.ferrero, jmmander, madmandrit, bangarang, carlbrugger, flatfileinfra, flatderek, bigcountrycrane, flatfilecolin, alnoor, rjhyde, sambarrowclough, meritmalling, mmccooyyy |
| Keywords | flatfile-plugins, category-transform |

## Links

- npm: https://www.npmjs.com/package/@flatfile/plugin-dedupe
- Repository: https://github.com/FlatFilers/flatfile-plugins
- Homepage: https://github.com/FlatFilers/flatfile-plugins#readme
- Issues: https://github.com/FlatFilers/flatfile-plugins/issues
- npm.io page: https://npm.io/package/@flatfile/plugin-dedupe

## Dependencies (2)

- [@flatfile/util-common](https://npm.io/package/@flatfile/util-common.md) ^1.6.0
- [@flatfile/plugin-job-handler](https://npm.io/package/@flatfile/plugin-job-handler.md) ^0.8.1

## Alternatives

- [mobx-react](https://npm.io/package/mobx-react.md) — 2.8M weekly downloads
- [rc-tree](https://npm.io/package/rc-tree.md) — 2.6M weekly downloads
- [@react-oauth/google](https://npm.io/package/@react-oauth/google.md) — 1.3M weekly downloads
- [@wagmi/connectors](https://npm.io/package/@wagmi/connectors.md) — 877.0K weekly downloads
- [vee-validate](https://npm.io/package/vee-validate.md) — 836.4K weekly downloads

## Recent versions

- 1.5.1 (latest) — 2024-12-19
- 1.5.0 — 2024-11-07
- 1.4.0 — 2024-10-31
- 1.3.0 — 2024-10-29
- 1.2.1 — 2024-10-03
- 1.2.0 — 2024-09-24
- 1.1.5 — 2024-09-03
- 1.1.4 — 2024-05-29
- 1.1.3 — 2024-05-20
- 1.1.1 — 2024-04-30
- 1.1.0 — 2024-04-23
- 1.0.4 — 2024-04-12
- 1.0.3 — 2024-04-10
- 1.0.2 — 2024-04-08
- 1.0.1 — 2024-03-29
- … 11 more at https://npm.io/package/@flatfile/plugin-dedupe/versions

## README

<!-- START_INFOCARD -->

This plugin dedupes records in a Sheet via a Sheet-level action.

**Event Type:**
`listener.on('job:ready')`

<!-- END_INFOCARD -->


> When embedding Flatfile, this plugin should be deployed in a server-side listener. [Learn more](/docs/orchestration/listeners#listener-types)


## Parameters

#### `jobOperation` - `string` - (required)

The `jobOperation` parameter specifies the name of the job operation to match on.

#### `opt.on` - `string`

The `on` parameter specifies which field key to match on.

#### `opt.keep` - `'first' | 'last'`

The `keep` option lets you choose whether to keep the first or last duplicate record.

#### `opt.custom` - `function()`

The `custom` parameter accepts a custom dedupe function. This will override the `keep` parameter.

#### `opt.debug` - `boolean`

The `debug` parameter lets you toggle on/off helpful debugging messages for development purposes.


## API Calls

- `api.records.get`
- `api.jobs.ack`
- `api.records.delete`
- `api.jobs.fail`
- `api.jobs.complete`


## Usage

An action with the operation name of "dedupe-email" must be configured on a Sheet for the plugin to be triggered.

```bash install
npm i @flatfile/plugin-dedupe
```

```js import
import { dedupePlugin } from "@flatfile/plugin-dedupe";
```

```ts contactsSheet.ts
import { Flatfile } from '@flatfile/api'

export const contactsSheet: Flatfile.SheetConfig = {
  name: 'Contacts',
  slug: 'contacts',
  fields: [
    {
      key: 'firstName',
      type: 'string',
      label: 'First Name',
    },
    {
      key: 'lastName',
      type: 'string',
      label: 'Last Name',
    },
    {
      key: 'email',
      type: 'string',
      label: 'Email',
    }
  ],
  // Add a Sheet-level action here for the dedupe plugin
  actions: [
    {
      operation: "dedupe-email",
      mode: "background",
      label: "Dedupe emails",
      description: "Remove duplicate emails"
    }
  ]
}
```

### JavaScript

#### listener.js 

```js listener.js
// common usage
// Keep the last record encountered (from top to bottom) based on the`email` field key.
// Must have a Sheet-level action specified with the operation name `dedupe-email`
listener.use(
  dedupePlugin("dedupe-email", {
    on: "email",
    keep: "last",
  })
);

// user specified dedupe function must return a list a record id's for deletion
listener.use(
  dedupePlugin("dedupe-email", {
    custom: (records) => {
      let uniques = new Set();
      let toDelete = [];

      records.forEach(record => {
        const { value } = record.values["email"];
        if (uniques.has(value)) {
          toDelete.push(record.id);
        } else {
          uniques.add(value);
        }
      });

      return toDelete;
    },
  })
);
```

### TypeScript

#### listener.ts 

```ts listener.ts
// common usage
// Keep the last record encountered (from top to bottom) based on the`email` field key.
// Must have a Sheet-level action specified with the operation name `dedupe-email`
listener.use(
  dedupePlugin("dedupe-email", {
    on: "email",
    keep: "last",
  })
);

// user specified dedupe function must return a list a record id's for deletion
listener.use(
  dedupePlugin("dedupe-email", {
    custom: (records: Flatfile.RecordsWithLinks) => {
      let uniques = new Set();
      let toDelete = [];

      records.forEach(record => {
        const { value } = record.values["email"];
        if (uniques.has(value)) {
          toDelete.push(record.id);
        } else {
          uniques.add(value);
        }
      });

      return toDelete;
    },
  })
);
```

---
_Source: https://npm.io/package/@flatfile/plugin-dedupe · Machine-readable twin of the npm.io package page. Health data is recomputed on every publish._