@flatfile/implementation-utils-dedupe-worker/README.md

Provides functionality for easily implementing record deduplication logic.

# @flatfile/implementation-utils-dedupe-worker

Provides a job worker for deduping records in a sheet as well as utility functions for deduping records.

## Features

- **`deduplicateRecords`**: A function for deduping records in a sheet based on specified dedupe keys.
- **`getDedupeActionInputForm`**: A function for getting the input form for the dedupe action.
- **`DedupeJobWorker`**: An abstract job worker for deduping records in a sheet, compatible with both `@flatfile/safe-api`'s FlatfileRecord and the legacy `Item` class from the embedded "safe" client.

## Usage

To use the `DedupeJobWorker`, you need to create a concrete class that extends it and implements the `fetchRecords` and `writeRecords` methods. This allows you to define how records are retrieved from and written back to your sheet.

### Example Implementation

Here's a complete example of how to set up a deduplication action on a sheet.

1.  **Create your Dedupe Worker**

    Create a file, for example `my-dedupe.worker.ts`, and define your worker by extending `DedupeJobWorker`. This implementation uses `@flatfile/safe-api` for easier record handling.

    ```typescript
    import type { Collection } from "collect.js";
    import { DedupeJobWorker } from "@flatfile/dedupe-worker";
    import api from "@flatfile/safe-api";

    export class MyDedupeWorker extends DedupeJobWorker {
      /**
       * Fetches all records from the sheet using a stream.
       */
      async fetchRecords() {
        return await api.records.stream({ sheetId: this.sheetId });
      }

      /**
       * Writes the updated, deleted, and new records back to the sheet.
       */
      async writeRecords(records: Collection) {
        return await api.records.write({ sheetId: this.sheetId }, records);
      }
    }
    ```

2.  **Register the Action on a Sheet**

    When defining your sheet, add an action with the input form from `getDedupeActionInputForm`.

    ```typescript
    import { getDedupeActionInputForm } from "@flatfile/dedupe-worker";

    const mySheetFields = [
      { key: "firstName", type: "string", label: "First Name" },
      { key: "lastName", type: "string", label: "Last Name" },
      { key: "email", type: "string", label: "Email" },
    ];

    export const MY_SHEET = {
      name: "My Sheet",
      slug: "my-sheet",
      fields: mySheetFields,
      actions: {
        dedupe: {
          operation: ACTION_DEDUPE,
          label: "Dedupe",
          description:
            "Dedupe records in this sheet based on the selected field.",
          mode: "foreground",
          primary: false,
          inputForm: getDedupeActionInputForm(mySheetFields),
        },
      },
    };
    ```

3.  **Register the Listener**

    In your main listener file (e.g., `index.ts`), import your worker and use the `worker()` utility from `@flatfile/implementation-utils-job-worker` to simplify listener setup.

    ```typescript
    import { FlatfileListener } from "@flatfile/listener";
    import { worker } from "@flatfile/implementation-utils-job-worker";
    import { getDedupeActionInputForm } from "@flatfile/dedupe-worker";
    import { MyDedupeWorker } from "./my-dedupe.worker";

    export default function (listener: FlatfileListener) {
      listener.use(worker(MyDedupeWorker));
    }
    ```