Ctrio Lab · Updated 2026-10-24

Merge Apify datasets and remove duplicates by URL or any field

Q: How do I keep the newest copy of each item?

List the datasets oldest first and set keep to last. The Actor then reads the input twice and writes the last occurrence of each key.

Q: Can it run custom JavaScript transforms?

No. Use the fields and rename inputs to choose and rename output fields.

When a scraper runs every day, each run has its own dataset, and the same items show up again and again. Our Merge & Deduplicate Datasets Actor on Apify reads any number of datasets in order, writes them into one dataset and drops duplicates by one or more fields.

Short answer: put the dataset IDs in datasetIds (each run's results are its default dataset, on the run's Storage tab), set dedupeFields to ["url"] or any combination of fields, and choose keep: "first" or "last". A SUMMARY record lists items read, unique items and duplicates removed.

Inputs

Field	What it does
`datasetIds`	Datasets to read, in order: dataset IDs or `username~dataset-name`. Required.
`dedupeFields`	Fields whose combined values must be unique. Dot paths for nested values, such as `address.city` or `images.0.url`. Empty means merge only.
`keep`	`first` (default) or `last`. With datasets listed oldest first, `last` keeps the newest copy. `last` reads the input twice.
`normalizeValues`	Compare after trimming, collapsing spaces and lowercasing (default on), so `ACME Inc` equals `acme inc`.
`fields`, `rename`	Optional: output only these fields in this order, and rename fields or dot paths.
`pageSize`	Items per API call (default 1,000). Lower it for very large items.

Items that have none of the dedupe fields are never treated as duplicates. They are kept and counted in itemsWithoutDedupeKey.

Example

Our public demo dataset CotvbQSQ340wBAwq3 has 19 coffee shop items scraped on different days, with the same URL written in different ways (/places/, /PLACES/, a trailing space). Input:

{
  "datasetIds": ["CotvbQSQ340wBAwq3"],
  "dedupeFields": ["url"],
  "keep": "last",
  "fields": ["url", "title", "rating", "address.city", "scrapedAt"],
  "rename": { "address.city": "city" }
}

Output on 2026-10-24: 10 items. The three copies of Blue Bottle Coffee (ratings 4, 3.9 and 4.1, scraped on 20, 22 and 23 October) became one item with the newest values:

[
  { "url": "https://example.com/places/blue-bottle-coffee", "title": "Blue Bottle Coffee",
    "rating": 4.1, "city": "San Francisco", "scrapedAt": "2026-10-23T09:00:00.000Z" },
  { "url": "https://example.com/places/joe-the-juice", "title": "Joe & The Juice",
    "rating": 4.1, "city": "Oakland", "scrapedAt": "2026-10-20T09:00:00.000Z" }
]

The SUMMARY record of the same run with keep: "first": {"itemsRead": 19, "uniqueItems": 10, "duplicatesRemoved": 9, "itemsWithoutDedupeKey": 0}.

Large datasets

The Actor never loads a whole dataset. It reads one page while the next one is fetched, writes the unique items and keeps only a 16-character hash of each unique key, about 55 MB per million unique keys. In our tests with 256 MB of memory:

Input	Keep	Output	Peak memory	Time
105,000 items of about 1 KB	first	10,200 items	125 MB	48 s
Same data, with field selection	last (two passes)	10,200 items	88 MB	not measured

If the platform moves the run to another server, the Actor resumes without writing the same item twice.

Limits

The Actor runs with limited permissions, so it cannot list your runs by itself. Pass dataset IDs.
Dataset URLs are not accepted, only IDs or username~dataset-name.
No custom JavaScript transforms. Use fields and rename.
Output order follows the input: dataset order, then item order (for keep: "last", the position of the last copy).

Price

$0.05 per 1,000 items written to the output dataset, plus an Actor start fee of $0.00005. Items read and duplicates removed are not charged. Higher Apify plans get up to 30% off.

Open Merge & Deduplicate Datasets Send a dataset to Google Sheets

FAQ

How do I keep the newest copy of each item?

List the datasets oldest first and set keep to last. The Actor then reads the input twice and writes the last occurrence of each key.

Does it change my source datasets?

No. The Actor runs with limited permissions, only reads the datasets you pick and writes to the run's own dataset.

Can it run custom JavaScript transforms?

No. Use the fields and rename inputs to choose and rename output fields.

Sources: Apify dataset documentation; our own runs on 2026-10-23 and 2026-10-24.