CSV Deduplicator

Drop a CSV in, or paste a few rows, and the repeats come out. The first copy of each row stays exactly where it was, and you get told how many were kept and how many went. The whole comparison happens inside this page.

Need to sort or filter before you decide what counts as a repeat? Open the app

Where the repeats come from

Hardly anybody types the same row twice. Duplicates arrive through the plumbing, and the shape of the plumbing decides which columns you should be matching on.

Two exports stitched together. Monday's pull and Tuesday's pull overlap by a day, and the overlap ends up buried in the middle of the combined file where no scroll is going to find it.
A join that fanned out. One order with three line items comes back as three rows, with the order-level columns repeated on each. Whole-row matching leaves those alone, since the line items differ. Matching on the order id collapses them to one.
Retries and replays. A webhook that timed out gets resent, so the same event lands twice with the same id and a different received-at stamp. That one differing column is why whole-row matching finds nothing wrong.
Merged mailing lists. Conference badge scans, a newsletter signup form and a CRM export in one pile, with a few hundred addresses sitting in more than one of them.
A form submitted twice. Someone double-clicked, and the two rows differ only in a timestamp measured to the second.
A report pasted below itself. Usually with its header line still attached, halfway down the file.

Notice how few of those produce rows that repeat end to end. That is why this page asks what counts as a duplicate rather than deciding for you.

Worked example: six orders, two of them twice

A small orders file. The fourth and sixth lines repeat the second and third exactly.

order_id,customer,email,amount
10441,Rita Menon,rita.menon@example.com,240.00
10442,Sam Okafor,sam.okafor@example.com,89.50
10441,Rita Menon,rita.menon@example.com,240.00
10443,Rita Menon,rita.menon@example.com,15.00
10442,Sam Okafor,sam.okafor@example.com,89.50
10444,Priya Nair,priya.nair@example.com,320.75

With the setting left on every column, that comes back as:

order_id,customer,email,amount
10441,Rita Menon,rita.menon@example.com,240.00
10442,Sam Okafor,sam.okafor@example.com,89.50
10443,Rita Menon,rita.menon@example.com,15.00
10444,Priya Nair,priya.nair@example.com,320.75

Above the result the widget reads 6 rows in, 4 kept, 2 duplicates removed, matched on every column, and a file named orders.csv comes back as orders-deduped.csv. Both survivors kept their original positions, so 10443 is still between 10442 and 10444.

Now switch to chosen columns and tick email. The question stops being which rows repeat and becomes which people repeat:

order_id,customer,email,amount
10441,Rita Menon,rita.menon@example.com,240.00
10442,Sam Okafor,sam.okafor@example.com,89.50
10444,Priya Nair,priya.nair@example.com,320.75

Three kept, three removed, and the summary now says matched on email. One of the casualties is order 10443, a genuine second purchase of fifteen dollars, gone because Rita's address had already been seen. For a mailing list that is the correct answer. For a revenue total it is a disaster, and the difference between the two is one setting, which is exactly why the summary prints the columns it used.

What the matching actually does

The separator is sniffed from the text, so semicolon, tab and pipe files work without being announced. The first line becomes the header and is set aside. Every remaining row is then reduced to a key built from its match columns, and those keys go into a set as they are met; a row whose key is already there gets skipped. One pass, top to bottom. That is the entire method, and two things follow from it.

First, order survives. Nothing is sorted, grouped or shuffled, so the rows you get back are in file order with gaps where the repeats used to be. Run the output through again and it is unchanged, which makes this safe to bolt onto a routine you repeat every week.

Second, the comparison is on raw text. No trimming, no case folding, no interpreting digits as quantities. 007 and 7 are two values. So are Rita@Example.com and rita@example.com. That is stricter than a human reader, and it errs in the direction you want: a pair that arguably should have merged is still sitting in front of you where you can see it, rather than a pair that should not have merged being quietly gone.

In whole-row mode the columns are compared by position rather than by name, so a file that somehow carries two columns both called id still has both of them checked. The result is written back as comma-separated CSV with quotes only where a field needs them.

When the file is bigger than a spreadsheet

Remove Duplicates in Excel is one button, and for a file you can open it is the right button. It becomes unavailable at 1,048,576 rows, which is the moment most people start searching for something else. A three million row extract will not open at all, never mind deduplicate, and the answers waiting in the forum threads are a Python script, a sort -u pipeline, or a Power Query load long enough to make tea during.

Nothing here counts rows. The only ceiling is 100 MB of file, which at ordinary CSV widths is a few million lines, and in practice what gives out first is however much memory your browser tab is prepared to hand over, since the input and the result both live in it. Past 100 MB the widget declines and points at the full editor, which reads a file in pieces rather than all at once. There is no metering either, so running the same file four times while you work out which columns matter costs nothing but four clicks.

Before you trust the count

Trailing spaces hide repeats. A hand-edited export where one row has 02215 and the next has a space in front of the same digits gives two keys, and both rows stay. If the count comes back lower than you expected, that is the first thing to check.
A misspelled column name fails quietly. In chosen-columns mode a name that is not in the header gets skipped with a note listing it, and if none of the names you asked for survive that check, the match falls back to the first column in the file. Read the summary line before believing the number: it states which columns were used.
A stray header line in the middle is just a row. Two concatenated exports leave one behind, and it only vanishes if some other row matches it exactly. Nothing usually does, so that line is yours to remove.
Ragged rows get padded first. A row with fewer fields than the widest one is filled out with empty cells before any comparison, and the note says how many rows that happened to. A padded row and a naturally full row are still different keys.
Subset mode does not merge anything. Match on email and the survivor keeps its own name, amount and dates, taken from the first occurrence. Values sitting on the rows that were dropped are not folded in anywhere, so if the later copy held the only phone number, extract it before you run this.
Blank rows never make it in. Empty lines are dropped while the file is being read, so a file padded with them at the bottom does not report a pile of duplicates that were never really there.

Frequently Asked Questions

Which copy of a duplicate row is kept?

The first one. Rows are visited from the top, the first time a combination of values turns up it is kept, and every later row carrying the same combination is dropped. Nothing is sorted along the way, so what comes back sits in the order it had in the file, minus the repeats. If the copy you want is the later one, sort the file so it comes first and then deduplicate.

Can I match on one column instead of the whole row?

Yes. Switch the setting from every column to chosen columns and a picker appears listing your header. Tick email and two rows count as the same when the address matches, however much the other columns differ. Tick several and all of them have to match. Be deliberate about it: the row that survives keeps its own values everywhere else, and the rows that went are not merged into it.

Are 007 and 7 treated as the same value?

No. Cells are compared as the exact text in the file, with no trimming and no reading digits as numbers, so 007 and 7 are two different values and so is 007 with a space in front of it. Case counts too. That strictness is also why leading zeros are still intact in the download rather than being helpfully shortened.

How many rows can it handle?

There is no row cap at all. The ceiling is 100 MB of file, which at ordinary CSV widths is a few million rows, well past the 1,048,576 rows a worksheet will hold. What tends to run out first is the memory your browser tab is willing to give up, since the input and the result are both held there. Over 100 MB the widget stops and offers the full editor, which reads a file in pieces.

Does my file get uploaded?

There is no server side to this page. The CSV is read by JavaScript inside your tab, compared there, and written back out there, so closing the tab is the entire cleanup story. Given that the files people deduplicate are usually customer lists, order histories and mailing lists, that seemed like the only defensible design.

What happens to the header row?

The first line is treated as the header and is never a candidate for removal, and its names are what the column picker offers you. A second header line further down, which is what you get when two exports were concatenated, counts as an ordinary row. It only disappears if some other row matches it exactly, which usually nothing does, so delete that line yourself.

Clean the repeats out of your CSV

Drop the file, choose whether a duplicate means the whole row or a couple of columns, then read the count before you download. Free, and no account to make.

Back to the deduplicator