Fuzzy Dedupe CSV

Fuzzy deduplication merges rows whose key values are close without being identical. Pick the columns that identify a duplicate, pick a similarity metric, and move the threshold until the groups on screen look right. Acme Inc, acme inc, Acme Inc. and Acme Incorporated collapse into one customer. Everything runs in this tab, so no file is uploaded.

Need exact matching instead? The plain deduplicator is faster and has no threshold to argue with.

Most tools that say "fuzzy" are not

There is a pattern in this corner of the web worth naming, because it wastes a lot of people's afternoons. A tool offers fuzzy deduplication. You give it a customer list. It offers two settings, usually called something like strict and loose. Under the hood, strict lowercases the value and squeezes runs of spaces, and loose additionally strips punctuation. Then it compares the results with an equals sign.

That is normalization, and normalization is genuinely useful. It is not fuzzy matching. Fuzzy matching means there is a distance between two values and a threshold you can put that distance against. Without a distance you get a tool where Acme Inc and acme inc merge, which is nice, and Acme Inc and Acme Inc. merge on the loose setting, which is also nice, and Acmé Inc sits there on its own forever because one character has an accent on it and an equals sign has no opinion about how close that is.

This page normalizes too, because normalization is cheap and it is what makes the expensive part fast. Then it measures. That is the whole difference, and it is the reason there is a threshold control on the page at all.

Worked example: one customer, four spellings, and a fifth that misses

This is the file behind the example button, companies.csv:

id,company,city,contact,revenue
1,"Acme, Inc.",Boston,ada@acme.example,120000
2,acme inc,Boston,,118000
3,Acme Incorporated,Boston,grace@acme.example,121000
4,Acmé Inc,Boston,,
5,Acme  Inc,boston,alan@acme.example,119500
6,Globex Corporation,Austin,kj@globex.example,90000
7,Globex Corp.,Austin,,91000
8,Initech,Dallas,radia@initech.example,45000
9,Initech LLC,Dallas,,45500
10,Umbrella Health,Raleigh,edsger@umbrella.example,300000

Click the example and it runs on arrival with no key column picked, so the guess is printed above the result: company. It is not id, because every value in an id column is unique by construction and matching on one merges nothing. Switch the output to the merged groups report and this comes back:

10 rows in · 5 rows out · 5 duplicates merged · key: company
· Jaro-Winkler at 0.90 · 3 groups of two or more · keeping the first row

group,rows,kept,merged_values,row_numbers
1,4,"Acme, Inc.","acme inc | Acmé Inc | Acme  Inc","2, 3, 5, 6"
2,2,Globex Corporation,Globex Corp.,"7, 8"
3,2,Initech,Initech LLC,"9, 10"

Row numbers count the header as row 1, so they line up with what a spreadsheet shows you. Three of the four Acme variants merged. Acme Incorporated did not, and that is the most useful thing on this page: Jaro-Winkler scores it against acme inc at 0.894, six thousandths under the threshold. Nothing is wrong. The tool is telling you that an extra nine characters is a real difference and asking you to decide.

Drop the threshold to 0.85 and it joins the group: 6 duplicates merged, four rows out. Push it up to 0.98 and the Acme group survives intact at four rows while Globex and Initech fall apart, because those four normalize to the identical string acme inc before any measuring happens, so they score exactly 1.0 whatever the threshold is. That is the shape of the whole tool. Normalization handles case, spacing, punctuation and accents for free, and the threshold governs only the differences that are actually differences.

One more control worth trying on this file. Set Keep to the longest key value and the survivors change to Acme Incorporated, Globex Corporation and Initech LLC, which is usually the version you want on a mailing label. Set it to the row with fewest blanks and you get the record somebody actually finished filling in.

Three metrics, and when each one is right

Jaro-Winkler counts characters that match within a sliding window, subtracts half a point for each pair that appears in the wrong order, and then adds a bonus of up to a tenth for a shared opening prefix. That prefix bonus is why it is the default: people and companies are identified by how their names start, and the variation is almost always at the end. Acme Inc against Acmé Inc still scores 0.942 with accent folding switched off, comfortably over the default threshold.
Levenshtein ratio counts single-character insertions, deletions and substitutions, then divides by the length of the longer value and subtracts from one. It is the strict one. Acme Inc against Acme Incorporated is nine edits over seventeen characters, which is 0.47, well under any sensible threshold. If you consider those two different companies, this is the metric that agrees with you.
Token set throws away order and length and compares the two sets of words. Smith, John and John Smith score 1.0. Neither of the other two metrics comes close on that pair, because character-level distance is enormous when a string has been reversed. Use this one on anything where a name might have been entered surname first.

All three return a number between 0 and 1 where 1 is identical, so the threshold control means the same thing whichever you pick and you can switch between them without relearning the scale.

Normalization comes first, and you control it

Four switches decide what the metric never has to see. All four are on by default, which is the setting most people want and the setting that makes the comparison fastest, because two values that normalize identically are handled as one and never measured at all.

Letter case. Off is for a file where IBM and Ibm are genuinely different records, which is rare and worth knowing about.
Spacing. Collapses runs of whitespace and trims the ends. It also converts non-breaking spaces, which are what you get when data has been pasted out of a web page and are invisible on screen.
Punctuation. Turns everything that is not a letter or a digit into a space, so Acme, Inc. and Acme Inc arrive at the metric as the same string.
Accents. Folds to plain ASCII, and it handles the letters that Unicode decomposition alone misses: ß becomes ss, ø becomes o, æ becomes ae. Without those, Straße refuses to match Strasse no matter how the threshold is set.

How it stays fast on a big file

Comparing every row against every other row is quadratic. A hundred thousand rows is five billion pairs, which no browser is going to finish. Three things bring that down to something that recomputes while you drag a threshold:

Distinct values, not rows. A hundred thousand orders might carry six thousand distinct customer names. The measuring happens on the six thousand, and the answer is mapped back onto the rows afterwards. That alone is usually a factor of ten or more.
Blocking. Values are grouped by the first letter of the normalized key and compared only within a group. The token metric blocks differently, putting a value into one group per word it contains, which is what lets a reversed name find its partner. When a block is still too large the key grows to two or three characters and the stats strip says so, because that is a real trade: a tighter block is faster and will not catch a typo in the opening letters.
DuckDB past twenty thousand distinct values. Both character metrics exist as native functions in DuckDB, and the blocked comparison is a self join it can run vectorized in WebAssembly. It is downloaded only when a file is big enough to need it, and if the download is blocked the JavaScript path answers instead, more slowly and identically.

The Levenshtein implementation also stops early. A threshold of 0.9 over a twenty-character value means any pair more than two edits apart is irrelevant, so the moment every cell in a row of the distance matrix passes two, the calculation returns rather than finishing the grid.

Things worth knowing before you trust the output

Merging is transitive here. If A matches B and B matches C, all three land in one group even when A and C would not have matched each other. That is usually what you want for a chain of gradually diverging spellings, and it is occasionally how one enormous group swallows half a file at a low threshold. The group sizes in the report are the early warning.
Only the key columns are compared. Everything else rides along on whichever row survives. If two rows in a group hold different revenue figures, the survivor's figure is the one in the output and the other is gone. Use the tagged output when the non-key columns matter and you want to reconcile them yourself.
A blank key is a value. Rows whose key columns are all empty normalize to an empty string and match each other, so they collapse into one. That is usually right and occasionally a surprise.
Several key columns are joined before comparing. Ticking company and city compares the pair as one string, so a company that moved cities will not merge. That is deliberate.
The file is never sorted. Survivors come back in the order they appeared, so the output is diffable against the input.
100 MB is the ceiling on this page. Past that the widget points you at the full editor, which streams the file rather than holding it all in memory at once.

Frequently Asked Questions

What makes this different from a normal CSV dedupe?

A normal dedupe compares two values with an equals sign, so it merges rows only when the key matches character for character. This one measures how far apart two values are and merges them when they are closer than the threshold you set. Acme Inc and Acme Inc. are one edit apart, so they merge; Acme Inc and Globex Corp are not, so they do not.

Which similarity metric should I pick?

Jaro-Winkler for names of people and companies, which is the default and is forgiving about differences near the end of a value. Levenshtein ratio when you are chasing typos and want strictness, because it counts every inserted character against you. Token set when the same words appear in a different order, such as Smith, John against John Smith, which the other two both score badly.

What threshold should I use?

Start at 0.90 and look at the merged groups report before you download anything. Above 0.95 you catch punctuation and spacing differences only. Below 0.85 you start merging genuinely different companies that share a first word. There is no correct number for every file, which is why the result recomputes the moment you change it.

How do I see what would merge before it happens?

Set the output control to Every row, tagged with its group. Nothing is removed; three columns are added instead, holding the group number, how many rows are in that group, and whether that row is the one that would survive. Sort by group in a spreadsheet and the decision is in front of you. The merged groups report is the same information one row per group.

Which row survives a merge?

Whichever you choose. First row of the group keeps file order, which is the safe default. Longest key value keeps Acme Incorporated over Acme Inc, which is usually the more complete name. Fewest blanks keeps the row with the most fields filled in, which is usually the record somebody actually finished.

Will it cope with a hundred thousand rows?

Yes. Rows collapse to distinct key values first, so a 100,000-row file with 6,000 distinct names costs 6,000 comparisons worth of work rather than 100,000. Those are then blocked so that comparison happens within a group rather than across the whole file, and past 20,000 distinct values the blocked join is handed to DuckDB, which has both metrics built in.

Does the file get uploaded anywhere?

No. There is no upload endpoint on this page. The file is read, compared and rewritten by JavaScript in your own tab, and the DuckDB engine that handles very large files is WebAssembly running in that same tab. Nothing is stored between visits, so a reload gives you an empty box.

Why did two obvious duplicates not merge?

Three usual reasons. The threshold is above their actual similarity, so lower it and watch the group count. The metric is the wrong one, most often Levenshtein where a whole extra word has been added. Or the file is large enough that comparison was blocked on the first two or three characters of the key, in which case a difference in the opening letters keeps two values apart; the stats strip says when that has happened.

One customer, one row

Free, no account, no upload. Pick the key, move the threshold, read the groups, take the file.

Back to the matcher