Remove Duplicates

Clean out repeated rows, either whole rows or matches on the columns you name.

What it does

Remove Duplicates deletes rows that repeat. It works in two ways. Leave the column picker empty to compare whole rows. Or pick a few columns, and the operation compares only those.

Remove exact duplicate rows

  1. Type duplicates in the Search transforms box, in the Pipeline panel.
  2. Select Remove Duplicates in the results list.
  3. Leave Select columns empty.
  4. Click Apply.

Every value in a row must match for the row to count as a duplicate. One different cell makes the row unique. It stays.

Match on the columns you choose

Often only one column decides identity. An email column is a good example.

  1. Click the Search and select columns... box under Select columns.
  2. Type part of a column name to narrow the list.
  3. Click a column to select it. Repeat for more columns.
  4. Click Apply.

Example: a contact export lists sam@example.com three times. Select the email column. Two of the three rows go away. The other columns come along for the ride.

Which row survives

  • When you pick columns, the first matching row stays. The later ones go.
  • Row order is the order in the grid at that point in your pipeline.
  • More columns make the match stricter. Fewer columns remove more rows.
  • With an empty picker, no row is favoured. Only one copy of each identical row remains.

For SQL users

An empty picker runs SELECT DISTINCT *. A picker with the email column runs as:

SELECT DISTINCT ON ("email") * FROM data
Try Remove Duplicates with sample data →

Related Operations