Find personal data in a CSV

Ten detectors, and a checksum wherever one exists

A regex that matches sixteen digits will call your order numbers credit cards. Run that across a warehouse export and half the columns light up, at which point everybody stops reading the report and the tool has done worse than nothing. So every type here that carries a mathematical check is put through it before the cell counts as a match.

• US Social Security number, high. The Social Security Administration never issued area 000, area 666 or anything from 900 upward, never issued group 00, and never issued serial 0000. All four rules are applied. On top of that, a bare nine-digit run is only accepted as an SSN when the column name says so; without a helpful header the value has to be punctuated as 078-05-1120 or 078 05 1120. Otherwise every zip+4 and every invoice number in the file becomes a false alarm.
• Payment card number, high. Thirteen to nineteen digits, spaces or hyphens allowed, and the Luhn check has to pass. Storing these puts the file in PCI scope, which is why they sit at the top of the report.
• IBAN, high. Validated with mod-97: move the first four characters to the end, turn letters into numbers with A as 10, and the remainder against 97 must be exactly 1. The arithmetic runs in chunks because the number is far larger than a double can hold. Treat a hit as a payment credential, not a reference code.
• Email address, medium. A shape check that rejects the commas, semicolons and angle brackets that mail clients leave behind, so a pasted address list does not read as one enormous address.
• Phone number, medium. Seven to fifteen digits, the E.164 range, and either a leading plus, some punctuation, or a column name that says phone, mobile, tel, cell or fax. Seven unpunctuated digits with no supporting header is an order number.
• Date of birth, medium. A date shape plus a header containing dob, birth or born. A date column on its own is a date, not a birth date, and guessing otherwise would flag every created_at in the file.
• Person name, medium. Two or three capitalized words with no digits, in a column whose header mentions name, contact, customer, employee, patient, owner or author. Headers like username, filename, product_name and company_name are excluded, because those are the ones that make a name detector unusable.
• IPv4 and IPv6, low. Both, and the IPv6 detector handles the compressed double-colon forms and the IPv4-mapped tail. An address is personal data under GDPR whenever it can be tied to a person, which is most of the time in an application log.
• Postal code, low. UK, US zip and zip+4, Canadian, and the plain four and five digit European shapes, again only in a column that says zip, postal or postcode. Harmless on its own. Dangerous in combination, which is a theme on this page.

Order matters as much as the patterns. Dates and postal codes are tested before phone numbers, because to a regex they look the same: 1987-04-12 is eight digits with separators and 94043-1351 is nine, which is exactly the shape of a phone number. Check phones first and every birth date in the file gets reported as a mobile. Checksummed types come first of all, so a valid card is never demoted to a phone.

What a value scan cannot see

This is the part most scanners leave out, so here it is up front. The report tells you what a set of pattern detectors could recognize in the cells of one file. Three whole categories of personal data are invisible to that approach, and no amount of extra regexes fixes any of them.

• Free text. A support notes column containing "called Priya back about the Camden flat" holds a name and a location, and neither is in a shape a detector can match. Long text columns are where personal data accumulates without anybody deciding to put it there. Read them.
• Internal identifiers. A customer key, a device id, a session token, a case number. None of them identifies anybody by itself, all of them identify somebody the moment they meet the system that issued them. Under GDPR that makes them pseudonymous data, which is still personal data, and the scan cannot tell them apart from an ordinary reference code.
• Quasi-identifiers in combination. The famous result here is Latanya Sweeney's: a five digit zip code, a full date of birth and a gender identify a large majority of the US population uniquely. Each of those columns is unremarkable alone. Together they are a name. Postal code plus birth date plus job title usually does the same thing on an HR export. A per-column scan is structurally incapable of seeing a risk that only exists across columns.

When nothing matches, the report does not print a reassuring green tick. It says nothing was detected and then spells out those three limits in the output itself, because a clean scan misread as a clean file is the specific failure that gets data shared when it should not be. Use the finding list as a floor. Read the columns it did not name.

Practical notes

• Every cell is read. No sampling, because the whole risk is that the leak is in the rows you skipped. The ceiling is twenty million cells; past it the page stops and reports the count rather than locking the tab, and suggests splitting the file.
• Blank cells are excluded from the denominator. A finding reads as matched against non-blank, so a sparse column is judged on the values it actually holds.
• One column, several findings. A generic contact column carrying emails on some rows and mobiles on others produces two findings with two counts and two sets of samples, rather than one guess.
• Headers help, and are used carefully. A column called dob promotes a date to a birth date and a column called full_name promotes two capitalized words to a name. Neither promotion happens on the value alone, because both patterns are far too loose to trust without the header.
• The findings download is redacted too. It carries the column, the type, the severity, the matched and non-blank counts, the share and the redacted samples. Nothing in it is a real value.
• Nothing is uploaded. There is no endpoint to upload to. Check the network tab if you want; that is a reasonable thing to do before feeding a tool a file full of card numbers.

Once you know what is in the file, the next step is usually to rewrite it. Send the same CSV to the anonymizer, which hashes, masks, fakes or generalizes column by column and tells you exactly which columns it left alone.