Slugify a CSV Column

The bug that eats the vowels

Almost every slug implementation is one line: lowercase the text, replace everything outside [a-z0-9] with a hyphen, collapse the runs. It works beautifully on English product names and destroys everything else.

Crème Brûlée     →  cr-me-br-l-e      (the naive version)
Crème Brûlée     →  creme-brulee      (folding first)

The accented characters are not in [a-z0-9], so they are thrown away along with the spaces, and what is left is a URL nobody can read and no search engine can do anything with. A catalogue of French or Spanish products becomes a hundred pages of hyphens.

Folding first fixes it, and folding is more than stripping combining marks. Unicode normalization splits é into an e plus a combining acute, so removing the marks leaves the letter behind. That covers most of Europe. It does not cover the letters where the mark is welded into the glyph, and those get spelled out by name first:

ß → ss     Straße Map    →  strasse-map
ø → o      Ørsted        →  orsted
æ → ae     Æthelred      →  aethelred
ł → l      Łódź          →  lodz
þ → th     þing          →  thing

Without that list, Straße comes out as stra-e and quietly refuses to match strasse anywhere it is used as a key.

Separators, and the length cap

Four separators, and the choice is usually made for you by whatever is going to read the slug.

• Hyphen for anything that will appear in a URL. Search engines treat a hyphen as a word boundary and an underscore as a letter, which is the entire reason web slugs settled on hyphens.
• Underscore for a value that is about to become a filename, a table name or a field in code.
• Dot for hierarchical keys, the kind that end up in a config file or a metrics namespace.
• None when the slug is a compact identifier and every character costs.

Max length exists because plenty of systems cap a slug field, and truncation is where slug tools get careless. Cutting at a fixed character count leaves you with creme-brulee-k, which reads like a mistake. The cut here backs up to the last separator when one is within reach, so a slug capped at 14 characters comes back as creme-brulee on a whole word, and any separator left dangling at the cut is removed.

One consequence worth knowing: truncation creates collisions. Two products whose names differ only after character 30 produce the same 30-character slug. Turn the numbering on whenever the cap is on.

Details

• Numbering is per column. Two different columns can each hold blue-shirt without one of them being renamed, because a collision only means something within a single column.
• The numbering is first-come. The first occurrence in file order keeps the plain slug and later ones get -2, -3. That is stable across runs on the same file, so a re-run does not reshuffle your URLs.
• Invisible characters are removed first. Zero-width characters and non-breaking spaces are cleared before folding, so a title pasted out of a web page does not produce a slug with a phantom separator in it.
• Case can be left alone. Switch Letter case to Leave alone when the slug is going somewhere case-sensitive that you do not control. Lowercase is the default because URLs are compared case-sensitively by most servers and mixed-case slugs are a support ticket waiting to happen.
• The original is kept by default, with the slug in a new column named <column>_slug right beside it, so you can eyeball the pairs before committing.