URL Extractor

Strip first, then deduplicate. The order is the whole trick

A marketing team posts one article six times. Each post carries a different utm_source, and two of them also carry a utm_campaign. A seventh copy arrives from somebody who pasted the clean link. To a string comparison those are seven distinct URLs, so a tool that deduplicates first removes nothing at all, and you are left doing the work by eye.

Strip the tracking parameters first and the same seven collapse to one:

https://example.com/blog/release-4-2?utm_source=newsletter&utm_medium=email&utm_campaign=july
https://example.com/blog/release-4-2?utm_source=twitter
https://example.com/blog/release-4-2

  becomes

https://example.com/blog/release-4-2

The list of stripped parameters is explicit, not a prefix rule. That matters: ref means "referrer" on some sites and "reference number" on others, and id is nearly always load-bearing. Everything on the list is unambiguously a campaign tag or a click identifier: the utm_ family, gclid, gbraid, wbraid, dclid, fbclid, msclkid, twclid, yclid, ttclid, igshid, mc_cid, mc_eid, mkt_tok, _hsenc, the Matomo and Piwik pairs, and a couple of dozen more. Whatever it removed is named in the note above the table, so you can see it happened.

Worked example: nine matches, five rows

The sample text holds nine link-shaped runs. With the defaults on, five rows come out.

url
https://example.com/blog/release-4-2
http://exmpl.co/r42
https://docs.example.com/guide/getting-started#install
https://en.wikipedia.org/wiki/Ada_(programmer)
https://www.example.org/about
ftp://mirror.example.net/pub/release-4.2.tar.gz
mailto:support@example.com
tel:+442079460018

The Wikipedia bracket survived. The fragment #install survived, because a fragment identifies a section and removing it would point at a different place on the page. The three campaign variants of the release post became one. And mailto: and tel: are in there, which most extractors miss entirely because their pattern demands a :// that neither scheme has.

Set Keep to http and https only and the last three drop out with the count reported. Set it to https only and the plain-http short link goes too. Tick Domain column and you get a second column reading example.com, exmpl.co, docs.example.com and so on, with any www. prefix removed so that www.example.org and example.org group together.

Things people use this for

• Auditing a newsletter before it goes out. Paste the rendered HTML, get every link, check the domains column for anything you did not expect.
• Cleaning a research bookmark dump. A month of saved links arrives full of campaign tags and duplicates; this collapses it to the distinct pages.
• Pulling references out of a document. Every URL cited in a report, in the order it appeared, ready to check.
• Extracting outbound links from a page's source. Paste the HTML; href values in the markup are found the same as links in prose.
• Making a crawl list. The domain column plus a sort gives you a per-host grouping in one step.

Once the links are a CSV, the full workbench will group them by domain, count them, and chart the result without you writing anything.