The task
The agency needed to spot trusted domains among those about to expire (drop) — domains with solid metrics, decent age and a clean history — so it could catch and reuse them. Drop lists are scattered across dozens of marketplaces, every source has its own format, and the volumes are enormous — there is no way to cover that manually.
What we did and what data we collected
We compiled a list of platforms that publish expired- and drop-domain lists — GoDaddy, DropCatch, SnapNames, nicsell, Dynadot, Kifdom, Sav, UKBackorder, Catched, park.io, Namecheap, NameSilo, Gname, Aftermarket, Domainlore.uk and others — and set up regular collection of their feeds into a single stream.
Then came enrichment. We normalized drop dates to a single timezone so it was clear when each domain actually becomes available. We pulled metrics via the Ahrefs and Moz APIs (authority and link profile), added WHOIS data for age and registration history, and for shortlisted domains fetched Wayback Machine snapshots — to see what used to live on the domain and in which language. On top of that we built an interface where the agency's analysts run the final manual scoring over the already collected and enriched candidates.
The project was structured as data search and enrichment on top of a domain name database, delivered into a convenient interface and export (DaaS).
Challenges and how we solved them
The main challenge was scale. Hundreds of thousands of domains drop every day, and running the full battery of checks on all of them makes no sense. We built a pipeline: first collect and deduplicate the drop lists, then enrich with metrics, and run the expensive steps (web archive) only for shortlisted domains that pass the initial screening. The pipeline runs daily.
The second was source heterogeneity. Every marketplace has its own list format and its own way of stating the drop date and time. We mapped everything to a single schema and a single timezone; more on bringing data to a common shape in our article on data normalization.
The third — and the key one — was filtering out domains with a "dirty" past: those that had already dropped before and had spam hosted on them. By raw metrics such a domain can look attractive, but in reality it is burned. We checked history through the web archive and signs of previous drops so that these candidates would not slip through. Working with external APIs (Ahrefs, Moz, WHOIS) also proved useful for fast enrichment.
Worth noting: these platforms have no anti-scraping protection — the task was never about bypassing blocks, it was about scale and data quality.
The result
The agency got a single daily stream of drop domains, enriched with metrics, age and history, plus an interface for manual scoring — candidate evaluation now happens in one place instead of manually visiting dozens of marketplaces and cross-checking metrics by hand.
Services in this case study: Data Search and Enrichment · Domain Name Database · DaaS and data via API · Data for SEO and Search
Useful reading: Data enrichment · Data normalization · API scraping