The task
The startup needed the most complete list possible of content websites in the .co.uk zone — blogs, news sites and other editorial platforms — as the foundation of a training corpus. The value lay precisely in completeness and cleanliness: capture all the content sites without dragging commercial websites and junk domains into the list.
What we did and what data we collected
We crawled the .co.uk domain space, identified the live websites and collected content-site signals for each one: section structure, feeds and RSS, publishing frequency and topic. English language was a hard requirement for inclusion — non-English sites in the zone were excluded. The news segment was additionally cross-checked against the Google News index — that is how we verified and supplemented the list of outlets that definitely qualify as news media.
Then we worked from the opposite direction — not only finding "our" sites but actively cleaning out the noise:
- we removed websites of companies listed on Google Maps under categories we did not need (shops, services, food outlets and the like) — this filtered out commercial sites that could structurally pass for content platforms;
- we dropped sites with a zero rating in SEO tools — those are almost certainly junk or abandoned domains.
Besides the domain registry itself (typed as blog / news), we also collected the HTML pages of these websites for the corpus.
Challenges and how we solved them
First, zone completeness: there is no public exhaustive list. We assembled the registry from several sources and reconciled it with the Google News index to close the gaps on the news side.
Second — the key one — separating content sites from commercial ones. At this scale there are few reliable positive signals that "this is a blog or a news outlet", so we inverted the problem and subtracted known commercial domains by matching against Google Maps: if a domain belongs to a business in an irrelevant category, it is not our site.
Third, junk and abandoned websites. We removed domains with a zero rating in SEO tools so the corpus would not pick up noise.
The result
The startup received both a clean registry of the zone's content domains (blogs and news, no commerce and no junk) and the content itself — an archive of HTML pages, gzip-compressed and stored in S3-type object storage we gave the client access to. In other words, a ready-to-use corpus, not just a list of websites.
Services in this case study: Domain Name Database · News Scraping · Media Database · Google Maps Scraping · Data for AI/ML · DaaS and data via API
Useful reading: RSS feed parsing · Site structure crawling · Website data extraction