A sitemap is an XML file listing a site's URLs that the owner publishes for search engines and crawlers. For data collection it's invaluable: instead of blindly following link after link from page to page, you get a ready-made list of addresses, often with a last-modified date and a priority. That's why the smart move is to crawl the sitemap first, before you ever start parsing HTML. This article continues our overview of document formats; and since a sitemap is a special case of XML, the general tree-handling techniques live in the article on parsing XML.
File structure
A standard sitemap follows the sitemaps.org schema. Each URL is a <url> element with a required <loc> and optional <lastmod>, <changefreq>, and <priority>.
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/product/101</loc>
<lastmod>2026-05-12</lastmod>
<priority>0.8</priority>
</url>
<url>
<loc>https://example.com/product/102</loc>
<lastmod>2026-06-01</lastmod>
</url>
</urlset>
The important detail is the namespace http://www.sitemaps.org/schemas/sitemap/0.9. Because of it, a "naive" search for the loc tag without accounting for the namespace returns nothing; that's the single most common mistake when parsing sitemaps (more on namespaces in the XML article).
Sitemap index and nested maps
On large sites, a single file can't hold all the links (the limit is 50,000 URLs or 50 MB), so a sitemap index is used — a file that points to other sitemaps. Its root element is <sitemapindex>, and inside it are <sitemap> elements with nested <loc> values.
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://example.com/sitemap-products.xml</loc></sitemap>
<sitemap><loc>https://example.com/sitemap-articles.xml.gz</loc></sitemap>
</sitemapindex>
So a reliable crawler has to be recursive: first read the index, then download and parse each nested map in turn. Note that nested files are often served as gzip (.xml.gz) — you need to decompress them on the fly.
Where to find the sitemap
The sitemap's address usually sits in robots.txt, on a Sitemap: line. If it isn't there, check the standard paths /sitemap.xml and /sitemap_index.xml. Start with robots.txt — it's the most reliable way to learn the current location, and a single site can list several sitemaps there.
Python
The most transparent path is to download the file and parse it as ordinary XML with lxml, remembering the namespace and the recursion over the index.
import gzip
import requests
from lxml import etree
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def fetch(url):
data = requests.get(url, timeout=30).content
if url.endswith(".gz"):
data = gzip.decompress(data)
return etree.fromstring(data)
def parse_sitemap(url, urls=None):
urls = urls if urls is not None else []
root = fetch(url)
# if this is an index — descend into the nested maps
for loc in root.xpath("//sm:sitemap/sm:loc/text()", namespaces=NS):
parse_sitemap(loc.strip(), urls)
# regular links
for loc in root.xpath("//sm:url/sm:loc/text()", namespaces=NS):
urls.append(loc.strip())
return urls
links = parse_sitemap("https://example.com/sitemap.xml")
print(len(links), "URLs found")
If you'd rather not write the recursion and gzip handling by hand, there's the dedicated library ultimate-sitemap-parser: it finds the sitemap on its own, walks the indexes, and returns a flat list of pages.
from usp.tree import sitemap_tree_for_homepage
tree = sitemap_tree_for_homepage("https://example.com/")
for page in tree.all_pages():
print(page.url, page.last_modified)
JavaScript / Node.js
In Node, a sitemap is convenient to parse with fetch plus fast-xml-parser (the same one used for XML).
const { XMLParser } = require("fast-xml-parser");
async function parseSitemap(url) {
const xml = await (await fetch(url)).text();
const parser = new XMLParser();
const doc = parser.parse(xml);
if (doc.sitemapindex) {
const maps = [].concat(doc.sitemapindex.sitemap);
const all = [];
for (const m of maps) all.push(...await parseSitemap(m.loc));
return all;
}
return [].concat(doc.urlset.url).map((u) => u.loc);
}
parseSitemap("https://example.com/sitemap.xml").then((urls) =>
console.log(urls.length)
);
Turning the sitemap into an efficient crawl queue
A raw list of URLs is only the start. The <lastmod> and <priority> fields let you turn a one-shot dump into a smart crawl queue, so you fetch what matters and skip what hasn't changed.
from datetime import datetime, timezone
from lxml import etree
import requests
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def parse_entries(url):
"""Return (loc, lastmod, priority) tuples for a single urlset."""
root = etree.fromstring(requests.get(url, timeout=30).content)
entries = []
for u in root.xpath("//sm:url", namespaces=NS):
loc = u.findtext("sm:loc", namespaces=NS)
lastmod = u.findtext("sm:lastmod", namespaces=NS)
priority = u.findtext("sm:priority", namespaces=NS) or "0.5"
entries.append((loc, lastmod, float(priority)))
return entries
# Keep only pages changed since your last crawl, highest priority first
last_run = datetime(2026, 5, 1, tzinfo=timezone.utc)
def changed_since(lastmod):
if not lastmod:
return True # no date -> treat as changed, fetch to be safe
return datetime.fromisoformat(lastmod).replace(tzinfo=timezone.utc) > last_run
queue = sorted(
(e for e in parse_entries("https://example.com/sitemap-products.xml")
if changed_since(e[1])),
key=lambda e: e[2],
reverse=True,
)
print(f"{len(queue)} pages to crawl this run")
What to do with the last-modified dates
The <lastmod> field is the main reason to love sitemaps. By comparing it with the time of your previous crawl, you can download only the pages that have changed and avoid hitting the site with pointless requests. That turns a one-off scrape into efficient recurring monitoring: you crawl not the whole site, but only the delta. Bear in mind that lastmod isn't always filled in, and isn't always honest, so treat it as a hint rather than absolute truth.
Quick FAQ
Is a sitemap crawler the same as a scraper? No. A sitemap crawler only discovers which URLs exist. Extracting the data from each of those pages is a separate step — the job of an HTML scraper.
What if the site has no sitemap? Then you fall back to link-based crawling: start from the homepage, follow internal links, and enqueue new URLs as you find them. It's slower and heavier, which is exactly why a sitemap is worth checking for first.
Should I obey robots.txt? Yes — read it not just to find the Sitemap: line but to respect Disallow rules and crawl politely, with sensible delays.
Once the URL list is gathered, the actual data extraction from the pages begins — that's the domain of HTML parsing. If you need to crawl a large site regularly, with an index sitemap, gzip, and date filtering, and then deliver results in the format you want — say CSV or Excel — scraping.pro can set up the whole pipeline as a managed data-as-a-service feed, including scheduling and delivery.