A dictionary scraper that runs without a single exception and writes naïveté into the CSV where the page said naïveté has exactly one bad line in it, and it is not the XPath. It is lxml.html.fromstring(resp.text). Dictionaries are the worst possible place to make that mistake: they are wall-to-wall accented loanwords, IPA glyphs and non-Latin etymologies, so the corruption lands in every third row instead of showing up once and getting noticed. An earlier version of this walkthrough shipped that line. This one explains it, along with the class selector that silently drops half your senses and the pagination loop that never ends.
What the code here was checked against
Everything below was run on Python 3.11.15 with lxml 6.1.1 (released 18 May 2026, linked against libxml2 2.14.6), requests 2.34.2 (14 May 2026) and cssselect 1.5.0 (27 July 2026). Versions and dates come from each project's PyPI page and from lxml's own CHANGES.txt, read on 13 August 2026. One thing to note before you copy anything: requests 2.34 requires Python 3.10 or newer, so "modern Python 3" now has a floor.
pip install "requests>=2.34" "lxml>=6.1" cssselectcssselect is a separate package. Without it tree.cssselect() raises ImportError, which is a confusing five minutes if you assumed CSS support came in the box.
This is a worked job rather than a tour of the library; for the API surface itself, the companion lxml tutorial covers it in reference form.
Look for the API before you write a selector
Dictionaries are one of the few corners of the web where the data is genuinely available without parsing HTML, and the arithmetic usually favours the API. The free Dictionary API answered on 13 August 2026 with a JSON array whose objects carry word, phonetic, phonetics, meanings, license and sourceUrls; each entry in meanings carries partOfSpeech, definitions, synonyms and antonyms.
import requests
data = requests.get(
"https://api.dictionaryapi.dev/api/v2/entries/en/serendipity", timeout=15
).json()
for meaning in data[0]["meanings"]:
for sense in meaning["definitions"]:
print(meaning["partOfSpeech"], "|", sense["definition"])Two things about that response matter more than the convenience. The license object reads CC BY-SA 3.0, and sourceUrls points at Wiktionary. The data is not public domain and it is not yours: share-alike obligations travel with it into whatever you build. The project site publishes no rate limit at all and asks for donations to keep the server running, which is a polite way of saying the ceiling is social rather than technical.
Where paid dictionaries publish numbers, they publish them plainly. Read on 13 August 2026:
| Source | Terms |
|---|---|
| Wordnik | Free Basic $0 / mo at 100 calls/hour; Hobby $10 / mo at 1000 calls/hour; Pro $59 / mo at 20,000 calls/hour; Enterprise $149 / mo at 45,000 calls/hour |
| Merriam-Webster | Free keys, conditional on "Reference queries do not exceed 1000/day/reference work", "You do not use more than two reference works" and "Your website or application is non-commercial" |
| Oxford | Self-serve trial, with the note that "our Enterprise licenses are for commercial use only, and start from an annual fee of £5,000 per language" |
A hundred calls an hour is 2,400 words a day for nothing. If your job is a fixed word list rather than a full crawl, that finishes the project before you have chosen a selector. When a JSON API is available, take it, and keep lxml for the sites that hand out data only as HTML.
Fetching, and the one line that corrupts the output
Requests decides the encoding of resp.text from the HTTP headers, and its documentation says so in as many words: "When you make a request, Requests makes educated guesses about the encoding of the response based on the HTTP headers." The guess is not clever. In requests/utils.py the whole rule is four lines:
if "charset" in params:
return params["charset"].strip("'\"")
if "text" in content_type:
return "ISO-8859-1"A server that sends Content-Type: text/html with no charset parameter, which is most of them, gets decoded as Latin-1 no matter what the document's own <meta charset> says. Here is the whole failure, reproducible in one file against a local server:
import http.server, threading, requests, lxml.html
BODY = ('<html><head><meta charset="utf-8"></head><body>'
'<li class="sense">borrowed from French <i>naïveté</i></li>'
'</body></html>').encode("utf-8")
class H(http.server.BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header("Content-Type", "text/html") # no charset, like most sites
self.send_header("Content-Length", str(len(BODY)))
self.end_headers(); self.wfile.write(BODY)
def log_message(self, *a): pass
srv = http.server.HTTPServer(("127.0.0.1", 8731), H)
threading.Thread(target=srv.serve_forever, daemon=True).start()
r = requests.get("http://127.0.0.1:8731/")
print(r.encoding) # ISO-8859-1
print(lxml.html.fromstring(r.text).xpath("string(//li)")) # naïveté
print(lxml.html.fromstring(r.content).xpath("string(//li)")) # naïveté
srv.shutdown()Pass bytes, not text. resp.content hands lxml the undecoded body, libxml2 reads the <meta charset> declaration itself, and the accents survive. lxml's own documentation is blunt about it: "You should generally avoid converting XML/HTML data to unicode before passing it into the parsers. It is both slower and error prone." The slower part is minor, about five per cent on the pages measured for this article. The error-prone part is the whole ballgame. There is even a case where passing a string fails outright rather than quietly: a document carrying an XML declaration raises ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.
import requests
import lxml.html
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/141.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
def fetch(session, url):
resp = session.get(url, headers=HEADERS, timeout=(5, 20))
resp.raise_for_status()
return lxml.html.fromstring(resp.content) # bytes, not resp.textThe timeout is a tuple on purpose: five seconds to establish the connection, twenty to finish reading. A single scalar applies to both phases and gives you the worst of each. A bare requests.get() with no timeout at all waits forever, which is how overnight jobs end up still running at noon.
The old shortcut is now an error, not a bad habit. Tutorials written before 2025 call lxml.html.parse() on a URL directly. lxml 6.0.0 records that its binary wheels moved to libxml2 2.14.4 and that "this disables direct HTTP and FTP support for parsing from URLs"; libxml2's own NEWS then records for v2.15.0, 15 September 2025, that "The built-in HTTP client and support for LZMA compression were removed." On the build used here, etree.LIBXML_FEATURES contains html, xpath, regexp, zlib, iconv, catalog, xmlschema and schematron, and no http at all. Feeding it a URL produces OSError: ... Attempt to load network entity. Fetch with requests; there is no longer a second option.
A session that survives a long run
One Session reuses the TCP connection and the cookie jar across every page of a crawl. Bolt a retry policy to it and transient failures stop ending runs.
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
def make_session():
retry = Retry(
total=5,
backoff_factor=0.5, # 0.5s, 1s, 2s, 4s, 8s
status_forcelist=(429, 500, 502, 503, 504),
allowed_methods=frozenset({"GET", "HEAD"}),
respect_retry_after_header=True,
)
s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=8))
return srespect_retry_after_header is the one people leave off. A dictionary site that answers 429 usually tells you how long to wait; obeying it is the difference between a slow crawl and a banned one. Note that urllib3 caps its own backoff at 120 seconds, so total=5 cannot strand you for an hour.
XPath that survives the next redesign
Open the page in DevTools, right-click the element, choose Copy XPath, and you get something like this:
/html/body/div[2]/div/table/tbody/tr[3]/td[2]It is a starting point and nothing more. Two things are wrong with it.
tbody is still a trap in 2026, and now for a documented reason. Browsers insert a <tbody> into the live DOM even when it is absent from the source, so DevTools shows you a node that is not in the bytes lxml parsed. libxml2 2.14.0, released 27 March 2025, made the HTML tokenizer standards-compliant, but its release notes are explicit about the half that did not change: "The HTML tokenizer now conforms fully to HTML5. Several non-standard syntax warnings were removed. Note that HTML5 tree construction isn't implemented yet." Tokenizing is not tree building, and tbody insertion is a tree-building rule. Parsing <table><tr><td>alpha</td></tr></table> on lxml 6.1.1 and asking for //table/tbody/tr still returns zero nodes, while //table/tr returns one. Drop the /tbody step or replace it with //.
Absolute paths break on the next redesign. One inserted <div> shifts every index after it, and nothing raises: you get an empty list and a CSV full of blanks.
The bigger trap is the one the previous version of this article walked straight into. //li[@class='sense'] is an exact string comparison against the entire class attribute. Dictionary markup is full of elements carrying two or three classes, and every one of them is invisible to that expression:
html = ("<ul><li class='sense'>one</li>"
"<li class='sense hi-lite'>two</li>"
"<li class='nonsense'>three</li></ul>")
t = lxml.html.fromstring(html)
t.xpath("//li[@class='sense']") # 1 node - misses 'sense hi-lite'
t.xpath("//li[contains(@class,'sense')]") # 3 nodes - also matches 'nonsense'
t.cssselect("li.sense") # 2 nodes - correctOne expression under-matches, the other over-matches, and both fail quietly. The token-safe XPath is ugly enough that almost nobody writes it by hand:
t.xpath("//li[@class and contains(concat(' ', normalize-space(@class), ' '), ' sense ')]")That string is not invented for this article. It is literally what cssselect emits, and you can watch it happen:
from cssselect import HTMLTranslator
HTMLTranslator().css_to_xpath("li.sense")
# descendant-or-self::li[@class and contains(concat(' ', normalize-space(@class), ' '), ' sense ')]cssselect is not a second engine. Its own README describes it as a library "to parse CSS3 selectors and translate them to XPath 1.0 expressions", which then run on lxml. So for class matching, write CSS and let the translator produce the correct XPath. Keep hand-written XPath for what CSS cannot express: axes, text predicates, and position relative to a sibling heading. There is a fuller comparison in the piece on CSS selectors for web scraping.
Getting the text out without duplicating it
Here is the second bug from the previous version, kept intact so the fix is visible:
definition = sense.text_content().strip()
example = sense.xpath(".//span[@class='example']/text()")[0].strip().text_content() flattens everything inside the node, including the example. Run it on a sense whose markup nests the example inside the list item and you get:
definition = 'The occurrence of events by chance in a happy way.\n a fortunate stroke of serendipity'
example = 'a fortunate stroke of serendipity'The example is in both columns, the definition field is unusable for anything that compares strings, and no exception ever fires. Two ways out. Select the definition node precisely rather than taking the whole list item, or remove the example subtree before flattening:
def parse_sense(sense):
ex_nodes = sense.xpath(".//span[contains(concat(' ', normalize-space(@class), ' '), ' example ')]")
example = ex_nodes[0].text_content().strip() if ex_nodes else ""
for node in ex_nodes:
node.drop_tree() # detach, keeping the tail text in place
return sense.text_content().strip(), exampledrop_tree() is an lxml.html method with no equivalent in the plain ElementTree API, and it is the cleanest tool for this: it removes the element while preserving its tail text, so punctuation sitting after the example does not vanish with it. Three more idioms worth knowing:
string(.//span[@class='pos'])returns the joined text of the first match, or an empty string if nothing matched. NoIndexError, noifguard..textgives only the text before the first child tag,.text_content()gives everything flattened, and.itertext()gives the pieces separately so you can join them with a space instead of gluinga<b>b</b>cintoabc..//a/@hrefcollects attribute values. The leading.makes the expression relative to the current node; drop it inside a loop and every iteration silently matches the whole document.
Use .find() when you want one node. lxml's performance page explains why: "tree iteration can be substantially faster than XPath if your code short-circuits after the first couple of elements were found. The XPath engine will always return the complete result set." Measured on the test page below, tree.xpath("//li[@class='sense']")[0] costs 2.2 ms because the engine materialises all 2,400 matches first, while tree.find('.//li') returns before the timer can resolve it. Wrapping the XPath in a positional predicate does not help; (//li[@class='sense'])[1] still costs 1.5 ms.
Pagination that terminates
The obvious loop has a hole in it. If rel="next" on the last page points back at the current URL, or two pages point at each other, the crawler runs until you kill it and the output file grows without limit. Keep a set of visited URLs and a hard page cap:
from urllib.parse import urljoin
def iter_pages(session, start_url, max_pages=50):
url, seen = start_url, set()
while url and url not in seen and len(seen) < max_pages:
seen.add(url)
tree = fetch(session, url)
yield tree
nxt = tree.xpath("//a[@rel='next']/@href")
url = urljoin(url, nxt[0]) if nxt else None
polite_delay()urljoin from urllib.parse is the direct import; requests.compat.urljoin is the same function re-exported through a compatibility shim left over from the Python 2 era, and there is no reason to reach for it in new code. It resolves both forms correctly: urljoin("https://x.test/word/serendipity?page=1", "?page=2") gives https://x.test/word/serendipity?page=2, and a leading-slash path replaces the whole path.
import time, random
def polite_delay():
time.sleep(1.0 + random.uniform(0, 1.0))Where the URL pattern is predictable, generating ?page=1, ?page=2 and stopping on the first page that yields no rows is simpler and gives you resumability for free. Following links is for sites whose paging scheme you cannot guess.
Writing the rows out
Stream to disk instead of accumulating in memory, so a crash at page 8,000 leaves you 8,000 pages of data rather than nothing.
import csv
FIELDS = ["word", "part_of_speech", "sense", "definition", "example"]
def open_writer(path="definitions.csv"):
f = open(path, "w", newline="", encoding="utf-8")
w = csv.DictWriter(f, fieldnames=FIELDS)
w.writeheader()
return f, wnewline="" is not decoration. Without it, a definition containing a line break gets written with stray carriage returns on Windows and the file no longer round-trips. If the CSV is going to be opened in Excel rather than a script, write it with encoding="utf-8-sig" so the byte-order mark tells Excel not to mangle the accented characters you just worked to preserve. For JSON, ensure_ascii=False keeps naïveté readable in the file instead of storing it as escape sequences.
The whole thing
import csv, time, random
from urllib.parse import urljoin
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import lxml.html
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/141.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
FIELDS = ["word", "part_of_speech", "sense", "definition", "example"]
def make_session():
retry = Retry(total=5, backoff_factor=0.5,
status_forcelist=(429, 500, 502, 503, 504),
allowed_methods=frozenset({"GET", "HEAD"}),
respect_retry_after_header=True)
s = requests.Session()
s.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=8))
return s
def fetch(session, url):
resp = session.get(url, headers=HEADERS, timeout=(5, 20))
resp.raise_for_status()
return lxml.html.fromstring(resp.content)
def polite_delay():
time.sleep(1.0 + random.uniform(0, 1.0))
def parse_definitions(tree, word):
# Selectors below are written as CSS so cssselect emits token-safe XPath.
for block in tree.cssselect("div.pos-block"):
pos_node = block.cssselect("span.pos")
pos = pos_node[0].text_content().strip() if pos_node else ""
for i, sense in enumerate(block.cssselect("li.sense"), start=1):
ex_nodes = sense.cssselect("span.example")
example = ex_nodes[0].text_content().strip() if ex_nodes else ""
for node in ex_nodes:
node.drop_tree()
yield {"word": word, "part_of_speech": pos, "sense": i,
"definition": sense.text_content().strip(), "example": example}
def scrape_word(session, start_url, word, max_pages=50):
url, seen = start_url, set()
while url and url not in seen and len(seen) < max_pages:
seen.add(url)
tree = fetch(session, url)
yield from parse_definitions(tree, word)
nxt = tree.xpath("//a[@rel='next']/@href")
url = urljoin(url, nxt[0]) if nxt else None
polite_delay()
if __name__ == "__main__":
session = make_session()
n = 0
with open("definitions.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=FIELDS)
writer.writeheader()
for row in scrape_word(session, "https://example.com/define/serendipity",
"serendipity"):
writer.writerow(row)
n += 1
print(f"Saved {n} definitions")The class names in parse_definitions are placeholders, and they are placeholders deliberately. Hardcoding another site's class names into a tutorial is precisely what makes tutorials rot: the target redesigns, the article stays online, and readers copy selectors that have matched nothing for two years. Save one page with curl -o page.html <url>, work out the selectors against the saved copy in a shell, and only then point the loop at the live site. That habit also keeps you off the server while you experiment, which is the part the site owner cares about.
Better still, make the scraper notice on its own. Six lines of assertion turn a silent empty CSV into a loud failure:
def check_selectors(tree):
counts = {sel: len(tree.cssselect(sel))
for sel in ("div.pos-block", "li.sense", "span.pos")}
empty = [sel for sel, n in counts.items() if n == 0]
if empty:
raise RuntimeError(f"selectors matched nothing, page layout changed: {empty}")
return countsRun it on the first page of every job. A scraper that fails on day one of a redesign costs an afternoon; one that returns empty strings for three weeks costs the dataset.
What breaks at ten thousand pages
Memory is not where you think it is. lxml keeps the parsed tree in C memory allocated by libxml2, outside anything tracemalloc can see, so a memory profiler will report a scraper as lean while the process climbs. Measured here, a 532 KB HTML page becomes about 6.9 MiB of resident memory as a parsed tree, roughly thirteen times the source bytes; holding twenty of them pushed the process from 18 MiB to 155 MiB. Never keep trees in a list. Parse, extract, let the tree go out of scope, write the row.
Politeness has a price you can compute. A 1.0-to-2.0 second delay averages 1.5 seconds, so ten thousand pages is about four hours and ten minutes of sleeping before you count a single response. Add a 400 ms median response and you are at five hours. Halving the delay halves the wall clock and roughly doubles your odds of a rate limit, and that trade is the whole design decision. Anything larger wants concurrency, which is a different article: see async web scraping in Python.
Resumability is not optional at this size. Write a done-list of completed words alongside the CSV and skip them on restart. Without it, one failure at hour four costs you hour one.
The wall may not be technical. Cloudflare announced Content Independence Day on 1 July 2025, stating that the company "along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content." Reference sites were among the first to switch it on, and the symptom is a 403 that no header tinkering fixes. What sites permit varies more than the discourse suggests: Merriam-Webster's robots.txt, read on 13 August 2026, disallows seven narrow paths and none of them touch /dictionary/, and it publishes a sitemap index. Cambridge Dictionary, by contrast, returned 403 to every automated request made while researching this article. Read the robots file before you write the selector, and treat a blanket 403 as an answer rather than a puzzle. Sites that answer 403 to everything are where a managed extraction service earns its keep, because the cost moves from parsing to access. Recurring delivery of the same word list on a schedule is closer to data as a service than to a scraping script, and it is worth being honest about which of the two you are actually building.
Where lxml stops being the answer
XPath 1.0, and only 1.0. libxml2 implements XPath 1.0, so the functions people reach for out of habit are simply absent. All three of these raise XPathEvalError: Unregistered function on lxml 6.1.1:
tree.xpath("//p[matches(@class, 'sense')]") # XPath 2.0
tree.xpath("//p[ends-with(@class, 'ense')]") # XPath 2.0
tree.xpath("//p[lower-case(@class) = 'sense']") # XPath 2.0Case-insensitive matching in XPath 1.0 means translate() with two 26-character alphabets, which is why cssselect and post-processing in Python are usually the better answer.
No pseudo-elements. cssselect rejects span.def::text with ExpressionError: Pseudo-elements are not supported. If you came from Scrapy, that shortcut is a Scrapy extension, not CSS.
No JavaScript. lxml parses bytes. A dictionary that renders its senses client-side gives you an empty shell, and the fix is either the XHR the page itself calls or a browser engine. The tour of JavaScript-rendered pages covers both.
lxml.html.clean left the building. lxml 5.2.0, 30 March 2024, moved the HTML sanitiser into a separate project after "several (only if used) security issues in the past". If you sanitise scraped markup, the dependency is now lxml[html_clean] and not plain lxml. That module has a long CVE history; lxml 6.1.1 fixed a related bypass through the xlink:href attribute as recently as May 2026.
What "fastest parser" is actually worth
The claim that opened the earlier version of this article, that lxml is the fastest HTML parser in the Python ecosystem, does not survive contact with a stopwatch in 2026. Here is a synthetic dictionary page of 400 entries by 6 senses, 532,256 bytes, 2,400 sense nodes, parsed and fully extracted:
| Approach | CPU time | Relative |
|---|---|---|
| lxml, bytes in | 18 ms | 1.0 |
| lxml, str in | 18 ms | 1.05 |
| selectolax, lexbor backend | 18 ms | 1.0 |
| BeautifulSoup with the lxml backend | 330 ms | 19 |
| BeautifulSoup with html.parser | 450 ms | 27 |
Split by phase, parsing costs lxml 11 ms and lexbor 11 ms, while BeautifulSoup with lxml underneath costs 260 ms for the identical parse. Selecting 2,400 nodes from an already-built tree costs 4 ms via lxml XPath, 8 ms via cssselect, 1.5 ms via lexbor CSS and 54 ms via soup.select.
Read that table twice. lxml and lexbor are a tie, and lexbor is several times quicker at selection. The order-of-magnitude gap is not the parser at all: BeautifulSoup wraps lxml and is still nineteen times slower, because the cost is building a Python object for every node. Choosing lxml over BeautifulSoup is worth roughly twenty times; choosing lxml over selectolax is worth arguing about only if parsing is your bottleneck, and on a scraper sleeping 1.5 seconds between requests it never is. Pick lxml for full XPath and a twenty-year-old API, not for a speed crown it no longer holds outright.
Measured on one shared two-core VM with an Intel Xeon at 2.80 GHz, Python 3.11.15, lxml 6.1.1, selectolax 0.4.11, beautifulsoup4 4.15.0. Figures are minimum-of-fifteen CPU time, not wall clock, because the machine was too noisy for wall clock to reproduce; the ratios held across three runs. Your absolute numbers will differ. The ordering will not.
Wrapping up
The shape of the job never changes: fetch with a session and real headers, parse the bytes, select with CSS or token-safe XPath, drop the subtrees you do not want before flattening, follow pagination with a visited set, and stream rows to disk. Everything that goes wrong in practice goes wrong quietly, which is why the assertions matter more than the selectors do. For the library itself, its API surface, XSLT and the ElementTree compatibility layer, the companion lxml tutorial is the reference. For the wider stack, step back to the guide on web scraping with Python.