Python lxml Web Scraping: Online Dictionary Example

lxml is the fastest HTML parser in the Python ecosystem, and paired with requests it makes a lean, powerful scraping stack. This tutorial walks through a complete, real-world job end to end: pulling word definitions from an online dictionary, using XPath to target the right elements, following pagination, and exporting clean data. Everything here is modern Python 3 — the original version of this walkthrough was written for Python 2, and the code below is fully updated. For a reference-style tour of the library itself, see the companion lxml tutorial; here we focus on a worked lxml web scraping example.

Why lxml plus requests

• Speed. lxml is built on the C libraries libxml2 and libxslt, so it parses large pages several times faster than pure-Python alternatives.
• Full XPath. XPath is more expressive than CSS selectors for the messy, nested tables dictionaries love — you can select by position, by text content, and by relationship between nodes.
• Forgiving HTML handling. lxml.html tolerates broken, real-world markup the way a browser does.

Install what you need:

pip install requests lxml

Step 1: Fetch the page

The old approach called lxml.html.parse() directly on a URL. Don't do that anymore — you want control over headers, timeouts, retries, and encoding. Fetch with requests, then parse the response text. Sending a real User-Agent alone prevents a surprising number of blocks.

import requests
import lxml.html

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/125.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch(url):
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    return lxml.html.fromstring(resp.text)

lxml.html.fromstring() returns an HtmlElement you can query with XPath and CSS.

Step 2: Find the right XPath

The single most important habit in XPath Python scraping: derive selectors from the page, and verify them — never trust a tool's auto-generated path blindly.

Open the target page in your browser's DevTools (F12), right-click the element, and choose "Copy > Copy XPath." That gives you a starting point, but auto-generated paths are usually brittle absolute chains like:

/html/body/div[2]/div/table/tbody/tr[3]/td[2]

Two problems with those:

1. tbody is a trap. Browsers insert a <tbody> into the live DOM even when it isn't in the raw HTML. lxml parses the HTML as delivered, so an XPath containing /tbody often matches nothing. Drop it, or use // to skip levels: //table//tr.
2. Absolute paths break constantly. One extra <div> and every index shifts.

Write relative, attribute-based XPath instead. Anchor on stable IDs, classes, or data-* attributes:

# Fragile (auto-generated)
tree.xpath("/html/body/div[2]/div/table/tbody/tr")

# Robust (attribute-anchored, tbody-free)
tree.xpath("//div[@id='definitions']//li[@class='sense']")

Test selectors interactively in a Python shell against a saved copy of the page until they return exactly the nodes you want.

Step 3: Extract the data

Say each definition sits in a list item, with the part of speech in a nearby heading. Loop the matched nodes and pull text and attributes. Use .text_content() to get all text inside a node (including nested tags), and always .strip() the result.

def parse_definitions(tree, word):
    rows = []
    # Part of speech blocks — adjust selectors to the real page
    for pos_block in tree.xpath("//div[@class='pos-block']"):
        pos = pos_block.xpath("string(.//span[@class='pos'])").strip()

        for i, sense in enumerate(pos_block.xpath(".//li[@class='sense']"), start=1):
            definition = sense.text_content().strip()
            example_nodes = sense.xpath(".//span[@class='example']/text()")
            example = example_nodes[0].strip() if example_nodes else ""

            rows.append({
                "word": word,
                "part_of_speech": pos,
                "sense": i,
                "definition": definition,
                "example": example,
            })
    return rows

A few XPath idioms worth knowing:

• string(.//span[@class='pos']) returns the joined text of the first match as a string — cleaner than indexing a list.
• .text_content() flattens nested markup (bold, links) into plain text.
• .//a/@href collects attribute values, e.g. links to related words.
• Leading . makes the XPath relative to the current node — essential inside a loop, or you'll accidentally match the whole document.

Step 4: Handle pagination

Long entries or "related words" lists often span pages. Two common patterns:

Follow the "next" link until it disappears:

def scrape_word(start_url, word):
    all_rows = []
    url = start_url
    while url:
        tree = fetch(url)
        all_rows.extend(parse_definitions(tree, word))

        next_href = tree.xpath("//a[@rel='next']/@href")
        url = requests.compat.urljoin(url, next_href[0]) if next_href else None

        polite_delay()
    return all_rows

Or build page URLs directly when the pattern is predictable (?page=1, ?page=2, …), stopping when a page returns no rows.

Always resolve relative links with urljoin so /word/next?page=2 becomes a full URL, and pace your requests so you don't hammer the server:

import time, random

def polite_delay():
    time.sleep(1.0 + random.uniform(0, 1.0))

Step 5: Export clean data

Write results to CSV (for spreadsheets) or JSON (for pipelines). CSV with DictWriter:

import csv

def save_csv(rows, path="definitions.csv"):
    if not rows:
        return
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys())
        writer.writeheader()
        writer.writerows(rows)

JSON is a one-liner:

import json

with open("definitions.json", "w", encoding="utf-8") as f:
    json.dump(rows, f, ensure_ascii=False, indent=2)

Full script

import csv, time, random
import requests
import lxml.html

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                  "AppleWebKit/537.36 (KHTML, like Gecko) "
                  "Chrome/125.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch(url):
    resp = requests.get(url, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    return lxml.html.fromstring(resp.text)

def polite_delay():
    time.sleep(1.0 + random.uniform(0, 1.0))

def parse_definitions(tree, word):
    rows = []
    for pos_block in tree.xpath("//div[@class='pos-block']"):
        pos = pos_block.xpath("string(.//span[@class='pos'])").strip()
        for i, sense in enumerate(pos_block.xpath(".//li[@class='sense']"), start=1):
            rows.append({
                "word": word,
                "part_of_speech": pos,
                "sense": i,
                "definition": sense.text_content().strip(),
            })
    return rows

def scrape_word(base, word):
    url = f"{base}/define/{word}"
    all_rows = []
    while url:
        tree = fetch(url)
        all_rows.extend(parse_definitions(tree, word))
        nxt = tree.xpath("//a[@rel='next']/@href")
        url = requests.compat.urljoin(url, nxt[0]) if nxt else None
        polite_delay()
    return all_rows

if __name__ == "__main__":
    rows = scrape_word("https://dictionary.example.com", "serendipity")
    with open("definitions.csv", "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=["word", "part_of_speech", "sense", "definition"])
        w.writeheader()
        w.writerows(rows)
    print(f"Saved {len(rows)} definitions")

The selectors above are illustrative placeholders — swap in the ones you confirm against your actual target in DevTools. The structure (fetch → parse with XPath → paginate → export) is what carries over to any site.

A modern shortcut: check for an API first

Before scraping any dictionary, look for an official API — it's faster, cleaner, and more stable than parsing HTML. Free options like the open Dictionary API return JSON directly:

import requests
data = requests.get("https://api.dictionaryapi.dev/api/v2/entries/en/serendipity").json()
print(data[0]["meanings"][0]["definitions"][0]["definition"])

When a JSON API is available, prefer it. Reserve lxml scraping for sites that expose data only as HTML. Either way, respect each site's terms of service and rate limits.

Wrapping up

requests plus lxml is a fast, dependable combination for HTML scraping in Python 3. The workflow never changes: fetch with proper headers, write robust attribute-based XPath (mind the tbody trap), loop through pagination, and export to CSV or JSON. Master that loop and you can scrape almost any structured page — dictionaries, catalogs, or listings. For a deeper look at XPath, CSS selectors, and performance, continue with the full lxml tutorial, or step back to the broader guide on web scraping with Python.

Need this data delivered on a schedule instead of maintaining a scraper yourself? scraping.pro can build and run it for you as a done-for-you data extraction service.