lxml Tutorial: Fast HTML and XML Parsing in Python

lxml is the fastest HTML and XML parsing library in the Python ecosystem. It's built on the C libraries libxml2 and libxslt, so it outpaces pure-Python parsers several times over, and it gives you full XPath — more powerful than CSS selectors. If you've hit a performance wall with BeautifulSoup on large volumes, or you need complex selections, Python lxml is the right choice. This lxml tutorial walks through the parts that matter for real scraping.

It goes deeper on a topic introduced in the overview Web scraping with Python, where lxml appears alongside other parsers. Here we cover it in detail.

Contents

1. Installation and lxml's two modules
2. Parsing HTML
3. XPath: the core of lxml's power
4. CSS selectors via cssselect
5. Extracting data: text, attributes, links
6. Character encoding in lxml
7. Large documents: iterparse
8. lxml vs BeautifulSoup
9. Pros and cons

1. Installation and lxml's two modules

pip install lxml

lxml has two key modules:

• lxml.html — for parsing HTML (forgives "dirty" markup, like a browser does).
• lxml.etree — for strict XML (see the separate article Parsing XML in Python).

For web scraping, 95% of the time you want lxml.html.

2. Parsing HTML

import requests
from lxml import html

resp = requests.get("https://example.com", timeout=10)
tree = html.fromstring(resp.content)   # pass bytes — lxml figures out the encoding

fromstring returns the root element of the tree. From there you apply XPath or CSS selectors. You can also parse from a file:

tree = html.parse("page.html").getroot()

Important: pass resp.content (bytes), not resp.text (a string) — then lxml reads the encoding declaration from the HTML itself. That removes most encoding headaches up front.

3. XPath: the core of lxml's power

XPath is a language for addressing nodes in the document tree, and this is where lxml comes into its own. If you're new to lxml xpath, these patterns cover most day-to-day scraping.

# text of all h2 headings with class "title"
titles = tree.xpath('//h2[@class="title"]/text()')

# the href attribute of every link
links = tree.xpath('//a/@href')

# text inside a specific block
price = tree.xpath('//div[@class="price"]/text()')[0]

Useful XPath constructs

# by partial class match (when class="title big featured")
tree.xpath('//h2[contains(@class, "title")]/text()')

# by element text
tree.xpath('//a[text()="Read more"]/@href')

# the n-th element (numbering starts at 1!)
tree.xpath('(//div[@class="item"])[3]')

# relative path from a found node
for card in tree.xpath('//div[@class="card"]'):
    name = card.xpath('.//h3/text()')      # dot = "from the current node"
    link = card.xpath('.//a/@href')

The leading dot (.//) in a relative XPath is critical: without it the search runs from the document root, not from the current card. It's the single most common beginner mistake.

XPath axes

XPath can walk the tree in any direction — something CSS can't do:

# the element's parent
tree.xpath('//span[@class="price"]/parent::div')

# the next sibling at the same level
tree.xpath('//h2/following-sibling::p[1]/text()')

# an ancestor with a specific class
tree.xpath('//a[@id="buy"]/ancestor::div[@class="product"]')

4. CSS selectors via cssselect

If XPath feels clunky, lxml supports CSS selectors (you need the cssselect package):

pip install cssselect

# .cssselect() returns a list of elements
cards = tree.cssselect("div.product-card")
title = tree.cssselect("h1.title")[0].text_content()
links = [a.get("href") for a in tree.cssselect("a.product-link")]

CSS reads more naturally for those coming from front-end work. Under the hood, cssselect translates CSS into XPath, so the speed is preserved. For complex selections (by text, by ancestor) you'll still need XPath.

5. Extracting data: text, attributes, links

el = tree.cssselect(".product")[0]

el.text_content()          # all text inside, including nested tags
el.get("href")             # an attribute value
el.attrib                  # dict of all attributes
el.tag                     # the tag name
el.text                    # only the node's direct text (no nested tags)

The difference between .text and .text_content() matters:

# <p>Hello, <b>world</b>!</p>
p.text              # "Hello, "         (only up to the nested tag)
p.text_content()    # "Hello, world!"   (all text, recursively)

Turn relative links into absolute ones:

tree.make_links_absolute("https://example.com")
links = tree.xpath('//a/@href')   # all links are now absolute

make_links_absolute saves you the manual URL joining — handy when crawling a site.

6. Character encoding in lxml

lxml handles non-ASCII text well, as long as you give it bytes rather than an already-decoded string:

# CORRECT: bytes — lxml reads the encoding from <meta charset>
tree = html.fromstring(resp.content)

# RISKY: a string already decoded by requests (possibly incorrectly)
tree = html.fromstring(resp.text)

If you need to force the encoding (say, the server lies in its headers):

from lxml import html

parser = html.HTMLParser(encoding="windows-1252")
tree = html.fromstring(resp.content, parser=parser)

The full theory of encoding problems is covered in Web scraping with Python.

7. Large documents: iterparse

When a document is huge (an XML feed of hundreds of megabytes), loading it whole into memory is wasteful. iterparse reads it as a stream, processing elements as they appear and freeing memory:

from lxml import etree

for event, element in etree.iterparse("huge.xml", tag="item"):
    title = element.findtext("title")
    process(title)
    element.clear()                       # free memory
    while element.getprevious() is not None:
        del element.getparent()[0]        # drop processed nodes

This trick (read + clear()) lets you parse files that don't fit in RAM. More on streaming XML parsing in Parsing XML in Python.

8. lxml vs BeautifulSoup

In practice they're often combined: BeautifulSoup can use lxml as its backend (BeautifulSoup(html, "lxml")) — you get the friendly bs4 API and lxml's speed. If you need XPath or streaming parsing of gigabyte files, use lxml directly. A comparison of different parsers is in Web scraping with Python.

9. Pros and cons of lxml

Pros:

• The fastest parser in the Python ecosystem (C core).
• Full XPath with axes (parent, sibling, ancestor) — out of reach for CSS.
• CSS support via cssselect — the best of both worlds.
• iterparse for streaming huge documents.
• Lean memory usage.

Cons:

• A steeper learning curve, especially XPath.
• The API is stricter and less forgiving than BeautifulSoup's.
• Error messages can be cryptic.
• Installation occasionally needs system libraries (on some OSes) — though modern wheels have made this rare.

Bottom line: lxml is the choice for performance and complex selections. For one-off tasks and maximum readability, BeautifulSoup is simpler; for huge volumes with CSS selections, take a look at selectolax too. It's precisely because of its speed that lxml is most often paired with asynchronous scraping: when hundreds of pages download at once, a slow parser becomes the bottleneck, and lxml's C engine removes it.

If you'd rather have clean, structured data delivered than tune parsers and crawlers yourself, scraping.pro can build and run the extraction for you as a data extraction service.