XPath for Web Scraping: A Practical Guide

XPath (the XML Path Language) is a compact, expression-based language for locating and selecting parts of a structured document. Although it was originally designed for XML, it is one of the most useful tools you can have when extracting data from web pages, because an HTML page — once it has been parsed into a tree — can be queried in exactly the same way as an XML document.

This guide rebuilds the fundamentals of XPath from the ground up and then shows how those fundamentals are applied in real-world screen scraping: which libraries turn messy HTML into a queryable tree, and how to discover working XPath expressions directly inside your browser instead of writing them by hand.

The official, normative reference for the classic version of the language is the W3C XPath 1.0 recommendation: https://www.w3.org/TR/xpath-10/. A more modern and far more powerful revision, XPath 3.1, is documented at https://www.w3.org/TR/xpath-31/. For a friendly, example-driven overview, Mozilla's documentation is excellent: https://developer.mozilla.org/en-US/docs/Web/XPath.

1. What XPath Actually Is

XPath is not a programming language in the usual sense — you do not write loops or define variables with it. Instead, you write expressions that describe where something lives inside a document tree. When an XPath expression is evaluated, the result is typically a node-set: a collection of zero or more nodes that match the description. (Expressions can also evaluate to a string, a number, or a boolean, depending on what you ask for.)

The language was conceived to be embedded inside other technologies rather than used entirely on its own. It is the addressing mechanism behind:

• XSLT — XSL Transformations, which transform XML into other formats (https://www.w3.org/TR/xslt-30/);
• XPointer — a way of pointing at fragments of an XML resource;
• XQuery — a full query language for XML data (https://www.w3.org/TR/xquery-31/).

For our purposes the key insight is simpler: any document that the browser or a parser has loaded into a DOM (Document Object Model) tree can be navigated with XPath. That includes XML, XHTML, and ordinary HTML.

2. The Document as a Tree of Nodes

XPath treats every document as a hierarchy — a tree — built out of nodes. The XPath 1.0 data model recognises seven kinds of node:

The topmost element of a document is called the root element, and it lives directly beneath the (invisible) document node. Some nodes — for instance the value held inside a text node — are atomic values: they have neither children nor a parent of their own in the way elements do.

A worked example document

Throughout this guide we will refer back to one small XML document:

<?xml version="1.0" encoding="UTF-8"?>
<catalog>
  <item new="true">
    <title lang="en">DVD Music</title>
    <author>K. A. Bred</author>
    <year>2012</year>
    <price currency="USD">29.99</price>
  </item>
</catalog>

In this document:

• <catalog> is the root element node.
• <author>K. A. Bred</author> is an element node.
• lang="en" and new="true" are attribute nodes.
• K. A. Bred, 2012, and 29.99 are text nodes.

Family relationships between nodes

XPath borrows the vocabulary of family trees to describe how nodes relate to one another, and most navigation axes are built around these relationships.

• Parent — every element and attribute has exactly one parent. Here, <item> is the parent of <title>, <author>, <year>, and <price>.
• Children — an element may have zero, one, or many children. The four elements just listed are all children of <item>.
• Siblings — nodes that share the same parent. <title>, <author>, <year>, and <price> are siblings of one another.
• Ancestors — a node's parent, its parent's parent, and so on up to the root. The ancestors of <title> are <item> and <catalog>.
• Descendants — a node's children, its children's children, and so on. The descendants of <catalog> are <item>, <title>, <author>, <year>, and <price>.

Keeping these relationships clear in your head is the single most useful skill for writing correct expressions, because almost every path you write walks up, down, or sideways along one of them.

3. Location Paths: How You Navigate

The most common kind of XPath expression is a location path, which selects a set of nodes relative to a starting point. XPath actually defines two notations: a verbose unabbreviated syntax and a compact abbreviated syntax. In practice almost everyone uses the abbreviated form, so that is the one we focus on here; the unabbreviated form spells out the axis names explicitly and is mostly of interest when you need axes that have no shorthand.

There are two flavours of location path.

Relative location paths

A relative path is a series of location steps joined by /, evaluated starting from the current context node:

catalog/item/author

Because there is no leading slash, this path is interpreted relative to wherever evaluation currently stands.

Absolute location paths

An absolute path begins with /, which anchors it at the document root regardless of the context node:

/catalog/item[price < 20.0]

This selects every <item> (whose price is below 20) that is a child of <catalog> at the root. Since evaluation always starts from the root, the context node is irrelevant for absolute paths.

Anatomy of a single location step

Each step in a path is itself made of up to three parts:

1. an axis, which defines the tree relationship between the context node and the nodes being selected (child, descendant, parent, attribute, and so on);
2. a node test, which names or types the nodes to keep within that axis;
3. zero or more predicates, square-bracketed conditions that further filter the result.

In catalog/item/author, each segment is a step: catalog selects nodes relative to the context, item narrows to nodes inside that result, and author narrows again.

4. Selecting Attributes

To reach an attribute rather than an element, prefix its name with @. For example, this path finds every <price> element whose currency attribute equals USD:

/catalog/item/price[@currency='USD']

And this returns the value of the lang attribute wherever it appears:

//@lang

5. Context: Where Evaluation Happens

XPath never evaluates an expression in a vacuum — it always evaluates relative to a context. The host technology (XSLT, XPointer, a scraping library, the browser console) supplies that context, which includes a context node, a context size, a context position, and assorted other data.

For scraping, the context node is what matters most: it is the node from which a relative path begins walking the tree, and the part of the tree your data extraction is anchored to.

6. Predicates: Filtering the Result

A predicate is a boolean test wrapped in square brackets, [ ]. Each candidate node in the current node-set is checked against the predicate, and is kept only if the test evaluates to true. Predicates are how you turn "all the items" into "exactly the item I want."

/catalog/item[price > 300]
/catalog/item/price[@currency='USD']

The first keeps only items priced above 300; the second keeps only prices in US dollars.

Predicates support the relational operators >, <, >=, <=, =, and !=, as well as the boolean operators and and or. One practical wrinkle: inside XML or XSLT, the raw < and > characters are reserved, so you often have to escape them as < and >:

item[number(minutes) <= 60]

Here < simply stands in for <.

7. Syntax Cheat Sheet

Common shorthands in action

@ — attributes. Combine attribute tests with and/or:

item[@new='false' or price/@currency='USD']

* — every child element. The first path returns all element children of each <item>; the second returns every element in the document:

/catalog/item/*
//*

[n] — positional selection. Brackets with a number pick a node by its position. This selects the second <item>:

catalog/item[2]

// — descendants at any depth. The first selects every <item> anywhere; the second selects only <item> elements that have a <catalog> parent:

//item
//catalog/item

. — the current node. This finds every <item> that is a descendant of the context node:

.//item

.. — the parent. Starting from <author> nodes, this steps back up to their <item> parents:

/catalog/item/author/..

8. Operators

The union operator

The pipe | merges two node-sets into one. This collects every <author> and every <year> element together:

//author | //year

Numeric operators

XPath provides + (addition), - (subtraction), * (multiplication), div (division), and mod (the remainder of a truncating division). Note that division uses the keyword div, not /, because / is already the path separator.

Strings and quoting

String literals are wrapped in single or double quotes. When your XPath itself lives inside an XML or XSL document and needs to contain a quotation mark, you have two options:

1. Escape the inner quote with an entity — ' for ' or " for ".
2. Alternate the quote style — wrap the expression in double quotes if it contains single quotes, and vice versa:

select = "item[@private='false' or price/@currency='USD']"

Here the outer double quotes let the inner single quotes pass through untouched.

9. The Core Function Library

XPath ships with a built-in set of functions — the core function library — that you call from inside predicates and expressions. Each function has a name, a return type (never void), and a defined set of arguments. XSLT extends this library further. The most useful functions fall into four families.

Node-set functions

These report on, or operate over, a set of nodes.

• last() — returns the context size, i.e. how many nodes are in the current context. (This is the count, not the last node itself.)
• position() — returns the context position of the current node within its set. A common idiom, position() = last(), tests whether you are on the final node.
• count(node-set) — returns how many nodes the argument node-set contains.
• id(object) — returns the element whose unique ID-typed attribute matches the argument (the ID type must be declared, e.g. in a DTD).

String functions

These manipulate text.

• string(object?) — converts its argument (a node-set, number, boolean, etc.) to a string. A node-set becomes the string value of its first node in document order. This is not meant for number formatting for display — use format-number() or xsl:number for that.
• concat(s1, s2, ...) — joins two or more strings. concat("First ", "mile of ", "Orient Express") yields First mile of Orient Express.
• starts-with(haystack, needle) — true when the first string begins with the second. starts-with("Miles Smiles album, CD", "Miles") is true.
• contains(haystack, needle) — true when the first string contains the second anywhere. contains("Miles Smiles album, CD", "album") is true.
• normalize-space(string?) — trims leading and trailing whitespace and collapses internal runs of whitespace into single spaces. With no argument it operates on the string value of the context node. This one is invaluable in scraping, where stray indentation and line breaks are everywhere.

Other string functions include substring(), substring-before(), substring-after(), string-length(), and translate(). See section 4.2 of the XPath 1.0 specification for the full list: https://www.w3.org/TR/xpath-10/#section-String-Functions.

Boolean functions

These produce or coerce true/false values.

• boolean(object) — coerces its argument to a boolean. A number is true unless it is zero or NaN; a string or node-set is true unless it is empty.
• not(boolean) — inverts its argument.
• true() and false() — return the literal boolean values. They exist because bare true and false would otherwise be read as ordinary strings.
• lang(string) — true when the language of the context node (from xml:lang) matches, or is a sub-language of, the argument. If no xml:lang is in scope, lang("en") returns false.

Number functions

These always return numbers.

• number(object?) — converts its argument to a number: boolean true becomes 1 and false becomes 0; a string is parsed as a number; a node-set is first turned into a string, then into a number. For example, selecting items whose time is at most 60:

item[number(time) <= 60]

• sum(node-set) — adds up the numeric values of every node in the set (after applying number() to each).
• floor(x) — the largest integer not greater than x. floor(1.7) is 1.
• ceiling(x) — the smallest integer not less than x. ceiling(3.65) is 4.
• round(x) — the integer closest to x. round(6.51) is 7.

The full normative catalogue of functions, including the additional ones defined in later versions, lives in the W3C specifications: https://www.w3.org/TR/xpath-functions-31/.

10. Quick Reference: Path Examples

11. XPath in Screen Scraping

This is where XPath earns its keep. When you scrape the web you are usually working with live HTML pages (plus the occasional RSS feed), and that HTML is frequently malformed — unclosed tags, missing quotes, stray markup. A strict XML processor will refuse to parse such pages outright.

The standard fix is to run the raw HTML through a tolerant parser that cleans it up and builds a proper DOM tree. Once the page is a well-formed tree, XPath (and by extension XSLT and XQuery) can query it normally. For a broad overview of the tooling landscape around this — scraping software, services, and plugins — see this round-up: /web-scraping-tools/.

Libraries that give you XPath over HTML

Depending on your stack, several mature, well-documented libraries can parse messy HTML and expose XPath querying:

• HTML Agility Pack (.NET / C#) — a free, open-source HTML parser that builds a read/write DOM and supports XPath and XSLT directly; it is deliberately forgiving of real-world broken markup. Project site: https://html-agility-pack.net/; source and issues on GitHub: https://github.com/zzzprojects/html-agility-pack; package on NuGet: https://www.nuget.org/packages/HtmlAgilityPack/. A minimal example loads a page and selects with an XPath:

csharp var web = new HtmlWeb(); var doc = web.Load("https://example.com/"); var nodes = doc.DocumentNode.SelectNodes("//div[@class='price']");

• lxml (Python) — a fast C-backed library wrapping libxml2/libxslt, with first-class XPath support and an HTML parser that copes with broken input. Documentation: https://lxml.de/, specifically the XPath section at https://lxml.de/xpathxslt.html.

• Parsel / Scrapy selectors (Python) — Scrapy's selector layer, also usable standalone as Parsel, lets you query responses with XPath (or CSS). Scrapy docs: https://docs.scrapy.org/en/latest/topics/selectors.html; Parsel docs: https://parsel.readthedocs.io/.

• jsoup (Java), Nokogiri (Ruby), and Cheerio (Node.js) are the equivalents in their respective ecosystems, with varying degrees of XPath versus CSS-selector support.

If you are working in .NET, the HTML Agility Pack documentation also hosts a series of articles and examples that walk through common scraping patterns, which is a good next step once you have the basics down.

12. Finding XPath Without Writing It by Hand

You rarely have to compose a full path from scratch. Modern browsers, and a handful of extensions, can show you a working XPath for any element on a page — and let you test expressions live before you put them in code.

Built-in browser DevTools

Every major browser ships with developer tools that already understand XPath:

• Inspect and copy. Right-click an element on the page and choose Inspect to jump to it in the Elements panel. Right-click the highlighted node and use Copy → Copy XPath (a short, relative-ish path) or Copy → Copy full XPath (an absolute path from the root). The relative version is usually more robust against page changes.
• Test in the console. The Chrome/Edge DevTools console exposes a helper, $x(), that runs an XPath against the current page and returns the matching nodes. For example, typing $x("//h1") lists every <h1>. This is the fastest way to confirm an expression matches what you expect before committing it to a scraper.

Inspecting the DOM this way also reveals the actual rendered structure of the page, which often differs from the raw HTML you receive over the wire — a crucial thing to verify when a path that "should" work returns nothing.

Browser extensions for generating and verifying selectors

Several extensions specialise in building and validating selectors interactively:

• SelectorsHub — a widely used, free plugin (with a paid Pro tier) that auto-generates and verifies XPath, CSS, and other selector types as you inspect elements, including tricky cases like Shadow DOM, iframes, and SVG. It appears as a tab inside DevTools. Site: https://selectorshub.com/; available for Chrome, Firefox, Edge, and Opera.
• XPath Finder and similar lightweight extensions highlight an element and display its XPath for one-click copying — handy when an element lacks a stable id.

A historical note: ChroPath, long a popular choice, has been deprecated and removed from the Chrome Web Store; SelectorsHub is effectively its successor, and the built-in DevTools Copy XPath remains a dependable fallback.

13. Further Reading and Official References

• XPath 1.0 specification (the classic, still ubiquitous version): https://www.w3.org/TR/xpath-10/
• XPath 3.1 specification (current): https://www.w3.org/TR/xpath-31/
• XPath and XQuery functions and operators 3.1: https://www.w3.org/TR/xpath-functions-31/
• MDN — XPath: https://developer.mozilla.org/en-US/docs/Web/XPath
• XSLT 3.0: https://www.w3.org/TR/xslt-30/
• HTML Agility Pack: https://html-agility-pack.net/ · https://github.com/zzzprojects/html-agility-pack
• lxml (Python): https://lxml.de/
• Scrapy selectors / Parsel: https://docs.scrapy.org/en/latest/topics/selectors.html · https://parsel.readthedocs.io/
• SelectorsHub: https://selectorshub.com/
• Scraping software, services, and plugins — overview: /web-scraping-tools/