XPath for Web Scraping: A Practical Guide

XPath 1.0 became a W3C Recommendation on 16 November 1999, and it is still the version your scraper runs. The browser console evaluates 1.0. Selenium hands the expression to the browser, so it gets 1.0. lxml, Parsel, Scrapy, jsoup, Nokogiri and the HTML Agility Pack all speak 1.0. The specification meanwhile moved to 3.1 in 2017, and a 4.0 draft is being edited as you read this. Almost none of that reaches the line of code you write to pull a price off a product page.

That gap explains a large share of XPath pain. You paste matches(@href, "product") from a cheat sheet and lxml answers XPathEvalError: Unregistered function. You copy a path out of Chrome DevTools and it returns an empty list against the same page in Python. Neither failure is a syntax error. Both are questions about which tree, and which version of the language, you are actually talking to.

This guide covers XPath as it exists inside a scraper: the data model, the thirteen axes, the comparison rules that drop rows without saying so, and the parsers that build the tree you query. Every expression below was run on 13 August 2026 against lxml 6.1.1 with libxml2 2.14.6 on Python 3.11, and the outputs quoted are what came back. Specification dates come from the W3C pages themselves; library versions come from PyPI, NuGet, RubyGems and each project's own release notes, read on the same day.

1. Which XPath You Are Actually Writing

The frozen version. XPath 1.0 is not merely old, it is closed. W3C attached a note to the Recommendation in October 2016: "Although XPath 1.0 remains widely used, and is referenced normatively from other W3C specifications, readers are advised that later versions exist, and that no further maintenance (including correction of reported errors) is planned for this document." Whatever is surprising about 1.0 will stay surprising. The upside is stability. The semantics in section 6 of this guide have not shifted under anyone's feet in twenty-six years.

XPath 2.0 and 3.0 came and went without reaching a single browser. XPath 3.1 is the current Recommendation, dated 21 March 2017, and it adds maps, arrays and the => arrow operator, largely to make JSON tolerable. A 4.0 revision is being drafted by the QT4CG community group rather than by W3C; the copy we read on 13 August 2026 carried the date 11 August 2026 and describes itself as "work in progress and should not be considered either stable or complete." Nothing implements it yet.

What matters day to day is the number your own tooling speaks:

What that costs you. matches(), lower-case(), tokenize(), ends-with() and string-join() all arrived in 2.0 or later. Try any of them in lxml and you get XPathEvalError: Unregistered function, which is the good outcome: a loud failure rather than an empty result you mistake for a missing element. Section 9 covers the two practical ways to get those functions anyway.

Assume 1.0. Prove otherwise before you depend on anything else.

2. The Tree You Query Is Not the HTML You Fetched

XPath never queries HTML. It queries a tree some parser built out of HTML, and two parsers routinely build different trees from identical bytes.

The tbody you never wrote. The HTML Standard permits the tag to be left out: "A tbody element's start tag may be omitted if the first thing inside the tbody element is a tr element, and if the element is not immediately preceded by a tbody, thead, or tfoot element whose end tag has been omitted." Omitted from the markup, present in the tree. Every parser implementing the WHATWG algorithm inserts that element: browsers do, html5lib does, and so does Nokogiri in HTML5 mode on gumbo. libxml2, the engine under lxml, does not.

Here is the whole problem in one fragment, parsed both ways on 13 August 2026:

<table id="t"><tr><td class="k">Price</td><td class="v">29.99</td></tr></table>

That table is why a path copied out of Chrome fails in your scraper. Chrome builds its tree to spec, so Copy full XPath on a table cell hands you something shaped like /html/body/div[2]/table/tbody/tr[3]/td[2]. The tbody step is real in the browser and absent in the tree lxml built. Your expression is valid, the page is unchanged, the answer is an empty list. It works in reverse too: a path you tested successfully in Python can return nothing in the console, for the same reason with the sign flipped.

The safe form sidesteps the argument entirely:

//table[@id="t"]//tr/td[2]

Buying the browser's tree, and what it costs. You can have the conformant tree in Python, because html5lib parses straight into an lxml tree and everything downstream stays the same. It is not free. On a synthetic page of 371,659 bytes holding 2,000 product rows, lxml.html.fromstring took 10.2 ms at best and 13.6 ms at the median. The same bytes through html5lib.parse with the lxml treebuilder took 773 ms at best, 825 ms at the median. That is roughly seventy-five times the parse cost.

Push it through a crawl and the number stops being academic. Ten thousand pages cost about 102 seconds of parsing with libxml2 and about 2 hours 9 minutes with html5lib, before a single expression is evaluated. The usual answer is to parse fast and write paths that do not care about implied elements, reserving the conformant parser for the handful of sites where table structure genuinely matters.

Measured on one container, single-threaded, Python 3.11 with lxml 6.1.1 and html5lib 1.1: best of fifteen runs for lxml, best of five for html5lib. Your absolute numbers will differ. The ratio will not.

Element names are case-folded, your expression is not. An HTML parser lowercases tag and attribute names while XPath name tests stay case sensitive. Parse <BODY><DIV CLASS='Card'> with lxml and //div returns one node while //DIV returns zero, and @class matches where @CLASS finds nothing. Attribute values are untouched, so @class='Card' still needs its capital C. In XML none of this folding happens, and <Item> requires //Item exactly.

Whitespace is data. Every newline and indent between two tags becomes a text node. That is why text() on a tidy, pretty-printed page returns a list of blank strings before it returns anything useful, and why normalize-space() shows up in almost every real expression. Our guide to HTML parsers covers the tree-building side in more depth.

3. The Node Types, and What the Tutorials Get Wrong

XPath models a document as a tree of nodes. The 1999 Recommendation recognises exactly seven kinds:

The topmost element is the root element, and it sits directly beneath the invisible root node. Attributes and namespaces hang off elements without being their children, which is why /catalog/item/* never returns an attribute.

Atomic values are not part of XPath 1.0. An earlier version of this guide claimed that the value inside a text node is an atomic value, a node with neither children nor a parent. That sentence is wrong twice, and it is borrowed: W3Schools still states, verbatim, "Atomic values are nodes with no children or parent." XPath 1.0 has seven node types and atomic values are not among them; they arrive with the XQuery and XPath Data Model alongside 2.0. And a text node certainly has a parent, which is precisely what .. and the parent:: axis walk up to. If you have been carrying that line around, drop it.

A worked example document

Every result quoted in this guide came from running expressions against this document:

<?xml version="1.0" encoding="UTF-8"?>
<catalog>
  <item new="true">
    <title lang="en">DVD Music</title>
    <author>K. A. Bred</author>
    <year>2012</year>
    <price currency="USD">29.99</price>
    <price currency="EUR">410.00</price>
  </item>
  <item new="false">
    <title lang="fr">Livre</title>
    <author>M. Dupont</author>
    <year>2019</year>
    <price currency="USD">12.50</price>
  </item>
</catalog>

Two items, three prices, and one item carrying two of them. That last detail looks artificial and is not: multi-currency prices, a sale price beside a list price, two phone numbers in one contact block. Section 6 shows what it does to a comparison.

Family relationships

• Parent: every element and attribute has exactly one. Here <item> is the parent of <title>, <author>, <year> and both <price> elements.
• Children: zero, one or many. Attributes are not children.
• Siblings: nodes sharing a parent. The two <price> elements are siblings of each other and of <title>.
• Ancestors: parent, parent's parent, up to the root. The ancestors of <title> are <item>, <catalog> and the root node.
• Descendants: children, their children, and so on down.

Almost every path you write walks one of those five relationships. Learn them once and the axis names in section 5 stop needing memorisation.

4. Location Paths, Steps and Context

The workhorse expression is a location path, which selects a node-set relative to a starting point. A relative path starts from wherever evaluation currently stands:

catalog/item/author

An absolute path starts with / and anchors at the root node, ignoring the context entirely:

/catalog/item/price

Each step has three parts. An axis naming the tree relationship, a node test naming or typing what to keep, and zero or more predicates filtering what survives. Written out in full, the abbreviated path above is:

/child::catalog/child::item/child::author

Nobody writes that, but knowing it exists explains the shorthands, because each abbreviation is a substitution for an axis:

That last row does more work than it looks. //price[1] expands to /descendant-or-self::node()/child::price[1], so the predicate belongs to the final step and is evaluated once per parent, not once for the document. Section 6 shows what that produces.

Context is supplied by the host. XPath never evaluates in a vacuum. XSLT, a browser, or your scraping library provides a context node, a context size and a context position. For scraping the context node is what matters: it is where a relative path starts walking, and it is what . refers to. Nesting selectors in a loop is exactly the operation of changing the context node, and forgetting to write the leading dot is the most common way to break it. Scrapy documents the failure directly: "Keep in mind that if you are nesting selectors and use an XPath that starts with /, that XPath will be absolute to the document and not relative to the Selector you're calling it from."

5. The Thirteen Axes

Earlier versions of this guide skipped the axes entirely, which was a real hole. Axes are where XPath stops being a fancier CSS selector and starts being able to walk sideways and upward.

Reverse axes count backwards. This is the axis detail that costs people an afternoon. On a reverse axis, position() is measured from the context node outward, so [1] means nearest rather than first in document order. Standing on the EUR price in the worked document, preceding-sibling::*[1] returns the USD <price> and preceding-sibling::*[last()] returns <title>. Both were verified against lxml, and the intuition that [1] means "the earliest one on the page" is inverted here.

That inversion is what makes label anchoring work:

//b[.="Weight"]/parent::td/following-sibling::td[1]
//img[@alt="In stock"]/ancestor::a[1]/@href

The first walks up from a label to its cell, then across to the value beside it. The second walks up from an image to the nearest enclosing link. Both survive a redesign that reorders rows, because neither depends on counting from the top.

following and preceding are not the siblings axes. They sweep the entire document in order, ignoring hierarchy, minus ancestors and descendants. On a large page that is a lot of nodes, and the result is rarely what a scraper wants. Reach for following-sibling first and only widen if the structure forces you to.

Two axes you can ignore in practice: namespace, deprecated after 1.0 and unsupported in several engines, and self used on its own, which mostly appears inside the abbreviation ..

6. Predicates, and Why Comparisons Lie

A predicate is a test in square brackets, applied to every node the step produced. Keep the node when the test is true, discard it otherwise. So far, so obvious. The parts below are where correct-looking expressions return wrong data.

A bare number means position. catalog/item[2] keeps the second item. Any other expression is coerced to boolean instead, which is why //item[@new] keeps items that merely have the attribute, whatever its value.

//price[1] is not the first price. The predicate binds to its step and runs once per parent, so on the worked document it returns two nodes, 29.99 and 12.50, one per item. Wrapping the path first changes the meaning:

//price[1]      -> 2 nodes: the first price inside each item
(//price)[1]    -> 1 node: 29.99, the first price in the document

Nothing warns you. The expression looks right until a page ships a second container.

Comparisons against a node-set mean "any node". When one side of =, !=, < or > is a node-set, XPath 1.0 returns true if some node in that set satisfies the test. On our document //item[price > 300] matches the first item, because one of its two prices is 410.00. The item is also, at the same time, an item whose price is 29.99. If you wanted a single specific price, say so:

//item[price > 300]      -> 1 item, because ANY price exceeds 300
//item[price[1] > 300]   -> 0 items, because the FIRST price is 29.99

!= is not the negation of =. This is the sharpest form of the same rule, and the numbers are from the run on 13 August 2026:

//item[price/@currency != "USD"]       -> 1 item
//item[not(price/@currency = "USD")]   -> 0 items

Both expressions read as "items not priced in dollars". The first asks whether some currency attribute differs from USD, which is true for the item carrying both USD and EUR. The second asks whether no currency attribute equals USD, which is false for every item here. When you want negation, write not(...) around an equality. Writing != gives you something else that happens to agree with it whenever the node-set holds exactly one node, which is exactly why the bug survives testing.

number() fails to NaN, and NaN loses every comparison. Real markup rarely holds a bare number. <price> cells arrive as $12.50, 12,50 EUR, From 9.99. XPath converts the string to a number before comparing, number("12.50 USD") is NaN, and every comparison involving NaN is false. The row does not raise, does not log, and does not appear in your result. It disappears. If your prices come with symbols, strip them in the host language, or filter with contains() and translate() and do arithmetic after extraction.

Escaping inside XML. When your expression lives in an XML or XSLT document, raw < and > are reserved and have to be written as entities:

item[number(minutes) <= 60]

That constraint does not apply in a Python string, a Java string or the browser console, where you write <= directly.

Class attributes are token lists, contains() is a substring test. Against card, card-header and discard, contains(@class, "card") matches all three. The token-safe idiom pads both sides so only whole tokens match:

//div[contains(concat(" ", normalize-space(@class), " "), " card ")]

Meanwhile //div[@class="card"] is an exact string comparison and will miss class="promo card featured". Pick on purpose.

7. The Core Function Library

XPath 1.0 ships a small built-in library. Every function has a return type, none returns void, and XSLT adds more on top of it. Four families cover nearly everything a scraper needs.

Node-set functions

• last() returns the context size, meaning how many nodes are in the current set. It is a count, not the last node.
• position() returns the position of the current node inside that set. The idiom position() = last() tests for the final node.
• count(node-set) returns the size of any node-set you hand it. count(//price) gives 3 on the worked document, and //ul[count(li) > 3] finds lists past a threshold.
• id(object) returns the element whose ID-typed attribute matches. The type has to be declared, in a DTD or schema, which almost no scraped HTML does. Use //*[@id="x"] instead.

String functions

• string(object?) converts to a string. Handed a node-set it takes the string value of the first node in document order and drops the rest. string(//price) returns 29.99 even though three price elements exist. Silent truncation, and a common source of a scraper that always reports the first product's data for every row.
• concat(s1, s2, ...) joins two or more strings.
• starts-with(haystack, needle) and contains(haystack, needle) do prefix and substring tests. There is no ends-with() in 1.0.
• normalize-space(string?) trims the ends and collapses internal whitespace runs to single spaces. With no argument it works on the context node. In scraping it belongs almost everywhere.
• translate(string, from, to) maps characters one for one. It is how you fake case-insensitivity in 1.0, since lower-case() does not exist: contains(translate(., "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"), "dvd") matches our DVD Music title.

Also present: substring(), substring-before(), substring-after() and string-length(). Absent, despite appearing in cheat sheets: format-number(), which belongs to XSLT rather than XPath and will not resolve in a plain XPath engine.

Boolean functions

• boolean(object) coerces. A number is true unless it is zero or NaN; a string is true unless empty; a node-set is true unless it is empty.
• not(boolean) inverts, and section 6 explains why you want it more often than !=.
• true() and false() exist because bare true and false would be read as element name tests.
• lang(string) compares against xml:lang in scope. With no xml:lang anywhere, lang("en") is false rather than an error.

Number functions

• number(object?) converts. Booleans become 1 and 0, strings are parsed, and a node-set is turned into a string first, with the truncation described above.
• sum(node-set) adds the numeric values of every node. sum(//price) returns 452.49 on the worked document, quietly mixing dollars and euros, which is a good reminder that XPath knows nothing about units.
• floor(x), ceiling(x) and round(x) do what their names say, with one wrinkle worth knowing. round() breaks ties toward positive infinity, so round(2.5) is 3 and round(-1.5) is -1, not -2. Both verified against lxml.

The normative catalogue for later versions lives in XPath and XQuery Functions and Operators 3.1, which is the single most useful spec page for daily lookup even if you write only 1.0.

Operators, and how to quote a quote

Arithmetic is +, -, *, div and mod. Division is spelled out because / is already the step separator, and mod is the remainder of a truncating division rather than a mathematical modulo, so -7 mod 3 is -1. The pipe | unions two node-sets, and in 1.0 it accepts nothing else: //author | //year is fine, "a" | "b" is an error.

String literals take single or double quotes, which becomes interesting the moment the expression itself lives inside a quoted string. Two ways out. Alternate the styles, wrapping in double quotes what contains single ones:

select = "item[@new='false' or price/@currency='USD']"

Or, when the expression sits inside an XML or XSL document, escape the inner quote as an entity, ' for ' and " for ". In Python and Java you have a third option that beats both: keep the XPath in single quotes and the literals in double, which is the convention used throughout this guide.

8. Quick Reference

Two notes. Indexing starts at 1, not 0. And //@lang returns attribute nodes; getting the string out is the host library's job, whether that is .get() in Parsel or reading Value in the HTML Agility Pack.

9. Running XPath Over Real HTML: The Libraries

Live pages are malformed by default. Unclosed tags, unquoted attributes, stray markup a strict XML processor refuses outright. Feed a page beginning with <!doctype html> to lxml's XML parser and it raises XMLSyntaxError: StartTag: invalid element name at line 1, column 2, before reaching any content at all. The fix is a tolerant parser that repairs the input and hands you a queryable tree. For a wider survey of the tooling around that job, see our round-up of scraping software, services and plugins.

Versions below were read from PyPI, NuGet, RubyGems and each project's own release page on 13 August 2026.

• lxml 6.1.1 for Python, released 18 May 2026, with 7.0 alphas landing since late May. The C layer is libxml2 and libxslt, and the documentation is exact about scope: "lxml supports XPath 1.0, XSLT 1.0 and the EXSLT extensions through libxml2 and libxslt in a standards compliant way." Calling .xpath() on a tree runs the query against the document; calling it on an element makes that element the context node for relative paths, while an absolute path still escapes to the root. Our lxml tutorial works through a full extraction with it.
• Parsel 1.11.0 and Scrapy 2.17.0, released 29 January and 7 July 2026. Parsel wraps lxml with .get(), .getall() and .attrib, and it is usable standalone without the rest of Scrapy. XPath behaviour is lxml's, EXSLT included.
• HtmlAgilityPack 1.12.4 for .NET, published 3 October 2025, with a 1.13.0 beta on 30 June 2026 and 347.6 million downloads on NuGet. It parses broken markup into a read/write DOM and runs its own XPath engine. One trap deserves naming: SelectNodes returns "null if no node matched the XPath expression", not an empty collection, so a foreach over the result throws NullReferenceException on exactly the pages where the site changed.

var web = new HtmlWeb();
var doc = web.Load("https://example.com/");
var nodes = doc.DocumentNode.SelectNodes("//div[@class='price']");
if (nodes == null) throw new InvalidOperationException("no price nodes on this page");
foreach (var n in nodes) Console.WriteLine(n.InnerText.Trim());

• jsoup 1.23.1 for Java, released 30 July 2026. XPath arrives through selectXpath(), and the API docs are explicit that "XPath 1.0 expressions are supported" by default, with an escape hatch: put another implementation such as Saxon-HE on the classpath and set the javax.xml.xpath.XPathFactory:jsoup system property to reach 2.0 and beyond.
• Nokogiri 1.19.4 for Ruby, released 18 June 2026. It ships two HTML parsers, libxml2 for the HTML4 path and gumbo for the HTML5 one, which means Ruby is the ecosystem where you can pick your tree shape without changing query code. See section 2 for why that choice matters.
• PHP has had DOMXPath for two decades, and PHP 8.4 added Dom\XPath, described in the manual as "the modern, spec-compliant equivalent of DOMXPath" that "allows to use XPath 1.0 queries on HTML or XML documents". Same language version, better tree, because the new Dom\HTMLDocument parses to the WHATWG algorithm. Our walkthrough of DOMXPath for page parsing in PHP covers the older API still running in most codebases.
• Cheerio for Node.js has no XPath at all. An earlier version of this guide grouped it with jsoup and Nokogiri as an equivalent with "varying degrees of XPath versus CSS-selector support". That was wrong. Cheerio's selector engine is css-select, its documentation covers CSS selectors only, and reaching XPath in Node means a separate package over a DOM implementation.

What your library hands back

The expression decides the return type, and half the surprises at the seam between XPath and your code start there. In lxml, a path ending in an element gives you a list of elements. A path ending in @href or text() gives you a list of strings, except they are not plain strings: lxml returns _ElementUnicodeResult objects that remember where they came from, so doc.xpath('//a/@href')[0].getparent().tag answers a and .is_attribute answers True. Handy when you need to walk back up from a matched value.

Wrap the same path in a function and the shape changes underneath you. string(//span) returns one string rather than a list. count(//div) returns a float, 2.0 rather than 2. boolean(//nope) returns False. Code written as result[0] works until somebody adds a count() for a sanity check and gets TypeError: 'float' object is not subscriptable.

Parsel smooths this over and introduces its own edge. .get() on a selector that matched nothing returns None, not an empty string, and that None travels straight into your database unless you write .get(default=""). .getall() returns []. The HTML Agility Pack goes further and returns null from SelectNodes, as the bullet above warned. Three libraries, three different representations of "nothing matched", and none of them raises.

Getting 2.0 functions when you actually need them

Two routes, both verified on 13 August 2026.

EXSLT regular expressions, already compiled into lxml and therefore into Parsel and Scrapy. Register the namespace and re:test becomes available, flags and all:

from lxml import html

doc = html.fromstring(response_body)
ns = {"re": "http://exslt.org/regular-expressions"}
doc.xpath('//a[re:test(., "^add to (cart|basket)$", "i")]/@href', namespaces=ns)

The third argument is the flag string, so "i" buys case-insensitivity without the translate() alphabet dance. This is an extension rather than standard XPath, so it will not work in a browser console or through Selenium.

elementpath 5.1.4, released 8 August 2026, a pure-Python engine covering 1.0 through 3.1 that runs over the lxml tree you already built:

import elementpath
from elementpath.xpath31 import XPath31Parser
from lxml import html

doc = html.fromstring(response_body)
elementpath.select(doc, '//h2[matches(., "^Spec")]', parser=XPath31Parser)
elementpath.select(doc, 'string-join(//td/text(), " | ")', parser=XPath31Parser)

Both matches() and lower-case() returned correct results in our run, on the same tree where lxml rejects them. The trade is throughput: pure Python against a C library, so use it where correctness beats speed rather than as your default engine.

Driving a browser instead of a parser

When the page needs JavaScript, XPath moves into the driver, and the language you get is whatever the browser implements. Selenium, on 4.47.0 across every binding as of 10 August 2026, passes By.XPATH through to the browser engine, so you inherit DOM-level 1.0 and the browser's tree, tbody and all. Playwright takes an XPath through page.locator("xpath=//button"), and treats any locator string beginning with // or .. as XPath automatically. Its documentation pushes back on the whole approach: "We recommend prioritizing user-visible locators like text or accessible role instead of using XPath that is tied to the implementation and easily break when the page changes."

Puppeteer folded XPath into its selector syntax as ::-p-xpath(//h2), with xpath/ as the older spelling; the standalone page.$x() helper no longer appears in the current documentation. If you are porting a scraper written against an older version, that is the line to grep for.

The version trap resurfaces here in a mean form. An expression developed against lxml, where EXSLT regular expressions are available, will fail in Selenium with no obvious explanation, because the browser has no re:test and never did.

10. Where XPath Stops Working

Some failures are boundaries of the language rather than mistakes in your expression. Recognising them saves the hour you would otherwise spend rewriting a path that was never going to work.

Content that JavaScript renders. XPath queries the tree it is given. If the node is not in the response body, no expression conjures it, and the problem lives in your fetching layer. That is a different fix, covered in our notes on scraping dynamic content.

Shadow DOM. XPath cannot cross a shadow root. Playwright states it flatly, "XPath does not pierce shadow roots", and the browser's own document.evaluate behaves the same. Web components, design systems and most video players keep their internals behind that boundary. When $x() returns nothing while you can see the element in the Elements panel, look for #shadow-root between you and it, then switch to an API that pierces.

Iframes. Each frame is a separate document. No expression reaches across. Switch context first, then run the path inside the frame.

Namespaces. Parse a page as HTML and //div works. Parse the same bytes as XHTML or XML and it returns nothing, because the elements now live in the XHTML namespace and an unprefixed name test matches only the null namespace. Two ways through:

doc.xpath('//x:div', namespaces={"x": "http://www.w3.org/1999/xhtml"})
doc.xpath('//*[local-name()="div"]')

Inline SVG and MathML inside otherwise ordinary pages raise the same question, with the answer depending on parser mode. When a selector works in the browser and fails in code, check the parser before you rewrite the path.

11. Bisecting a Path That Returns Nothing

An empty result tells you nothing about which step failed. The fix is mechanical: stop selecting and start counting, then cut the path back from the right until the number stops being zero.

count() is the tool, because it works identically in lxml, in Parsel, in the browser console and in the DevTools search box, and it never throws on an empty set. Against a page whose price sits in a bare <span> rather than the nested <b> an older scraper expected, the ladder looks like this:

count(//div)                                             = 2
count(//div[@class="row"])                               = 2
count(//div[@class="row"]/span)                          = 1
count(//div[@class="row"]/span[@class="price"])          = 1
count(//div[@class="row"]/span[@class="price"]/b/text()) = 0

The break sits between the fourth line and the fifth, and the diagnosis follows in one step: the price element is found, the <b> inside it is gone. Read the second and third lines together as well. Two rows, one price span, which means one row carries no price at all, and any expression assuming one node per row was already lying before the <b> disappeared.

Two refinements make the ladder faster. Start from the middle rather than the top, since a path of eight steps rarely breaks in the first two. And when the count drops from n to zero at a predicate rather than a step, delete only the predicate and keep the step, which separates "the element is missing" from "the element is there but does not match my filter". Those two problems have nothing in common except their symptom.

If every line of the ladder returns a healthy count in the browser and zero in your code, the path is not the problem. Go back to section 2: you are querying a different tree.

12. What Ten Thousand Pages Do to a Path

A path that works once is not a path that works ten thousand times, and the difference is not performance. It is that the failure mode changes from an exception to silence.

Three silent modes have already appeared in this guide, and they share a shape. string() truncates a three-node set to its first node and reports the first product's price for every row. number() slides to NaN on From 9.99 and the row vanishes from a filtered result. A CMS upgrade starts emitting <tbody> where the previous template did not, and one site out of forty goes quiet. None of them raises, none of them logs, and all of them look like an ordinary drop in volume on a dashboard.

The cheapest defence is to assert what you expect at extraction time:

def field(sel, xpath, name, expect=1):
    # xpath should end in text() or an attribute, so .get() returns a value
    nodes = sel.xpath(xpath)
    if len(nodes) != expect:
        raise ValueError(f"{name}: expected {expect} node(s), got {len(nodes)} for {xpath}")
    return nodes.get().strip()

That converts a quiet data-quality problem into a loud parse failure, which is the trade you want. Run it on every field where the count is genuinely known, and log rather than raise where it is not.

The arithmetic argues for it. Suppose one page in three hundred renders a variant layout your path does not handle. Across ten thousand pages a night that is thirty-three rows of nulls, roughly a third of a percent, which no aggregate dashboard will ever flag, and which quietly biases whatever you compute from the data. Fill-rate per field, tracked per site and per night, catches the same drift with one number: a field that was 99.8% populated on Monday and 96% populated on Friday is a broken path, not a market movement.

Budget for maintenance rather than for CPU. Parsing dominates the machine cost, as section 2 measured, while expression evaluation and a count() check per field are microseconds. The expensive part is a human noticing that a selector went stale. Keeping a few thousand expressions green across dozens of sites is its own ongoing job, and it is where a managed extraction service earns its keep.

13. Finding Expressions Without Writing Them by Hand

You rarely need to compose a path from nothing. The fastest feedback loop is the browser already open in front of you.

The console. Chrome and Edge document $x(path) as returning "an array of DOM elements that match the given XPath expression", with an optional second argument, startNode, that gives you a context node for relative paths. Firefox goes further and documents a third: $x(xpath, element, resultType), where the result type may be "number", "string", "bool", "node" or "nodes". Typing $x("//h1") before you paste anything into a scraper is thirty seconds well spent.

The Elements panel. Press Ctrl+F, or Cmd+F on a Mac, and type an XPath straight into the search box. Matches are highlighted in the tree with a running count, which answers the question that matters most: how many nodes does this actually hit?

Copy XPath, with the caveat from section 2. Right-clicking a node offers Copy XPath and Copy full XPath. Both are generated from the rendered DOM, both are position-heavy, and the full variant will contain tbody steps that your parser never created. Use them to understand structure, not as selectors you ship.

Extensions. SelectorsHub is the plugin most testers land on, free with a paid Pro tier, available for Chrome, Edge, Firefox and Opera but not Safari. Its plan pages render prices in JavaScript, so a plain HTTP fetch on 13 August 2026 returned the feature list without a number; treat prices quoted in third-party round-ups as unverified. Older guides, this one included, also pointed at ChroPath. We could not reach anything authoritative about its current status on 13 August 2026, so treat that recommendation as stale rather than confirmed.

Our companion pieces go deeper on both: finding XPath with web developer tools and the XPath cheat sheets worth keeping open.

14. XSLT Is Leaving the Browser. XPath Is Not

XPath was designed to be embedded in other technologies rather than used alone, and the classic hosts are XSLT, XPointer and XQuery. One of those hosts is now being withdrawn from the platform, and the news travels in a garbled form worth correcting.

Google published a removal plan for XSLT in Chrome. Two APIs go: the XSLTProcessor class and the <?xml-stylesheet?> processing instruction. The schedule runs from console warnings in Chrome 142 on 28 October 2025, through deprecation in 143, to Chrome 158 on 17 November 2026, when "XSLT stops functioning on Stable releases", and full removal in Chrome 176 on 17 August 2027. The stated reasons are security and usage: the underlying transformation libraries are "complex, aging C/C++ codebases", and "only about 0.02% of web page loads today actually use XSLT at all".

XPath is not part of that removal. document.evaluate and the console's $x() are untouched, and the Chrome documentation is explicit that the change targets transformation rather than querying. If you read somewhere that browsers are dropping XPath, that is XSLT news with the wrong noun.

The practical consequence for scrapers is small but real. Browser-side XSLT pipelines are on a clock, and if any part of your workflow depends on a browser transforming XML for you, plan the move. The XPath you write against a parsed tree keeps working exactly as before.

15. Writing Paths That Survive a Redesign

Knowing the language tells you what an expression does. Choosing which expression to write decides whether your scraper still runs next quarter.

A ranking that holds up. Stable IDs and data attributes first, //*[@data-testid="price"]. Then a text anchor combined with an axis, as in section 5. Then class tokens through the concat() idiom. Only then positional paths. Anything containing div[3]/div[2]/span[1] should be read as a countdown timer.

Three habits are worth building.

Anchor on labels, not positions. //th[normalize-space()="Price"]/following-sibling::td[1] survives rows being reordered. Be honest about what it does not survive: a new <td> inserted between the label and its value, which [1] will happily grab.

Prefer structured data when the page offers it. JSON-LD, microdata, or an internal JSON API behind the render will outlive any DOM path. On many pages the most maintainable XPath is //script[@type="application/ld+json"]/text(), and that is one of the few places where text() beats normalize-space(.), since collapsing whitespace would corrupt string literals inside the JSON.

Do not over-optimise the expression itself. Precompiling with etree.XPath() saved about 3 microseconds per call in our measurements, meaningful only for a cheap expression run millions of times. On a query that returns 2,000 nodes the difference vanished into noise: 7.6 ms as a string, 7.4 ms precompiled. Across ten thousand pages pulling thirty fields each, precompilation buys back under a second. Correctness is where the time actually goes.

One last comparison, since it comes up in every planning meeting. CSS caught up on structure. MDN dates :has() as "available across browsers since December 2023", and in Python the cssselect layer compiles it straight into XPath: div:has(a) becomes descendant-or-self::div[descendant::a], verified against cssselect 1.5.0. Selecting a parent by its children stopped being a reason to switch languages. The same translator also answers the class question from section 6, since div.card compiles to the padded concat() idiom rather than a bare contains(). What CSS still cannot do is match on text content or compute inside a filter. There is no CSS equivalent of //th[normalize-space()="Price"], nor of count() or string-length() in a predicate. That narrow gap is exactly where the interesting scraping selectors live, and it is the reason this twenty-six-year-old language is still in your stack. Our side-by-side on CSS selectors for scraping covers the other direction.

16. Further Reading and Official References

Every link below was opened on 13 August 2026.

• XPath 1.0, the version your scraper runs. Short enough to read in an evening, and it settles arguments.
• XPath 3.1, the current W3C Recommendation, 21 March 2017.
• XPath and XQuery Functions and Operators 3.1, the best daily lookup page for function signatures.
• XSLT 3.0, Recommendation of 8 June 2017, still the largest host of XPath outside scraping.
• XPath 4.0 community draft at QT4CG, work in progress and not a W3C Recommendation.
• MDN, XPath. Note the path: the docs live under Web/XML now, and older links to Web/XPath resolve to it rather than breaking.
• lxml, XPath and XSLT and Scrapy selectors for engine behaviour, plus Parsel if you want the selector layer without the framework.
• elementpath for XPath 2.0 and up in Python.
• HTML Agility Pack, with source on GitHub and releases on NuGet.
• jsoup for Java, Nokogiri for Ruby, Playwright locators and Chrome DevTools console utilities for browser-side work.
• Scraping software, services and plugins, the wider tooling survey on this site.

When an expression misbehaves, work down four questions in order. Is the node in the tree you parsed, or only in the rendered DOM? Is it behind a shadow root or inside an iframe? Does your parser think the document has namespaces? Does your predicate apply where you think it does? Four questions, and they cover most of it.