When you build a scraper, the hardest part is rarely downloading the page — it is pointing your code at the exact piece of content you want to pull out. For HTML documents, one of the most common ways to address a single element is an XPath expression. The good news is that every modern browser ships with a built-in inspector that can hand you an element's XPath in a couple of clicks, so you don't have to write one from scratch. The catch is that the path the browser gives you is rarely the path you actually want to keep.
This guide walks through how to grab an XPath using the developer tools (also called the web inspector), what the browser is really doing when it generates one, and how to turn that raw result into a selector you can trust inside your scraping code.
What XPath is and why scrapers rely on it
XPath ("XML Path Language") is a small query language for navigating the tree structure of an XML or HTML document and selecting nodes from it. Because a rendered web page is just a DOM tree, XPath lets you say things like "the third row of this table" or "the link whose text contains Login" in a single expression.
If you want the formal definition and the full set of axes, functions, and operators, the authoritative sources are the W3C specification and Mozilla's reference:
- W3C XPath specification: https://www.w3.org/TR/xpath-31/
- MDN XPath reference: https://developer.mozilla.org/en-US/docs/Web/XPath
For a more scraping-focused overview of how XPath behaves in practice, see the companion piece at /xpath-review/.
Opening the developer tools
Every major browser bundles the inspector, and the entry points are nearly identical:
- Open the panel: press
F12, orCtrl+Shift+Ion Windows/Linux andCmd+Option+Ion macOS. - Jump straight to an element: right-click anything on the page and choose Inspect. The browser opens the inspector and pre-selects that node in the DOM tree.
In Chrome this whole environment is called Chrome DevTools, and Google maintains official documentation for it:
- DevTools overview: https://developer.chrome.com/docs/devtools/
- The Elements panel: https://developer.chrome.com/docs/devtools/elements/
- Ways to open DevTools: https://developer.chrome.com/docs/devtools/open/
Step by step: copying an XPath in Chrome
Once DevTools is open, the workflow to extract an XPath is short:
- Switch to the Elements tab. This panel shows the live DOM — the structure of the page as it currently exists in the browser, including anything JavaScript has added after load. That distinction matters and we'll come back to it.
- Turn on the element picker. Click the small "Inspect" / selector arrow at the
top-left of the DevTools window (shortcut
Ctrl+Shift+CorCmd+Shift+C). When it is active the icon turns blue. See Chrome's note on inspect mode: https://developer.chrome.com/docs/devtools/inspect-mode/. - Click the target element in the main browser window. DevTools highlights the matching node in the Elements tree.
- Right-click the highlighted node in the DOM tree.
- From the context menu, hover over Copy.
- In the submenu, choose Copy XPath (for a relative-style path the browser thinks is shortest) or Copy full XPath (for an absolute path from the document root). The expression is now on your clipboard, ready to paste into your code or a notes file.
That is the entire mechanical process. The interesting part is understanding what you just copied.
A word of caution: the browser's XPath is not your XPath
It is tempting to treat Copy XPath as a finished selector. It almost never is.
The browser composes the path with its own algorithm, optimized for "uniquely
identify this one node right now," not for "survive future changes to the page." As a
result you tend to get a long, position-based chain that is extremely brittle:
add one row to a table, wrap a section in a new div, or let an A/B test reorder the
layout, and the path silently stops matching. This is the single most common reason a
copied XPath works in the console but throws a "no such element" error when the
scraper runs later.
So the real skill is not copying a path — it's reading the one the browser gives you and rewriting it into something more robust. To do that well you need a working understanding of XPath itself (those W3C and MDN links above), not just the copy button.
What a browser-generated XPath actually looks like
Here is a representative example of the kind of absolute, position-driven path the inspector produces for a deeply nested value — in this case a quote field on a financial page:
//*[@id="main-0-Quote-Proxy"]/section/div[2]/section/div/section/div[2]/div[1]/div[1]/div/table/tbody/tr[3]/td[1]/span
Read it left to right and you can see the problem: after a single anchoring id, the
path is one long sequence of div[2] / div[1] / tr[3] / td[1] positional steps. Every
one of those numeric indexes is a dependency on the page's current structure. If the
site changes its markup — and large sites change constantly — most of these steps can
break at once.
A far more durable approach is to anchor on stable, meaningful attributes and use XPath predicates instead of raw positions, for example:
//div[@data-test='qsp-statistics']//td[contains(@class,'trailing-pe')]
The exact attributes depend on the site, but the principle is universal: prefer
id, data-*, and descriptive class values over absolute positions, and lean on
functions like contains() and normalize-space() to tolerate minor variation.
Validating and refining the path inside DevTools
Before you commit a selector to your code, test it where the page actually lives. DevTools gives you two quick ways to do this — one visual, one programmatic.
Searching the Elements tree
With the Elements panel focused, press Ctrl+F (Cmd+F on macOS) to open the search
box. Type an XPath expression and DevTools highlights every matching node in the DOM
tree, while the match counter shows how many elements your path hit. For a scraping
target you almost always want that count to read 1 of 1 — anything higher means
your expression is ambiguous and needs tightening. The same box also accepts plain
text and CSS selectors, so you can quickly compare approaches.
Evaluating selectors in the Console
For programmatic checks, switch to the Console tab (or press Esc to open it as a
drawer alongside any other panel) and use the Console Utilities helpers. These are
convenience functions that exist only inside the DevTools Console — they are not part
of the page and won't work if you paste them into your own scripts. The two relevant
to selector work are:
$x("xpath")— evaluates an XPath expression and returns an array of matching nodes. This is the dedicated XPath helper. Check.lengthto confirm uniqueness, and click any item in the returned array to jump straight to it in the Elements panel. It also takes an optional second argument, a start node, so you can scope the query to a subtree — for example inside a specificiframeor container:$x("//span", someElement).$$("css")— the CSS-selector counterpart. It is shorthand fordocument.querySelectorAll()and returns an array of elements matching a CSS selector. Despite the similar name,$$()does not evaluate XPath, so keep XPath queries on$x()and CSS queries on$$(). There is also a single-match$("css"), which mirrorsdocument.querySelector().
A practical pattern is to run $x("..."), glance at the array length, then iterate:
edit the expression, re-run, watch the count fall to one. That loop takes seconds and
catches brittle or ambiguous selectors long before they ever fail in a full scraping
run.
Versioning note. The exact set and behaviour of these console helpers can shift between Chrome releases, and the names overlap confusingly with library functions you may use elsewhere, so verify against the official Console Utilities reference for the build you're on rather than relying on memory: https://developer.chrome.com/docs/devtools/console/utilities/.
The same helpers exist in Chromium-based Edge; Firefox's console also supports $x()
and $$() with equivalent meanings.
The same idea in Firefox and Edge
You are not limited to Chrome. Firefox's built-in Inspector offers an equivalent
right-click Copy → XPath, and its console likewise supports $x(). Microsoft Edge
is Chromium-based, so its DevTools behave essentially like Chrome's.
- Firefox DevTools docs: https://firefox-source-docs.mozilla.org/devtools-user/
- Microsoft Edge DevTools docs: https://learn.microsoft.com/en-us/microsoft-edge/devtools/
The general lesson holds across all of them: the copy command is a starting point, and every browser's auto-generated path needs the same human review.
Putting the XPath to work in your scraper
Once you have a tested expression, almost every serious scraping or browser-automation library can consume it directly. Common options:
- lxml — fast XPath evaluation over parsed HTML/XML in Python: https://lxml.de/xpathxslt.html
- parsel — the standalone selector library used by Scrapy, supporting both XPath and CSS: https://parsel.readthedocs.io/
- Scrapy — its
Selectorobjects expose.xpath()for extraction: https://docs.scrapy.org/en/latest/topics/selectors.html - Selenium — locate elements with
By.XPATHwhen you need a real browser: https://www.selenium.dev/documentation/webdriver/elements/locators/ - Playwright — supports
xpath=locators for modern browser automation: https://playwright.dev/docs/locators
Whatever library you choose, feed it the refined expression — the one you rewrote and verified — rather than the raw output of Copy XPath.
Takeaways
- The inspector's Copy XPath / Copy full XPath command gives you an element's path in a couple of clicks, in any modern browser.
- What it produces is usually an absolute, position-heavy path that breaks the moment the page's structure changes.
- Treat that path as a draft: anchor on stable attributes, replace positional indexes with predicates, and test the result in the Elements search or the Console before shipping it.
- A little fluency in XPath itself is what separates a fragile scraper from a reliable one — start with the W3C and MDN references above, and the overview at /xpath-review/.
References
- XPath review and scraping context: /xpath-review/
- W3C XPath specification: https://www.w3.org/TR/xpath-31/
- MDN XPath reference: https://developer.mozilla.org/en-US/docs/Web/XPath
- Chrome DevTools: https://developer.chrome.com/docs/devtools/
- Firefox DevTools: https://firefox-source-docs.mozilla.org/devtools-user/
- Microsoft Edge DevTools: https://learn.microsoft.com/en-us/microsoft-edge/devtools/
- lxml XPath: https://lxml.de/xpathxslt.html
- parsel: https://parsel.readthedocs.io/
- Scrapy selectors: https://docs.scrapy.org/en/latest/topics/selectors.html
- Selenium locators: https://www.selenium.dev/documentation/webdriver/elements/locators/
- Playwright locators: https://playwright.dev/docs/locators