Regex and XPath for Web Scraping: Extract HTML Elements
Most scraping tutorials show you how to pull a value out of a tidy, nested element — a price inside <span class="price">, a title inside <h1>. Real pages are messier. Sometimes the data you want isn't wrapped in its own container at all; it sits as a run of sibling nodes between two landmarks, like every line item printed between a "List:" heading and a "The End" footer. Extracting that kind of sequential content is where a web scraper regex and a well-aimed XPath expression earn their place side by side.
This guide covers both tools on exactly that problem — flat, sibling-ordered HTML rather than clean hierarchies — then shows where each one shines, where regex scraping quietly betrays you, and how to combine them the way durable scrapers actually do. Examples are in Python (lxml/parsel and the re module), but the patterns port to any language with an XPath engine and a PCRE-style regex flavor.
The problem: elements in sequence, not in hierarchy
Consider this fragment. The items we want are not children of a shared parent — they're loose text nodes separated by <br/>, bracketed by two headings:
<html>
some text
<h1>List:</h1>
item1
<br/>
item2
<br/>
item3
<br/>
item4
<h2>The End</h2>
some other text...
</html>
There is no <ul>, no <li>, no class="item" to select. A naive //li or a CSS .item finds nothing. You have to reason about order — "everything after this heading and before that one." Two techniques handle it: XPath sibling axes, and a regex anchored between the landmarks.
XPath: use the sibling axes
XPath is built for structural relationships, and its preceding-sibling and following-sibling axes are exactly the vocabulary this problem needs. The idea is to select the text nodes whose nearest preceding heading is List: and whose following heading is The End:
from lxml import html
tree = html.fromstring(page_source)
items = tree.xpath(
"//text()"
"[preceding-sibling::h1[1][normalize-space()='List:']"
" and following-sibling::h2[normalize-space()='The End']]"
)
items = [t.strip() for t in items if t.strip()]
# ['item1', 'item2', 'item3', 'item4']
A few things are doing the work here:
//text()selects text nodes rather than elements, because the items live as bare text between tags.preceding-sibling::h1[1]looks back to the first<h1>above each node; the predicate[normalize-space()='List:']keeps only nodes sitting under the right heading.normalize-space()trims stray whitespace so the comparison doesn't fail on newlines.following-sibling::h2confirms there's a closingThe Endheading further down.
The result comes back as separate text nodes — item1, item2, item3, item4 — which is usually what you want for a list. XPath's readability is its advantage: the expression states the relationship in plain terms, and it keeps working when whitespace or attribute order shifts. If you're still finding your footing with these axes, our XPath scraping guide and notes on how to find XPath in the browser dev tools go deeper. CSS selectors can't express "between these two siblings," which is one of the few places CSS selectors genuinely lose to XPath.
Regex: anchor between the landmarks
When you're working with a raw string rather than a parsed tree — a log line, an inlined <script> payload, a fragment you've already isolated — a regex web scraper solves the same problem by anchoring on the two headings and capturing what's between them.
Start with the core pattern: capture a run of characters that isn't a tag, up to the next separator.
import re
fragment = "item1<br/>item2<br/>item3<br/>item4<h2>The End</h2>"
pattern = re.compile(r'(?P<item>[^<>]+?)\s*(?:<br/>|<h2>)')
for m in pattern.finditer(fragment):
print(m.group('item'))
# item1 / item2 / item3 / item4
Reading it left to right:
(?P<item>[^<>]+?)captures one item — one or more characters that are neither<nor>, so it never runs into a tag. The+?is lazy, so it stops at the earliest separator instead of swallowing the rest of the string.(?:<br/>|<h2>)is a non-capturing group matching either separator,<br/>between items or<h2>at the end. Because it's(?:...), it groups without cluttering your capture indices.\s*absorbs incidental whitespace so captured items come out clean.
(?P<item>...) is Python's named-group syntax. In .NET it's (?<item>...); in JavaScript it's (?<item>...) too. Naming the group means you read m.group('item') instead of a numbered index — a small habit that keeps a scraper legible months later.
To bind the extraction strictly to the region between the headings, wrap it in lookarounds. A lookbehind asserts the List: heading precedes the run; a lookahead asserts The End follows:
pattern = re.compile(
r'(?<=<h1>List:</h1>)\s*' # must follow the List heading
r'((?P<item>[^<>]+?)\s*(?:<br/>|<h2>))+'
r'\s*(?=The\sEnd</h2>)' # must precede the End heading
)
One caveat that trips people up: the + quantifier repeats the group, but a regex only keeps the last value each capture group matched. So this expression confirms the region matches, yet you still recover the individual items by iterating with finditer() on the inner pattern, not by reading the group once. Python's re also requires lookbehinds to be fixed width — (?<=<h1>List:</h1>) is fine because it's a literal, but a variable-length lookbehind would raise an error (that's where the regex PyPI module, which allows variable-length lookbehind, helps).
Parsing HTML with regex: the honest limit
Everything above works because the target is a small, flat, predictable fragment. That's the sweet spot for regex scraping. It is worth stating plainly, though: parsing HTML with regex does not scale to document structure. HTML is not a regular language — nested tags, optional attributes, comments, and CDATA can't be matched reliably by a finite pattern, and every attempt grows into an unmaintainable monster that shatters the first time the markup nests one level deeper.
The rule that keeps scrapers alive is simple: locate with a parser, refine with regex. Use XPath or CSS selectors to pull the right node out of the DOM, then apply a regex to the text of that node to extract or clean the value:
from lxml import html
import re
tree = html.fromstring(page_source)
price_regex = re.compile(r'([\d,]+\.\d{2})')
for node in tree.xpath("//span[@class='price']"):
text = node.text_content()
m = price_regex.search(text)
if m:
print(m.group(1).replace(',', '')) # 1299.00
The DOM parser handles structure it was designed for; regex handles the pattern inside the text, which is what it was designed for. That division of labor survives markup changes that would break either tool used alone. Our companion piece on regex for web scraping drills into the pattern cookbook — prices, emails, IDs, dates — and our web scraper regex in PHP walkthrough shows the same split with preg_match.
A browser-console shortcut: count and sum matching elements
Not every extraction needs a full scraper. When you just want a quick number off a page you're looking at — the total across a column of values, say — the browser console plus querySelectorAll is the fastest path. This is the modern take on an old utility for summing DOM element values.
Suppose a page lists hotel counts per country in <span class="hotel-num"> elements and you want the grand total. Open DevTools (F12, or Cmd+Option+I on macOS), go to the Console, and run:
function domSum(selector) {
return [...document.querySelectorAll(selector)]
.map(el => parseInt(el.textContent.replace(/[^\d]/g, ''), 10))
.filter(n => !Number.isNaN(n))
.reduce((total, n) => total + n, 0);
}
domSum('span.hotel-num'); // e.g. 48213
It grabs every matching element, strips non-digits from each with a tiny regex (/[^\d]/g removes commas, currency symbols, "views", etc.), drops anything that isn't a number, and adds up the rest. Swap the selector for whatever the page uses — td.count, .metric-value, li:nth-child(2) — and you have an instant total without writing or deploying anything. Because it reads textContent, it works on values the server already rendered into the DOM.
You can wrap it as a bookmarklet to reuse across sites:
javascript:(function(){
const sum = [...document.querySelectorAll('span.hotel-num')]
.map(el => parseInt(el.textContent.replace(/[^\d]/g,''),10))
.filter(n => !Number.isNaN(n))
.reduce((a,b)=>a+b,0);
prompt('Sum of matched elements:', sum);
})();
Save that as a bookmark, edit the selector to fit the page, and one click pops up the total ready to copy. It's a genuinely handy way to spot-check data before committing to a real scraper — and a reminder that querySelectorAll plus a small regex covers a surprising amount of casual extraction. For anything recurring or large, though, graduate to a proper scraper with parsing and storage rather than pasting into the console each time.
Which tool for which job
| Situation | Reach for |
|---|---|
| Selecting elements by structure, class, or relationship | XPath or CSS selectors |
| "Everything between these two siblings/headings" | XPath sibling axes |
| Pulling a pattern (price, email, ID) out of isolated text | Regex |
| Cleaning/normalizing an extracted string | Regex (sub/replace) |
| A quick one-off total on a page you're viewing | querySelectorAll + regex in the console |
| Parsing whole documents or nested markup | A DOM parser — never regex alone |
In practice a mature scraper uses all of them: XPath to walk the page, regex to sharpen each field, and a parser holding the whole thing together.
FAQ
Can I scrape a website using only regex? For small, flat fragments you've already isolated, yes. For parsing full HTML documents with nested tags, no — use a DOM parser (lxml, parsel, BeautifulSoup) to locate elements, then apply regex to the extracted text. Structure is the parser's job; patterns are regex's job.
How do I extract elements that sit between two headings?
Use XPath sibling axes: select nodes whose preceding-sibling is the opening landmark and whose following-sibling is the closing one. On a raw string, anchor a regex with a lookbehind on the first landmark and a lookahead on the second.
Why does my regex capture too much of the page?
A greedy quantifier like .* or [^<>]+ grabs as much as it can. Make it lazy with .*? or [^<>]+? so it stops at the first separator instead of the last.
What's the difference between preceding-sibling and ancestor in XPath?
ancestor walks up the tree to enclosing elements; preceding-sibling moves backward among nodes at the same level. Sequential, non-nested content needs the sibling axes, not ancestor/descendant.
Do named capture groups work in every regex engine?
The concept is universal but the syntax differs: (?P<name>...) in Python and PCRE, (?<name>...) in .NET and JavaScript. Pick the form your engine expects.
Wrapping up
Flat, sequential HTML is a common trap that clean-hierarchy tutorials skip. XPath's preceding-sibling and following-sibling axes describe "between these landmarks" precisely; a lazily-quantified, lookaround-anchored regex does the same on raw strings; and the golden rule — locate with a parser, refine with regex — keeps the whole thing from turning brittle. For quick totals, querySelectorAll plus a one-line regex in the console beats writing any code at all. When the targets multiply and the markup keeps shifting, handing the upkeep to a managed web scraping service turns a maintenance treadmill back into clean, structured data.