Open the page, hit view-source, and the forty-two rows you came for are not rows. They are bare text separated by <br>, sitting between an <h1> and an <h2>, with no wrapper, no class, and nothing for //li or .item to hold on to. The data is ordered but not nested. Every selector you know describes hierarchy, and this page does not have any.
Two techniques cover the case: XPath sibling axes, and a regex anchored between the two landmarks. Both appear in every write-up on the subject, including the earlier version of this one. Both are also wrong in ways that never raise an exception. We rebuilt every example here on 13 August 2026 against Python 3.11.15, lxml 6.1.1 (libxml2 2.14.6), parsel 1.11.0 and cssselect 1.5.0, then ran them against deliberately awkward markup and timed them. Almost every failure mode below returns an empty list or a short list instead of an error. That is the worst way for a scraper to be wrong, because nothing logs and the row count looks plausible.
Where flat markup comes from, and why it is not going away
Flat sibling runs are not a relic. They are what happens whenever markup is produced by something other than a component framework.
Rich-text editors. Every CMS with a WYSIWYG field lets an author press Enter and get a <br> rather than a new block. Addresses, opening hours, ingredient lists and spec sheets end up as one text blob with separators in it.
Converters and old templates. PDF-to-HTML and DOCX-to-HTML pipelines emit positional markup: they know where a line ended, not that four lines were a list. And anything written before 2010 that still earns money tends to build pages by string concatenation, where "<br/>".join(rows) is the shortest path from a database cursor to a page.
<br> is the usual separator because it is the element that cannot contain anything. The HTML Standard lists exactly thirteen void elements — area, base, br, col, embed, hr, img, input, link, meta, source, track, wbr — and a void element has no children by definition. That single fact is why the data cannot be nested under it, and why you are reduced to reasoning about order.
Worth pinning down now, because it loses you data later: the slash in <br/> means nothing. The spec is blunt about the trailing solidus on a void element start tag, saying it "does not mark the start tag as self-closing but instead is unnecessary and has no effect of any kind." A parser treats <br>, <br/> and <br /> as the same node. A regex does not.
The problem: elements in sequence, not in hierarchy
Here is the fragment the rest of this article works against. The items are not children of a shared parent. They are loose text nodes separated by <br/>, bracketed by two headings:
<html>
some text
<h1>List:</h1>
item1
<br/>
item2
<br/>
item3
<br/>
item4
<h2>The End</h2>
some other text...
</html>There is no <ul>, no <li>, no class="item". A naive //li or a CSS .item finds nothing at all. You have to express order instead of containment: everything after this heading and before that one.
XPath: use the sibling axes
XPath was designed for structural relationships, and the preceding-sibling and following-sibling axes are the vocabulary this needs. Select the text nodes whose nearest preceding heading is List: and which still have a The End heading below them:
from lxml import html
tree = html.fromstring(page_source)
items = tree.xpath(
"//text()"
"[preceding-sibling::h1[1][normalize-space()='List:']"
" and following-sibling::h2[normalize-space()='The End']]"
)
items = [t.strip() for t in items if t.strip()]
# ['item1', 'item2', 'item3', 'item4']That output is real. We ran it on the fragment above under lxml 6.1.1 and got exactly those four strings. Three pieces are doing the work:
//text()selects text nodes rather than elements, because the items live as bare text between tags. There is no element to select.preceding-sibling::h1[1]looks back to the nearest<h1>above each node, and the predicate[normalize-space()='List:']keeps only nodes under the right heading. The[1]is not "the first heading in the document". XPath 1.0 classifiesancestor,ancestor-or-self,precedingandpreceding-siblingas reverse axes, and defines proximity position as "the position of the node in the node-set ordered in document order if the axis is a forward axis and ordered in reverse document order if the axis is a reverse axis." On a reverse axis,[1]means nearest.following-sibling::h2asserts that a closing heading exists further down. It does not bound the region, which turns out to matter.
normalize-space() trims and collapses whitespace so the comparison survives the newlines a pretty-printer leaves behind.
Do not throw away the result type. lxml returns text results as smart strings, not plain str. The lxml XPath documentation puts it this way: "XPath string results are 'smart' in that they provide a getparent() method that knows their origin," and for //text() that parent is "the Element that contains the text or tail that was returned." You also get is_text, is_tail and is_attribute flags. On the fragment above, the first result reports is_tail=True and getparent().tag == 'h1', so you can see the item is the tail of the heading rather than its content. Calling .strip() on it, as the snippet above does, hands back an ordinary str and that context is gone. Strip late, or keep both.
If the axes themselves are still new, our XPath scraping guide covers them from the beginning, and the walkthrough on how to find XPath in the browser dev tools shows how to build one against a live page.
Seven ways the sibling expression fails without saying so
This is the part the tutorials skip. Every case below was run under lxml 6.1.1.
The heading is wrapped. Change <h1>List:</h1> to <div class="section-head"><h1>List:</h1></div> and the expression returns []. The <h1> is no longer a sibling of the items, so preceding-sibling::h1 finds nothing. A designer adding a wrapper div for spacing silently empties your dataset. This is the most common way the technique dies in production.
The closing landmark is missing. On the last page of a paginated list the <h2>The End</h2> often is not rendered. The expression returns [] again, because following-sibling::h2 is an unbounded existence test rather than a boundary.
Two lists on one page. Given two List: / The End pairs in sequence, the expression returns all eight items merged into one flat array, since every node in the second list also has a nearest preceding List: and a following The End. The predicate describes a shape, not a region.
Any item carries inline markup. This one loses data rather than all of it. Take item1<br/>item<b>2</b> here<br/><a href="#">item3</a>. The expression returns ['item1', 'item', 'here']. The bold tag splits item2 here into two text nodes and drops the 2 that lives inside <b>, and item3 disappears entirely because its text node's parent is the <a>, not the body. Three items in, three strings out, all three wrong. Nothing in the row count suggests a problem.
Whitespace-only nodes. //text() matches the newlines between tags. The if t.strip() filter handles it, and it is not optional.
The parser is not the browser. lxml uses libxml2's HTML parser, which predates the HTML5 parsing algorithm. Dev tools show the tree Chrome built; lxml may build a different one from the same bytes, particularly around tables and unclosed block elements. An XPath copied out of dev tools is a hypothesis until you run it against the parser your scraper uses.
XPath 1.0 is all you get. lxml states plainly that it "supports XPath 1.0, XSLT 1.0 and the EXSLT extensions through libxml2 and libxslt in a standards compliant way." We confirmed the ceiling by hand: the XPath 2.0 node-comparison operator <<, the 3.0 arrow =>, and matches() all fail under lxml 6.1.1 with Invalid expression or Unregistered function. If you need the 2.0 and 3.1 idioms, elementpath (5.1.4, 8 August 2026) evaluates them over an lxml tree, and it returned correct results for the same document in our test.
Six of those seven produce a number, not an exception. Assert your row count.
A version that survives real pages
Stop asking XPath to describe a region and walk the region instead. Select the opening landmark, then iterate siblings until you hit the closing one, accumulating text as you go. That keeps inline markup, handles <br> in any spelling, and still works when the closing landmark never arrives.
from lxml import html
def items_between(tree, start_xpath, stop_tag, stop_text, break_tags=('br',)):
"""Text runs between two landmark siblings, split on break_tags."""
start = tree.xpath(start_xpath)
if not start:
return []
start = start[0]
items, buf = [], [start.tail or '']
for el in start.itersiblings():
if not isinstance(el.tag, str): # comment or processing instruction
buf.append(el.tail or '')
continue
if el.tag == stop_tag and el.text_content().strip() == stop_text:
break
if el.tag in break_tags:
items.append(''.join(buf))
buf = [el.tail or '']
else:
buf.append(el.text_content() or '')
buf.append(el.tail or '')
items.append(''.join(buf))
return [' '.join(s.split()) for s in items if s.strip()]
items = items_between(tree, "//h1[normalize-space()='List:']", 'h2', 'The End')Two details are load-bearing. el.text_content() pulls the text of a whole subtree, so item<b>2</b> here survives as one string. And the isinstance(el.tag, str) guard exists because comments and processing instructions are also siblings: on a document containing <!-- note --> between items, calling text_content() on the comment raises ValueError: Input object is not an XML element: HtmlComment. That crash is what an unguarded version does on roughly every real page, since CMS platforms leave comments everywhere.
Same five documents, both approaches, measured:
| document | sibling-axis XPath | items_between() |
|---|---|---|
plain <br/> separators |
['item1', 'item2', 'item3'] |
['item1', 'item2', 'item3'] |
<br> and <BR /> |
correct | correct |
item<b>2</b> here and <a>item3</a> |
['item1', 'item', 'here'] |
['item1', 'item2 here', 'item3'] |
closing <h2> absent |
[] |
['item1', 'item2'] |
two List: sections on the page |
4 items, both lists merged | 2 items, first list only |
Entities come out decoded either way, because the parser does that before you see the tree: R&D reads back as R&D. Hold on to that, because the regex section is about to hand back the raw entity text.
What ten thousand pages do to that expression
The sibling-axis expression is quadratic. It has to be: for every text node on the page it walks the sibling axis backwards looking for an h1 and forwards looking for an h2. On a fifteen-item list nobody notices. On a long table of contents it becomes the whole runtime.
We generated flat pages with a fixed 100-link navigation block and a growing list, then timed three approaches on each. Medians of seven runs, or three at the top two sizes, on one shared 2-core Intel Xeon at 2.10GHz, Python 3.11.15, lxml 6.1.1:
| items | page bytes | regex finditer |
sibling-axis XPath | itersiblings() walk |
|---|---|---|---|---|
| 250 | 4,960 | 0.4 ms | 1.0 ms | 0.08 ms |
| 1,000 | 14,710 | 0.7 ms | 28 ms | 0.27 ms |
| 2,000 | 28,710 | 1.1 ms | 123 ms | 0.54 ms |
| 4,000 | 56,710 | 2.1 ms | 436 ms | 1.08 ms |
| 8,000 | 112,710 | 5.9 ms | 1,152 ms | 2.19 ms |
Measured on a single shared machine; a repeat run varied by up to 40% at the larger sizes. Your numbers will differ, the shape will not. All three returned identical item lists at every size.
Thirty-two times the items, roughly eleven hundred times the work. The walk stays linear and beats the regex.
The parser is not the problem. Parsing the 8,000-item page with html.fromstring() takes 0.91 ms. The other 1,151 ms is the XPath predicate. "Parsers are slow, regex is fast" is the most durable myth in this corner of scraping, and it is backwards here: libxml2 builds the tree in under a millisecond, and a badly shaped XPath expression spends a second on it. selectolax 0.4.11, which wraps the lexbor engine, parsed the same page and selected all 2,000 <br> elements in 3.6 ms.
Precompiling does not save you. The usual advice is to hoist the expression into etree.XPath() so it compiles once. We measured it: 1,402 µs for the string form against 1,398 µs for the compiled object on a 300-item page. Compilation is not where the time goes. Fix the algorithm, not the caching.
Put it in units you bill in. One extra second per page across a ten-thousand-page overnight crawl is close to three hours of wall clock, on a job with a morning deadline. The run that finished at 4 a.m. now finishes at 7. At that scale the question stops being which axis reads best and becomes who notices when a wrapper div appears, which is the practical argument for a monitored data feed over a cron job nobody owns.
Regex: anchor between the landmarks
When you are holding a raw string rather than a parsed tree — a log line, an inlined <script> payload, a fragment you already isolated — a regex solves the same problem by anchoring on the landmarks and capturing what lies between.
import re
fragment = "item1<br/>item2<br/>item3<br/>item4<h2>The End</h2>"
pattern = re.compile(r'(?P<item>[^<>]+?)\s*(?:<br/>|<h2>)')
for m in pattern.finditer(fragment):
print(m.group('item'))
# item1 / item2 / item3 / item4Reading it left to right:
(?P<item>[^<>]+?)captures one item: one or more characters that are neither<nor>, so it can never run into a tag. The+?is lazy, so it stops at the earliest separator rather than swallowing the rest of the string.(?:<br/>|<h2>)is a non-capturing group matching either separator. Because it is(?:...)it groups without shifting your capture indices.\s*absorbs whitespace before the separator only. The earlier version of this article claimed it makes captured items "come out clean"; it does not. Run the pattern against the pretty-printed document from the top of this page and the four matches come back as'\nitem1','\nitem2','\nitem3','\nitem4', leading newline intact. You still need.strip().
(?P<item>...) is Python's named-group syntax, and it travels worse than the usual advice suggests. We checked both engines directly. Python's re accepts (?P<x>a) and rejects both (?<x>a) ("unknown extension ?<x") and (?'x'a). Node 22.22.2 does the exact opposite: it accepts (?<x>a) and rejects (?P<x>a) with "Invalid regular expression: Invalid group". The two syntaxes are mutually exclusive, so a pattern shared between a Python scraper and a browser snippet needs two spellings.
To bind extraction strictly to the region between the headings, wrap it in lookarounds:
pattern = re.compile(
r'(?<=<h1>List:</h1>)\s*' # must follow the List heading
r'((?P<item>[^<>]+?)\s*(?:<br/>|<h2>))+'
r'\s*(?=The\sEnd</h2>)' # must precede the End heading
)That expression matches, and then hands you one item. The + repeats the group, but Python's re documentation is explicit: "If a group matches multiple times, only the last match is accessible." We confirmed it — m.group('item') returns item4 and nothing else. Use the lookaround form to confirm the region exists, then run finditer() with the inner pattern to recover the items.
Fixed-width lookbehind. The same documentation says the lookbehind's contained pattern "must only match strings of some fixed length, meaning that abc or a|b are allowed, but a* and a{3,4} are not." (?<=<h1>List:</h1>) is a literal, so it is fine; (?<=<h1>\s*List:</h1>) raises look-behind requires fixed-width pattern. Two escapes exist. The regex module on PyPI (2026.7.19) advertises "A lookbehind can match a variable-length string" and compiled the variable-width version without complaint in our test. JavaScript is the other: V8 accepted the same pattern in Node 22. Safari only shipped lookbehind at all in version 16.4, released 27 March 2023, so browser-side patterns that lean on it are younger than most scrapers.
Where the regex quietly drops data
The XPath failures returned empty lists. The regex failures return plausible lists, which is worse.
<br> without the slash. Real HTML overwhelmingly writes <br>, and the parser cannot tell the difference. The pattern can. On a six-item list written with <br>, the pattern above returns one match, and that match is a newline. Not five items, not an error. One piece of whitespace. <BR /> returns one match too, because the alternation is case-sensitive and space-sensitive. Written <br/> the same list returns seven matches, six items plus a trailing whitespace match, so even the happy path needs filtering.
Entities stay raw. R&D<br/>item2 yields R&D. The parser gives you R&D. Every , &, ’ and é on the page reaches your database as source text unless you remember to unescape it.
Comments become items. item1<!-- note --><br/>item2 yields a bare space as the first item, because a comment's contents are just characters to a regex. The parser skips comments; [^<>]+? walks straight into them.
Catastrophic backtracking. The lazy character class above is safe. The pattern people reach for next is not. (\s*[^<>]+\s*)+<h2> looks harmless and describes the same idea — a run of items followed by the closing heading — but nesting a quantified group inside another quantifier makes the engine try every partition of the input when the match ultimately fails. Timings from our run, on strings of five-character items with no <h2> present:
| items | input length | re.search() |
|---|---|---|
| 2 | 14 chars | 0.4 ms |
| 3 | 19 chars | 20 ms |
| 4 | 24 chars | 930 ms |
| 5 | 29 chars | 45 s |
Twenty-nine characters. Forty-five seconds. Each additional item multiplies the time by roughly forty-five, and Python's re has no backtrack ceiling to stop it, so the worker simply stops responding. PHP does have a ceiling: pcre.backtrack_limit defaults to 1,000,000 and pcre.recursion_limit to 100,000. That converts an infinite hang into a failed match, which is a different kind of trap, because a failed match and "no items on this page" look identical from the caller.
Lazy quantifiers, explicit character classes, no nested repetition. Or a parser.
Why regex cannot parse HTML, stated precisely
Everything above works because the target is small, flat and predictable. That is the honest boundary of the technique, and the reason is structural rather than stylistic.
HTML is not a regular language. Regular expressions recognise regular languages, which a finite state machine accepts with no memory of how deep it is. HTML nesting needs that memory. The HTML Standard's tree construction stage is built around a "stack of open elements," which the spec describes as growing downwards, initially empty, with the topmost node the first one added. A stack is exactly what a finite automaton does not have. Arbitrary nesting, optional end tags, implied tags, comments and foster parenting all live in that stack, so no fixed pattern can follow them.
The rule that keeps scrapers alive: locate with a parser, refine with regex. Use XPath or CSS to pull the right node out of the DOM, then apply a regex to the text of that node.
from lxml import html
import re
tree = html.fromstring(page_source)
amount_re = re.compile(r'([\d,]+\.\d{2})')
for node in tree.xpath("//span[@class='price']"):
m = amount_re.search(node.text_content())
if m:
print(m.group(1).replace(',', '')) # 1299.00Note text_content() rather than .text: it collects the subtree, so US$ <b>1,299</b>.00 still yields one string. The division of labour survives markup changes that would break either tool alone, and it is baked into the tooling. parsel, the selector library Scrapy uses, gives every selector a .re() and .re_first() method for exactly this hand-off, so sel.css('span.price::text').re_first(r'([\d,]+\.\d{2})') returns 1,299.00 in one line. Our companion piece on regex for web scraping covers the pattern cookbook — amounts, emails, IDs, dates — and the PHP walkthrough shows the same split with preg_match.
A correction: CSS lost this argument, and then won part of it back
The earlier version of this article said CSS selectors "can't express 'between these two siblings'." That was true when it was written and it is not true now, so here is the correction.
CSS has had the subsequent-sibling combinator ~ since Level 3, which covers "after". What it lacked was "before". The relational pseudo-class :has(), defined in Selectors Level 4 as matching "an E element, if there exists an element that matches either of the relative selectors," supplies it. Put a sibling combinator inside :has() and you have both bounds:
h1 ~ span:has(~ h2)That reads as "every span after an h1 that still has an h2 after it." We ran it on 13 August 2026 against soupsieve 2.9.2, the CSS engine behind BeautifulSoup, and it returned the three items between the headings while dropping the one after The End. cssselect 1.5.0 (released 27 July 2026, with a changelog entry for fixed :has() support) handles it in parsel too, and what it compiles to is instructive:
descendant-or-self::h1/following-sibling::span[following-sibling::h2]It becomes the same sibling-axis XPath. In browsers, :has() reached Baseline in December 2023 — Safari 15.4 in March 2022, Chrome 105 in September 2022, Firefox 121 on 19 December 2023 — so document.querySelectorAll('h1 ~ span:has(~ h2)') works in the console today. Two limits MDN records: :has() cannot be nested inside another :has(), and pseudo-elements are not valid inside it or as its anchor.
So why does CSS still lose on the fragment at the top of this page? Not because of the sibling relationship. Because CSS selects elements and the items here are not elements. There is no CSS syntax for a text node. Wrap each item in a <span> and CSS is fine; leave them bare between <br> tags and only XPath or a tree walk reaches them. Our notes on CSS selectors for scraping go through where each language is stronger. The line is text nodes, not siblings.
The console shortcut, and the three things that changed
Not every extraction deserves a scraper. When you want one number off a page you are already looking at, the console beats writing code. This is the modern take on an old utility for summing DOM element values, and three things have changed since it was written.
Chrome will refuse your paste. Pasting into the DevTools console triggers a self-XSS warning, and you have to type the phrase allow pasting once per profile before it accepts clipboard content. Chrome documented this in a post dated 5 December 2023. If your snippet appears to do nothing, check for the warning before you debug the code.
You do not need the spread. Chrome's Console Utilities API supplies $$(selector), documented as "equivalent to calling Array.from(document.querySelectorAll())", and $x(path), which "returns an array of DOM elements that match the given XPath expression." So $x("//h1/following-sibling::text()") tests a sibling expression against the live DOM before you commit it to a scraper, and copy(value) puts the result on the clipboard.
Suppose a page lists hotel counts per country in <span class="hotel-num">. Open DevTools with F12 or Cmd+Option+I, go to the Console, and run:
function domSum(selector) {
return [...document.querySelectorAll(selector)]
.map(el => parseInt(el.textContent.replace(/[^\d]/g, ''), 10))
.filter(n => !Number.isNaN(n))
.reduce((total, n) => total + n, 0);
}
domSum('span.hotel-num'); // e.g. 48213Third change: that number is probably wrong. Stripping every non-digit and calling parseInt is the standard recipe and it silently mangles four common formats. We ran a six-value column through it in Node 22:
| cell text | parseInt after stripping non-digits |
what it should be |
|---|---|---|
1,299.50 |
129950 | 1299.50 |
-40 |
40 | -40 |
1.234,56 |
123456 | 1234.56 |
2.5k |
25 | 2500 |
The decimal point vanishes and multiplies the value by a hundred. The minus sign vanishes and flips it. Over that six-value column the naive total came to 302,983 against a true total of 54,506.06, an error of more than five times, with no warning anywhere. If your values carry decimals or negatives, parse them properly:
function toNumber(text) {
const cleaned = text
.replace(/[^0-9.,\-]/g, '') // drop currency symbols, units, spaces
.replace(/,(?=\d{3}\b)/g, '') // remove thousands separators
.replace(',', '.'); // treat a remaining comma as the decimal
return parseFloat(cleaned);
}Swap that into domSum in place of parseInt, and use .filter(Number.isFinite) so a stray empty cell drops out.
That still cannot tell German 1.234,56 from English 1.234, and neither can any locale-blind function, because the string is genuinely ambiguous. Look at three cells on the page and decide the locale yourself before you trust a total.
One more boundary. The DOM Standard defines querySelectorAll as returning "all element descendants of node that match selectors", and a shadow root's contents are not descendants of its host in the node tree; the spec keeps a separate "shadow-including descendant" concept for that. Content inside a web component or an iframe is invisible here. When the count comes back zero on a page that visibly has values, check for a shadow root before blaming the selector.
Wrap it as a bookmarklet to reuse across sites:
javascript:(function(){
const sum = [...document.querySelectorAll('span.hotel-num')]
.map(el => parseFloat(el.textContent.replace(/[^0-9.\-]/g,'')))
.filter(Number.isFinite)
.reduce((a,b)=>a+b,0);
prompt('Sum of matched elements:', sum);
})();Save it as a bookmark, edit the selector, and one click pops up the total. Bookmarklets are supposed to be exempt from a page's Content Security Policy: the CSP specification says policy "SHOULD NOT interfere with the operation of user-agent features like addons, extensions, or bookmarklets." Read the verb. A bookmarklet that dies quietly on a strict site is not necessarily your bug. For anything recurring, graduate to a real scraper.
Which tool for which job
| Situation | Reach for |
|---|---|
| Selecting elements by structure, class, or relationship | XPath or CSS selectors |
| "Everything between these two siblings", where the items are elements | CSS ~ plus :has(), or XPath sibling axes |
| "Everything between these two siblings", where the items are bare text | XPath //text() on a short list, an itersiblings() walk on a long one |
| More than about a thousand nodes between the landmarks | An itersiblings() walk. The sibling-axis predicate is quadratic |
| Pulling a pattern (amount, email, ID) out of text you already isolated | Regex, lazily quantified |
| A quick one-off total on a page you are viewing | $$ in the console, with a number parser that handles decimals |
| Parsing whole documents or nested markup | A DOM parser, never regex alone |
A mature scraper uses all of them: a parser holding the document, XPath or CSS to reach the node, regex to sharpen the field.
Versions and dates behind this article
Everything above was re-run on 13 August 2026. Versions and release dates come from each project's own PyPI record, read the same day:
| package | version | released |
|---|---|---|
| lxml | 6.1.1 | 18 May 2026 |
| parsel | 1.11.0 | 29 January 2026 |
| cssselect | 1.5.0 | 27 July 2026 |
| soupsieve | 2.9.2 | 7 August 2026 |
| beautifulsoup4 | 4.15.0 | 7 June 2026 |
| selectolax | 0.4.11 | 15 July 2026 |
| regex | 2026.7.19 | 19 July 2026 |
| elementpath | 5.1.4 | 8 August 2026 |
Code ran under Python 3.11.15 with libxml2 2.14.6, and Node 22.22.2 for the JavaScript. The quoted re behaviour matches the Python 3.14.7 documentation. The XPath 1.0 Recommendation is dated 16 November 1999, status updated October 2016, and remains the only version lxml implements.
Wrapping up
Flat, sequential HTML is the case clean-hierarchy tutorials skip, and both standard answers fail quietly on it. The sibling-axis expression is readable and correct on the toy fragment, returns an empty list the moment a wrapper div appears around the heading, drops characters when an item contains a <b>, and goes quadratic past a thousand nodes. The anchored regex is fast and returns one whitespace match instead of six items the moment the page writes <br> rather than <br/>. Walking siblings from the opening landmark avoids all of that and stays linear.
Keep the division of labour: a parser owns structure, regex owns patterns inside text, and CSS with :has() now covers the sibling case whenever the items are wrapped in elements. Test every expression against the parser your scraper runs, not the tree in dev tools. When the targets multiply and the markup keeps shifting, handing the upkeep to a managed web scraping service turns a maintenance treadmill back into structured data.
Assert the row count. Six of the seven failures above hand you a number rather than an error, and the count is the only thing that tells them apart.