CSS Selectors for Web Scraping: Complete Reference

Right-click a price in Chrome, choose Copy selector, paste the result into your scraper, and get back an empty list. The string Chrome handed you contains tbody. The HTML you downloaded does not. The browser inserted that element while parsing, because the HTML Standard says a tbody start tag "may be omitted if the first thing inside the tbody element is a tr element" and the parser supplies it anyway. Your Python parser, in most configurations, does not.

That is the whole problem in one line. The selector syntax is small and learnable in an afternoon. The gap between the tree you inspected and the tree your code queries is where the bugs live, and no amount of selector cleverness closes it.

We covered how markup becomes a tree of nodes in the DOM parser piece, along with walking that tree by hand. Hand-walking breaks on the first redesign, so web scraping addresses a node with one short expression instead. This is the reference for those expressions, plus the parts usually left out: what your specific library supports, what is quietly wrong, and what costs you hours on a large crawl.

Everything below was rechecked on 10 August 2026 against beautifulsoup4 4.15.0, soupsieve 2.9.2, parsel 1.11.0, cssselect 1.5.0, lxml 6.1.1 and Scrapy 2.17.0, with version numbers and release dates read from the PyPI API on that date. Every compatibility claim below comes from running the selector, not from reading someone's table.

The tree you query is not the tree you inspected

Four parsers, one snippet of sloppy markup. This is real HTML: unclosed paragraphs, a table without tbody, list items that never close.

<div class="wrap"><table><tr><td>cell</td></tr></table><p>one<p>two<ul><li>a<li>b</ul></div>

Same document, same selectors, three different answers. html.parser nests the second paragraph inside the first, so .wrap > p finds one element instead of two. Only html5lib inserts the implied tbody, which is why table > tr returns nothing there and everywhere else returns one. The BeautifulSoup documentation is blunt about why: html5lib "parses pages the same way a web browser does."

A selector copied from DevTools is written against the browser's tree. If it names tbody, it fails under lxml and html.parser and works under html5lib. The reverse holds too: a selector that works in your Python code can be untestable in the browser console you built it in.

Ghost elements run the other way. MDN puts it plainly: "The Node.childNodes property of the <template> element is always empty, and you can only access said nested content via the special content property." No browser selector reaches inside a template. Every Python path we tested reaches inside without a word:

<div id="live"><span class="price">$1</span></div>
<template id="tpl"><span class="price">$999</span></template>
<noscript><span class="price">$777</span></noscript>

soup.select("span.price") returns three matches under every parser we tried, and sel.css("span.price") returns three as well. A browser returns one. The $999 is a client-side template placeholder and the $777 sits behind <noscript>, and both land in your dataset looking exactly like a price.

It gets quieter than that. Under html.parser and lxml, BeautifulSoup matches the element inside <template> but returns an empty string for its text, since bs4 stores those strings as TemplateString and get_text() skips that type by default. Under html5lib the same code returns $999. The same scraper reports a missing price on one machine and a wrong price on another, depending on which optional dependency was installed. Declarative shadow DOM behaves the same way: <template shadowrootmode="open"> is a real shadow root in the browser and plain markup to your parser.

Two habits close most of this gap. Verify selector counts against the HTML your scraper downloaded rather than the rendered page. And when a count is off by exactly the number of cards on the page, look for a template before you touch the selector.

Basic selectors

These are the atoms everything else is built from.

A class and a tag glued together with no space is a logical AND: a.button means a link that also carries the class button. div.card.featured is a div with both classes.

span.price.current     /* <span class="price current"> */
li#first-item          /* <li id="first-item"> */

Class names now need escaping more often than they used to. Utility-first CSS put colons and slashes into class attributes, and a colon in a selector starts a pseudo-class. Tested on both engines:

.md:flex        /* SelectorSyntaxError in soupsieve, ExpressionError in cssselect */
.md\:flex       /* matches */
.lg\:w-1\/2     /* matches */
.2xl            /* SelectorSyntaxError in both: an identifier cannot start with a digit */
.\32 xl         /* matches; \32 is "2", and the space ends the escape */

Doubling backslashes inside a Python string is easy to get wrong, so the readable escape hatch is an attribute selector: [class~="md:flex"] matches the same element with no escaping, because inside quotes a colon is just a character.

Grouping

A comma joins several selectors into one list, and you get everything matching any of them:

h1, h2, h3             /* all headings of these three levels */
.price-new, .price-old /* both the new and the old price */

This is the cheapest hedge against a store that A/B-tests its markup. The cost is silence: a grouped selector that switched branches looks identical to one that never changed, so log which alternative matched when the difference reaches your data.

Combinators: relationships between elements

Combinators describe how elements sit relative to one another. This is the part that decides whether a selector survives a redesign.

Descendant (space)

A B matches B anywhere inside A, at any depth.

.product-card .price   /* .price anywhere inside the card */

Direct child (>)

A > B matches only when B is a direct child of A.

ul.menu > li           /* top-level menu items, not nested ones */

The distinction earns its keep on menus, tables and comment threads, anywhere a structure nests inside itself. .menu li catches submenu items; .menu > li does not. The descendant version also survives a redesign that adds one more wrapper div, which sounds like an advantage until the new wrapper is a "related products" carousel and your price selector starts matching six prices.

Choose deliberately. > fails loudly when the structure changes, a space fails quietly by matching too much, and the loud failure is the one you can fix.

Adjacent sibling (+)

A + B matches the B that comes immediately after A under the same parent.

h2 + p                 /* the first paragraph right after a heading */
.label + .value        /* the value sitting immediately after a label */

Specification tables are the classic use: a dt/dd or label/value pair where only the label is reliably named.

General sibling (~)

A ~ B matches every B after A under a shared parent, adjacent or not.

.in-stock ~ .price     /* all prices coming after the "in stock" label */

Attribute selectors

When classes are generated, attributes are usually what is left.

a[href^="/catalog/"]           /* links within the catalog */
img[src$=".webp"]              /* images in webp format */
div[data-testid*="product"]    /* containers with a product marker */
input[name="csrf"]             /* the hidden token field */

Attribute selectors are what make ecommerce price monitoring survivable. Large retailers expose the same numbers in data-price, data-sku and data-currency, and those names change on a schema migration rather than on every front-end deploy.

Attribute values are case-sensitive, except for the forty-three that are not. The HTML Standard requires attribute selectors in HTML documents to treat the values of a fixed list of attributes as ASCII case-insensitive. The list runs to 43 names, type, rel, method, lang, dir, target, scope, disabled and selected among them. A browser follows that rule. Your parser probably does not:

html = '<input type="SUBMIT" rel="NOFOLLOW">'
BeautifulSoup(html, "html.parser").select('input[type="submit"]')   # 1 match
Selector(text=html).css('input[type="submit"]')                     # 0 matches
BeautifulSoup(html, "html.parser").select('a[rel="nofollow"]')      # 0 matches

soupsieve implements the rule for exactly one attribute: its parser hardcodes a case-insensitive flag when the attribute name lowercases to type, and for nothing else. cssselect implements it for none, since it compiles to XPath 1.0 and XPath string comparison is case-sensitive. So input[type="submit"] is a portability bug in both directions, and a[rel="nofollow"] under-matches everywhere outside a browser.

The i flag has its own portability problem: soupsieve accepts [type="submit" i] and [type="SUBMIT" s], and cssselect raises SelectorSyntaxError on both. On parsel, lowercase the value in Python instead.

Pseudo-classes: position and structure

Positional

li:first-child         /* the first child */
li:last-child          /* the last child */
tr:nth-child(2)        /* the second row, counting from 1 */
tr:nth-child(odd)      /* odd rows */
tr:nth-child(even)     /* even rows */
tr:nth-child(3n)       /* every third */
li:nth-last-child(1)   /* the last child, counted from the end */

The an+b form is more flexible than the keywords suggest: nth-child(n+3) is everything from the third onward, and nth-child(-n+3) is the first three. Both are one-indexed.

Positional selectors are the most fragile thing in this article and the most expensive to run. One inserted promo row shifts every index by one. Use them where order is the data, as in a results table, and not as a way to say "the price is the second span."

By tag type

:nth-child counts all siblings; :nth-of-type counts only siblings with the same tag.

p:first-of-type        /* the first paragraph among siblings */
img:nth-of-type(2)     /* the second image in order */
span:last-of-type      /* the last span */

Neither counts by class, which is the misreading that produces silent bugs. Take a list whose first and third items carry the class and whose second does not. li.x:nth-child(2) matches nothing at all, because the pseudo-class asks whether this element is the second child, not whether it is the second matching one. Selectors Level 4 added the form that does what people expect, and only one of our two engines takes it:

soup.select("li:nth-child(2 of .x)")   # matches the third li: the second one with class x
sel.css("li:nth-child(2 of .x)")       # cssselect: SelectorSyntaxError

Negation and grouping

input:not([type="hidden"])   /* all fields except hidden ones */
li:not(.sold-out)            /* products not in the "sold out" state */
:is(h1, h2, h3) > a          /* a link directly inside any of three headings */
:where(.card, .tile) .price  /* the same, with zero specificity */

:is() and :where() factor out repetition and behave identically in a scraper, since specificity only matters for styling. Both work in soupsieve and in cssselect, which has supported them since version 1.2.0 in October 2022 and fixed a batch of edge cases in 1.5.0 on 27 July 2026.

:not() splits along the same line. Simple arguments work everywhere; the list form li:not(.x, #z) and the complex form div:not(.card em) work in soupsieve and raise SelectorSyntaxError in cssselect. Chaining does the same job portably: li:not(.x):not(#z).

Note: pseudo-elements like ::before and ::after describe style-generated content that never appears in the source HTML, so they are dead weight in a scraper. The syntax was not wasted, though. Scrapy, parsel and Puppeteer all borrowed pseudo-element notation for their own extensions, which is why ::text and ::-p-text() look the way they do.

:has() ended the "CSS cannot select a parent" era

Two claims travel through every article on this subject, this article's own earlier version included: that CSS selectors cannot climb up to a parent, and that they cannot match on text. Both were true once. Neither is true now, and repeating them sends people to XPath for work CSS does today.

:has() selects an element by what it contains. MDN records it as Baseline widely available since December 2023: Safari 15.4 in March 2022, Chrome and Edge 105 in August 2022, Firefox 121 in January 2024.

div.product-card:has(.badge--sale)      /* cards that contain a sale badge */
div.product-card:not(:has(.out-of-stock))
tr:has(> td.price)                      /* rows with a price cell as a direct child */
li:has(+ li.active)                     /* the item just before the active one */

All four run in BeautifulSoup and in Scrapy today: cssselect shipped :has() in 1.2.0 and improved it in 1.5.0, and soupsieve has had it for years. The translation explains why it is cheap rather than expensive. div:has(em) becomes descendant-or-self::div[descendant::em], an ordinary XPath predicate. On a synthetic page of 3,000 cards, cssselect ran div.card:has(em) in 4 ms against 10 ms for the plain div.card that matched three times as many elements.

Two limits are worth carrying: Selectors Level 4 is still a Working Draft, dated 22 January 2026, and it flags :has() as at-risk. And :has() cannot nest inside another :has().

Matching on text

Neither engine can match text with standard CSS, and both ship an extension that does.

soup.select('span:-soup-contains("In stock")')    # soupsieve
sel.css('span:contains("In stock")')              # cssselect

soupsieve still accepts the older :contains() spelling and emits a FutureWarning pointing at :-soup-contains(). cssselect only knows :contains(). Neither name is standard CSS, and code that moves between the two libraries has to change the string.

The browser-driving tools went further. Playwright adds :has-text() for a substring anywhere inside an element, :text() for the smallest element containing it, :text-is() for an exact match and :nth-match() for the n-th result. Puppeteer adds ::-p-text(), ::-p-aria() for the computed accessible name and role, and ::-p-xpath(). For static HTML on the JavaScript side, Cheerio inherits :contains() from css-select and jQuery-style extras like :first and :eq() through cheerio-select.

Playwright's own documentation argues against leaning on CSS at all: "We recommend prioritizing user-visible locators like text or accessible role instead of using CSS that is tied to the implementation and could break when the page changes." That is written for tests, where the page is yours, and it transfers: on a site that reskins its classes twice a year, the accessible name is often the most stable handle available.

What your library actually supports

Run on 10 August 2026, soupsieve 2.9.2 through BeautifulSoup 4.15.0 against cssselect 1.5.0 through parsel 1.11.0. Every row was executed on the same markup.

The pattern is easy to remember. soupsieve tracks the draft specification and is the more complete engine; cssselect targets CSS3 plus a few Level 4 additions, because everything it accepts must survive translation into XPath 1.0.

:scope fixes a bug people work around for years. A selector run on a sub-element still searches its whole subtree, so card.select("a") finds links at any depth. :scope anchors to the element you called it on, in both engines:

card.select(":scope > a")        # BeautifulSoup: direct children only
card.css(":scope > a")           # parsel: same

Building a complex selector

Real expressions stack the pieces. Pulling the current price out of a catalog card:

div.product-card:has(a[href^="/product/"]) span.price:not(.price--old)

Read it right to left, the way an engine does: a span with class price that is not the old struck-through price, sitting inside a product card that contains a link to a product page. The :has() clause is doing real work. Without it the selector also matches the "recently viewed" strip, which uses the same card markup and links elsewhere. Reading right to left is also what keeps selectors short, since the rightmost part carries the specificity and everything left of it only rules out lookalikes.

Tools: where this works

Python, BeautifulSoup. The CSS engine is Soup Sieve, a hard dependency since bs4 declares soupsieve>=1.6.1.

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "lxml")

for card in soup.select("div.product-card"):
    title = card.select_one("h3.title").get_text(strip=True)
    price = card.select_one("span.price:not(.price--old)").get_text(strip=True)
    print(title, price)

select_one() returns None when nothing matches, so the chained .get_text() above raises AttributeError on the first card that differs from the rest: one crash somewhere in hour three of a long run.

Python, parsel and Scrapy, a wrapper over lxml:

from parsel import Selector

sel = Selector(text=html)
prices = sel.css("span.price::text").getall()
links = sel.css("a.product-link::attr(href)").getall()

::text and ::attr(name) are parsel extensions, and Scrapy's documentation warns that they "will most probably not work with other libraries like lxml or PyQuery." *::text collects descendant text nodes, and the has-class() XPath extension exists because class matching in raw XPath is a concat-and-contains incantation.

One trap here costs an afternoon. Sub-selectors keep their context for CSS and lose it for absolute XPath:

card = sel.css("div.card")[0]
card.css("a::attr(href)").getall()      # ['/1']  - stays inside the card
card.xpath("//a/@href").getall()        # ['/1', '/2']  - escapes to the document
card.xpath(".//a/@href").getall()       # ['/1']  - the leading dot is the fix

Browser console. Chrome documents $$("span.price") as equivalent to Array.from(document.querySelectorAll()), with an optional start node. Two caveats: a page that ships jQuery overwrites $, and the helpers exist only in the console. Per the DOM standard, querySelectorAll returns a static list while getElementsByClassName returns a live one, so removing nodes inside a loop over the live version skips half of them.

Playwright and Selenium. Both run selectors through the browser, so anything the browser supports works, including :has(). Playwright's CSS engine also pierces open shadow DOM, which its docs contrast with XPath: "XPath does not pierce shadow roots." Selenium's By.CSS_SELECTOR hands the string to the page and does not know Playwright's text extensions.

price = page.locator("div.product-card span.price").first.inner_text()

What breaks at ten thousand pages

Measured on 10 August 2026 on a single shared two-core container, Intel Xeon at 2.10 GHz, Python 3.11.15, against https://pypi.org/simple/lxml/ saved to disk: 2,364,601 bytes, 4,700 <a> elements, all siblings under one parent. Median of three to seven runs per row. Your absolute numbers will differ; the ratios will not.

Parsing the document once:

Choosing lxml as BeautifulSoup's parser does not buy you lxml's speed. Both rows above parse with the same C library; the difference is the Python object tree BeautifulSoup builds on top, and it costs eight times the parse. Across ten thousand pages that is 4.5 minutes against 36 minutes of pure parsing, before a single selector runs. The BeautifulSoup documentation says so itself: "if CSS selectors are all you need, you should skip Beautiful Soup altogether and parse the document with lxml: it's a lot faster."

Querying the parsed tree:

Sixteen seconds for one selector on one page. Nothing is broken; the selector is quadratic in the number of siblings. cssselect prints the reason if you ask it to compile the expression:

descendant-or-self::a[(count(preceding-sibling::a) +1) mod 3 = 0]

count(preceding-sibling::a) is evaluated for every candidate, so a list of n siblings does n²/2 comparisons. Doubling the siblings quadruples the time, which is exactly what the measurements show:

The last column is the fix, and it is not a selector. Match once and slice the list in Python: positional filtering is a list operation, and asking a tree engine to do it turns a linear job into a quadratic one. Sixteen seconds a page across ten thousand pages is forty-five hours, the difference between a nightly price monitoring run finishing before morning and not finishing at all.

Two smaller findings from the same run. soup.find_all("a", href=lambda h: h and ".whl" in h) took 14 ms against 39 ms for the equivalent soup.select('a[href*=".whl"]'), so in BeautifulSoup the CSS path is the convenience path, not the fast one. And in parsel, a[href*="cp313"] and the descendant-or-self::a[@href and contains(@href, 'cp313')] it compiles to both landed at 5 ms. Any article ranking CSS against XPath in Scrapy is comparing a thing to itself.

Where CSS selectors stop

State-dependent pseudo-classes cannot work on a downloaded file. Soup Sieve lists 25 of them as non-applicable, including :hover, :focus, :visited, :target and :host, because they "cannot be implemented outside a live, browser environment." They parse without complaint and match nothing.

:visible is not CSS. It exists in Playwright and in jQuery, and it means "has a non-empty bounding box," which requires layout. Static parsers have no layout, so an element hidden with display: none is still in the tree and still matches your selector.

Shadow DOM stops most tooling. document.querySelectorAll does not cross an open shadow boundary. Playwright's CSS engine pierces open roots and Puppeteer offers the >>> deep-descendant and >>>> deep-child combinators for the same job. Closed roots stay closed everywhere.

Text nodes are not elements. Pulling a value out of <span>Price: <b>$10</b></span> without the label needs ::text in parsel, .contents handling in BeautifulSoup, or regular expressions once the value is a string.

CSS selectors vs XPath

CSS covers more than it used to. With :has() for ancestry and :-soup-contains() or :contains() for text, the two classic reasons to reach for XPath are gone in Python, and in parsel the CSS you write becomes XPath anyway.

What is left to XPath is narrow and real: axes other than descendant and sibling, preceding and following in particular; conditions that combine one node's position with another node's text; and expressions that compute rather than select, since XPath has string and number functions and CSS has none. The honest default is CSS until it gets ugly, then XPath for the one awkward field.

Applying selectors in price monitoring

A typical ecommerce price monitoring job runs the same four steps per site:

1. Catalog listing. div.catalog__item finds every card on the results page. Compare the count against the number the site prints in its own "N products" label. A mismatch means a template ghost or a lazy-loaded tail, and both are cheaper to find now than after a week of data.
2. Product link. a[href^="/product/"]::attr(href) pulls the URL from each card.
3. Price and availability. [data-price]::attr(data-price) beats a class-based selector when the attribute exists, since it usually carries an unformatted number and skips currency parsing. .availability:not(.out-of-stock) confirms the item is for sale.
4. Old price and discount. span.price--old catches the struck-through price, and :not(.price--old) on the main selector keeps the two apart. Getting this pair backwards is the most common source of a feed that reports a permanent 30% discount.

Competitors use different markup, so each site gets its own selector set, and the scraping engine applies them on a schedule and writes the results into one comparable table. The selectors are the part that rots. Make a zero-match run an alert rather than an empty row.

Common mistakes

Latching onto generated classes. .css-1ab2c3 from a bundler, .styles__price___2Kx9f from CSS Modules, hashed Vue scope attributes: all of them change when the build changes, which can be twice a week. Prefer data-* attributes, meaningful ids, ARIA roles and structure. When a generated class is the only handle, a prefix match like [class^="styles__price"] survives the hash rotation that an exact match does not.

Trusting the selector DevTools generated for you. It is built from the browser's tree, it usually contains tbody and a chain of :nth-child, and both halves are wrong for a static parser.

Testing only against the rendered page. view-source: and a saved file agree with your parser; the Elements panel agrees with the browser.

Not checking for zero. select_one() returns None and .css() returns an empty list, and both look like a working scraper until you read the output.

Letting a positional selector do work a slice should do. See the timings above.

Assuming a selector that works in one library works in the next. The matrix above has six rows where soupsieve and cssselect disagree. Four of those constructs raise SelectorSyntaxError in cssselect instead of returning nothing, which at least fails loudly; the case-sensitivity gap is the one that stays quiet. Keeping a few hundred selectors current across sites that redesign on their own schedule is most of the standing cost of managed extraction, and none of it is visible in the syntax.

The reference above is the whole syntax; the sections around it are what the syntax does not tell you. In the next piece we take the other route out of the DOM and pull data out of raw text with regular expressions, for the fields where structure never was the answer.