Regex for Web Scraping: Extracting Clean Data from Text

This is the third piece in the series. In the DOM parser article we learned to turn a page into a tree of nodes, and in the piece on CSS selectors we learned to address the exact elements we want inside that tree. But once a node is found, its contents are often "dirty" text: "Price: $1,299.50 (was $1,599)", a phone number padded with symbols, a SKU tangled up with a description. Pulling exactly the number or code out of a string like that is what regular expressions are for.

A regular expression (regex) is a pattern describing a set of strings. The regex engine scans the text and finds fragments that match. In web scraping, regex is an indispensable tool for the final stage of cleaning and extracting data — the "last mile" between a raw node and a clean value.

When you need regex — and when you don't

First, an important warning. You do not parse whole HTML with regular expressions. HTML is a nested, non-regular structure; trying to take it apart with a single pattern leads to brittle, unreadable code. For navigating markup there are DOM parsers and CSS selectors from the previous articles.

Regex shines in a different role — processing already-extracted short text:

• pull the numeric price out of a string with currency and spaces;
• normalize a phone number or email;
• extract a SKU, part number, date, or discount amount;
• parse parameters out of a URL or the text of a <script> tag.

The rule is simple: take the structure with selectors, the value inside a node with regex.

Basic syntax

Most characters in a pattern stand for themselves: the pattern price finds the substring "price." The power comes from special characters (metacharacters).

To use a metacharacter as an ordinary one (for example, a dot inside a number), escape it with a backslash: \., \$, \(.

Character classes

Square brackets define a set of allowed characters in one position:

[aeiou]           one of the listed vowels
[0-9]             any digit (same as \d)
[a-zA-Z]          any Latin letter
[A-Fa-f0-9]       a single hex digit (handy for color codes, IDs)
[^0-9]            any character EXCEPT a digit (^ inside brackets = negation)
[\d.,]            a digit, dot, or comma

Ranges (a-z, 0-9) and enumerations combine freely. A ^ at the start of a class inverts the set.

Quantifiers: how many repetitions

A quantifier states how many times the preceding element repeats.

\d+               one or more digits: "1299"
\d{1,3}           one to three digits (a thousands group)
colou?r           "color" and "colour"

Greedy vs. lazy

By default, quantifiers are greedy — they grab as much as possible. Adding ? makes them lazy — they grab the minimum:

".*"              greedy: from the first quote to the LAST one in the string
".*?"             lazy: from the first quote to the NEAREST one

This is one of the most common traps: a greedy pattern "eats" too much. In practice, extracting short values almost always calls for the lazy variant or a narrower character class instead of ..

Anchors and boundaries

Anchors don't match characters — they pin the pattern to a position:

^Total            ^ — start of the string
end$              $ — end of the string
^\d+$             the string consists entirely of digits
\bSKU\b           "SKU" as a standalone word, not part of another

Anchors guard against false positives: ^\d+$ guarantees a cell holds only a number, not "12 pcs."

Groups

Parentheses ( ) group part of a pattern and capture the matched fragment so you can retrieve it later.

(\d+)[.,](\d+)        the integer and fractional parts of a number, separately

Groups are numbered left to right: group 1 is (\d+) before the separator, group 2 comes after.

Non-capturing groups

If you need the parentheses only for grouping (say, under a quantifier) but don't want the result, use (?: ):

(?:https?://)?(\S+)   "http://" or "https://" optional, without saving it

Named groups

To avoid counting numbers, give a group a name with (?P<name>...) (in Python) or (?<name>...):

(?P<dollars>\d+)[.,](?P<cents>\d{2})

Then you access it with match.group("dollars") — readable and resilient to reordering the groups.

Backreferences

Inside a pattern you can refer to an already-captured group with \1, \2:

<(\w+)>.*?</\1>       a matched pair of tags: opening and closing share a name
(["']).*?\1           a string in single or double quotes

\1 requires the second position to hold the same character that matched the first group.

Lookahead and lookbehind

Lookaround constructs check the context without including it in the result — very useful for "latching onto" a label but returning only the value.

(?<=\$)[\d,]+         a number PRECEDED by a dollar sign — returns only the number
(?<=Price:\s)\d+      a number after the label "Price: " — returns only the number
\d+(?!\d)             the last digit of a number

Lookbehind is especially handy when you need a value after a fixed label without dragging the label into the result.

Flags

Flags change the behavior of the whole pattern:

re.findall(r"price", text, re.I)   # "Price", "PRICE", "price"

Practical examples for scraping

Let's assemble typical patterns in Python (the re module). Useful functions: re.search (first match), re.findall (all, as a list), re.sub (replace), re.finditer (an iterator with groups).

Price from a "dirty" string:

import re

text = "Price: $1,299.50  (was $1,599.00)"

# Grab the first number, ignoring the thousands separators (commas)
m = re.search(r"(\d[\d,]*)(?:\.(\d{2}))?", text)
dollars = m.group(1).replace(",", "")   # "1299"
cents = m.group(2) or "00"              # "50"
price = float(f"{dollars}.{cents}")     # 1299.5

All prices in the text (new and old):

prices = re.findall(r"(?<=\$)[\d,]+(?:\.\d{2})?", text)
# ['1,299.50', '1,599.00']  → then strip the commas
clean = [float(p.replace(",", "")) for p in prices]   # [1299.5, 1599.0]

SKU / part number:

sku = re.search(r"\bSKU[:\s]+([A-Z0-9\-]+)", text)

Phone (normalize to digits):

digits = re.sub(r"\D", "", "+1 (415) 123-4567")   # "14151234567"

Email:

emails = re.findall(r"[\w.+-]+@[\w-]+\.[\w.-]+", text)

Extracting a JSON value from an inline script:

# On a product page the price often sits in a <script> as window.__DATA__ = {...}
m = re.search(r'"price"\s*:\s*"?(\d+(?:\.\d+)?)"?', script_text)

Regex paired with DOM and CSS

Regex works best not instead of, but alongside the tools from the previous articles:

from bs4 import BeautifulSoup
import re

soup = BeautifulSoup(html, "html.parser")

# 1) a CSS selector grabs the right node
raw = soup.select_one("span.price").get_text(strip=True)

# 2) regex cleans the text down to a number
price = int(re.sub(r"\D", "", raw))

The selector answers "where," the regex answers "what exactly out of that text." This separation makes a scraper both precise and readable.

Use in price monitoring

In price monitoring, regular expressions close the "last mile" of data extraction:

• Normalizing the price to one numeric format: strip the thousands separators, the currency, and standardize the decimal separator — otherwise values from different sites can't be compared.
• Parsing a discount out of strings like "−23%" or "save $30."
• Cleaning availability: pull the count \d+ from "only 4 left."
• Product identifiers — SKU, EAN/barcode (\b\d{13}\b), part number — to match the same product across different sellers.
• Data from scripts: many stores put the price in JSON inside <script type="application/ld+json"> or in inline variables — regex pulls the field out without fully parsing the JS.

The "selector → regex" combo is the working foundation of a website scraping engine: it crawls product cards on a schedule, pulls nodes with CSS, and reduces their contents to clean numbers with regular expressions.

Performance and pitfalls

• Compile frequently-used patterns — call re.compile(...) once instead of re-parsing inside a loop over thousands of products.
• Avoid catastrophic backtracking. Nested ambiguous quantifiers like (\d+)+ or (.*)* can "hang" on a specially (or accidentally) crafted string. Keep character classes narrow and prefer lazy quantifiers over greedy ones where it makes sense.
• Don't over-engineer. If a pattern becomes unreadable, split the task into two simple regexes or fall back to selectors — the value you want may sit in a separate data- attribute that's simpler to grab directly (see the CSS selectors article).
• Test on real examples. Prices are written in many ways: 1,990, 1990.00, $1,990, from 1990. Run your pattern against a set of live strings from different stores.

Bottom line

Regular expressions are a tool for extracting values from text, not for parsing structure. Metacharacters and classes describe which characters are expected, quantifiers how many of them, groups and lookaround how to isolate the fragment in context. Paired with DOM parsing and CSS selectors, they form a complete scraping pipeline: find the node, grab the text, reduce it to clean data. That same combo is what powers reliable price monitoring, where comparable numbers must be pulled regularly from competitors' varied markup.

That wraps up the core series on scraping tools: the DOM gives you the tree, CSS selectors give you addressing, and regular expressions give you the final data cleanup. If you'd rather have all three run for you as a service, scraping.pro handles custom data extraction end to end.