How to Parse Messy or Wrongly Encoded HTML

The HTML you scrape in the wild is rarely clean. Tags aren't closed, quotes are missing, the same page mixes two character encodings, and prices arrive as £220.00 instead of £220.00. Browsers hide all of this because they're extraordinarily forgiving — but your scraper isn't, unless you make it so. This guide covers the two problems that break extraction most often: wrong encodings and malformed markup ("tag soup"), and how to parse messy HTML reliably before you pull data out of it.

There are two independent failure modes, and it helps to treat them separately:

1. Encoding problems — the bytes are misinterpreted, so text looks like £ or ’, or entities show up literally.
2. Structure problems — the markup is broken, so a naive parser mis-nests elements and your selectors miss.

Part 1: Fixing encoding problems

How text encoding actually resolves

When a scraper downloads a page it gets bytes, and it needs the right character set to turn them into text. That charset can be declared in three places, checked roughly in this order:

1. The HTTP Content-Type response header (charset=utf-8).
2. A <meta charset="..."> tag near the top of the HTML.
3. A byte-order mark (BOM) at the very start of the file.

When these disagree — or lie — you get mojibake: € rendered as €, or ' as ’. That's the signature of UTF-8 bytes decoded as Windows-1252 (or vice versa).

Let the tools detect it

In Python, don't trust requests' guess from the headers alone. Either hand the raw bytes to a parser that reads the page's own <meta> declaration, or detect the encoding explicitly:

import requests
from bs4 import BeautifulSoup

resp = requests.get(url, timeout=15)

# Best: give BeautifulSoup the bytes and let it use the meta charset + a sniffer
soup = BeautifulSoup(resp.content, "lxml")   # note .content (bytes), not .text

# Or detect explicitly with charset-normalizer (ships with requests)
from charset_normalizer import from_bytes
best = from_bytes(resp.content).best()
html = str(best)   # decoded with the detected encoding

Passing resp.content (bytes) instead of resp.text (a string requests already decoded, possibly wrongly) is the single highest-leverage fix — BeautifulSoup's Unicode, Dammit then reconciles the header, the meta tag, and a statistical sniff for you. charset-normalizer is the modern, maintained successor to the old chardet library.

If you know the encoding, just say so:

html = resp.content.decode("windows-1252", errors="replace")

errors="replace" swaps undecodable bytes for � instead of crashing — useful when a single stray byte would otherwise kill the whole parse.

Decoding HTML entities

The other half of the encoding story is HTML entities — the escaped sequences like £ (£), & (&), or é (é). Suppose you're extracting a price and the markup is:

<div>cost: &#163;220.00</div>

A good parser decodes entities for you when you read the text of a node — soup.select_one("div").get_text() returns cost: £220.00 directly. But if you're handed an already-encoded string, decode it explicitly.

Python (standard library):

import html
print(html.unescape("cost: £220.00"))   # -> cost: £220.00

PHP:

echo html_entity_decode("cost: £220.00", ENT_QUOTES | ENT_HTML5, "UTF-8");

JavaScript (browser or with the he library in Node):

import { decode } from "he";
decode("cost: £220.00");   // -> "cost: £220.00"

Now you can strip the currency symbol and parse the number cleanly:

import re, html
raw = html.unescape("cost: £220.00")
amount = float(re.sub(r"[^\d.]", "", raw))   # 220.0

Watch for double-encoding — strings like £ where the & itself was escaped. You may need to unescape twice, or better, fix the upstream step that escaped it once too often.

Part 2: Taming malformed markup

Encoding fixed, the second problem is structure. Scraped HTML routinely has unclosed tags, stray <, attributes without quotes, tables missing <tbody>, and elements nested where the spec forbids. The wrong parser will silently build a broken tree and your selectors return nothing.

Choose a lenient parser

In Python's BeautifulSoup you pick the underlying parser, and they differ sharply in how they handle breakage:

# Fast and forgiving — the everyday default
soup = BeautifulSoup(resp.content, "lxml")

# When markup is truly broken, parse it the way a browser would
soup = BeautifulSoup(resp.content, "html5lib")

The rule of thumb: reach for lxml for speed, and drop to html5lib when a page is so mangled that lxml mis-parses it. html5lib follows the WHATWG HTML5 parsing algorithm — the same error-recovery rules browsers use — so if Chrome can read it, html5lib can too, just slowly.

Other languages

• JavaScript / Node: cheerio (built on the tolerant parse5/htmlparser2) for a jQuery-like API; jsdom when you need a fuller DOM.
• PHP: DOMDocument is lenient if you silence its warnings, or use Symfony's DomCrawler / Goutte on top of it:

$doc = new DOMDocument();
libxml_use_internal_errors(true);   // suppress warnings on broken HTML
$doc->loadHTML($messyHtml);
libxml_clear_errors();

• Go: golang.org/x/net/html implements the same forgiving HTML5 algorithm.

Query with selectors, not regex

Once the messy HTML is parsed into a tree, extract with CSS selectors or XPath — never regular expressions. Regex on HTML is fragile: it breaks on the first unexpected attribute, nested tag, or line break, which is exactly what messy markup throws at you. The whole point of parsing first is to get a structure you can query robustly.

# Robust: query the parsed tree
price = soup.select_one("span.price").get_text(strip=True)

More on picking between the two query languages is in XPath vs CSS selectors, and on the parsing libraries themselves in the BeautifulSoup tutorial and the wider Python web scraping guide.

A defensive parsing checklist

1. Fetch and work with bytes, then let the parser resolve the encoding (resp.content, not resp.text).
2. If mojibake persists, detect with charset-normalizer or force a known encoding with errors="replace".
3. Decode entities with html.unescape (or html_entity_decode / he), and watch for double-encoding.
4. Parse with lxml, escalating to html5lib for badly broken pages.
5. Extract with CSS/XPath selectors, not regex.
6. Guard every lookup — a missing node should return None and be handled, not crash the run.

FAQ

Why does my scraped text show £ or ’? That's mojibake — UTF-8 bytes decoded as Windows-1252 (or the reverse). Decode from the correct charset; passing raw bytes to BeautifulSoup usually fixes it automatically.

How do I decode HTML entities like £? html.unescape() in Python, html_entity_decode() in PHP, or the he library in JavaScript. A proper parser also decodes them when you read a node's text.

Which parser is best for broken HTML? lxml for speed in most cases; html5lib when the markup is severely malformed, because it recovers errors exactly like a browser.

Should I ever parse HTML with regex? For pulling structured data, no — parse into a tree and use CSS or XPath. Regex only makes sense for tiny, well-defined snippets you fully control.

Encoding quirks and broken markup are a big part of what makes scraping at scale finicky. If you'd rather receive clean, correctly decoded, structured data instead of debugging tag soup, scraping.pro handles it as a web scraping service.