HTML Parser Guide: Extract Links and Text from Pages

Parsing is the act of pulling the data you need out of web pages: breaking down HTML markup, collecting links, and isolating text content. At first glance it looks simple, but in practice there's almost always a catch — messy markup, dynamic content, blocks, and, most painful of all, encoding problems. An HTML parser is the tool that turns a raw page into something you can query, and knowing how HTML parsing really works is what separates a scraper that survives a redesign from one that breaks overnight.

In this guide we'll walk through three basic scenarios (HTML, links, text), the typical problems, and working examples in several languages.

1. Parsing HTML

Parsing HTML means building a tree of elements (the DOM) from a string of markup — a tree you can navigate and select nodes from. Most people work through:

• CSS selectors — .product .price, div#content > p;
• XPath — //div[@class="product"]//span[@class="price"];
• regular expressions — acceptable only for very simple cases; you shouldn't use them for real HTML (HTML is not a regular language).

The key idea: don't parse HTML with regex. Use a dedicated parser that can repair "broken" markup the same way a browser does.

2. Extracting links from HTML

Collecting links is the foundation of any crawler. It usually involves:

1. Selecting every <a> tag with an href attribute.
2. Resolving relative links to absolute ones (/page → https://site.com/page).
3. Filtering: dropping mailto:, tel:, javascript:, # anchors, and duplicates.
4. Normalizing the URL (host case, trailing slash, query-parameter order).

The classic beginner mistake is treating href as a ready-made absolute address. On real sites, links can be relative, protocol-relative (//cdn.site.com/...), or affected by a <base href> tag.

3. Turning HTML to text

Extracting text means getting the "human" content without tags. Here it's important to:

• strip <script>, <style>, and hidden elements;
• handle spaces, line breaks, and non-breaking spaces ( ) correctly;
• decode HTML entities (& → &, é → é, « → «);
• where possible, separate the main content from navigation, ads, and the footer (the "content extraction" problem).

4. Problems you'll run into

"Dirty," invalid HTML. Unclosed tags, wrong nesting, missing quotes. A good parser forgives these mistakes; regex doesn't.

Dynamic content (JavaScript). If data loads through AJAX/JS, it isn't in the source HTML. You'll need either a headless browser (Selenium, Playwright, Puppeteer) or a direct call to the API/XHR requests the page makes.

Volatile layout. Selectors break during a redesign. It's worth choosing stable signals (id, semantic attributes, microdata) over long, brittle chains of classes.

Anti-bot protection. CAPTCHAs, User-Agent checks, rate limiting, IP blocks, Cloudflare. You'll need pauses between requests, rotation of headers/proxies, and respect for robots.txt and common sense.

Encodings. The most common headache when scraping non-English or older regional sites — there's a dedicated section on it below.

Performance. At scale, what matters is async I/O, connection pools, and streaming (SAX-style) parsing instead of loading the whole DOM into memory.

Legal and ethical points. A site's terms of use, personal data, copyright, and the load you put on someone else's server.

5. Working with encodings and non-ASCII text

Most garbled-text problems — mojibake, where café shows up as café — come down to one thing: the bytes were read in a different encoding than the one they were written in.

On the modern web you'll mainly meet:

• UTF-8 — the modern standard and the default for the new web;
• Windows-1252 and ISO-8859-1 (Latin-1) — legacy single-byte encodings for Western European text, still alive on old sites;
• Language-specific legacy encodings — Shift_JIS / EUC-JP (Japanese), GBK / GB2312 (Chinese), EUC-KR (Korean), Windows-1251 (Cyrillic). You'll hit these when scraping older regional or non-English sites.

How to detect the right encoding

The priority order, the same way a browser does it:

1. The HTTP header Content-Type: text/html; charset=windows-1252.
2. The meta tag in the HTML: <meta charset="utf-8"> or <meta http-equiv="Content-Type" content="text/html; charset=windows-1252">.
3. BOM (Byte Order Mark) at the start of the file — for UTF.
4. Auto-detection from byte statistics (the chardet, charset-normalizer libraries).

An important subtlety: you can't detect the encoding from an already-decoded string. The encoding is determined from the raw bytes of the response, and only then is the content decoded into a string.

Typical mistakes

• Decoding Latin-1 as UTF-8 → an error or "garbage."
• The charset declared in the header doesn't match reality — the server "lies."
• Mixed encodings on a single page.
• Double decoding/encoding (mojibake).
• Forgetting to specify the encoding when writing the result to a file or database.

The universal recipe: read bytes → detect the encoding → decode to Unicode → work only in Unicode inside your program → encode to UTF-8 on the way out.

6. Implementation examples in different languages

In every example the task is the same: load a page, correctly detect the encoding, collect all links, and extract the title text.

Python — requests + BeautifulSoup

The most popular combo. requests tries to detect the encoding itself, and BeautifulSoup digests "dirty" HTML well.

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

url = "https://example.com"
resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})

# Detect the encoding from the content, not just the header
resp.encoding = resp.apparent_encoding  # uses charset-normalizer/chardet

soup = BeautifulSoup(resp.text, "lxml")  # the lxml parser is fast and robust

# Text extraction
title = soup.title.get_text(strip=True)
print("Title:", title)

# Link extraction: relative -> absolute, with filtering
links = set()
for a in soup.select("a[href]"):
    href = a["href"].strip()
    if href.startswith(("mailto:", "tel:", "javascript:", "#")):
        continue
    links.add(urljoin(url, href))

print(f"Links found: {len(links)}")

If you need to detect the encoding manually from the bytes:

import charset_normalizer

raw = resp.content                      # raw bytes
best = charset_normalizer.from_bytes(raw).best()
html = str(best)                        # already a correct Unicode string

JavaScript / Node.js — axios + cheerio

cheerio is a jQuery-like API for the server. It's best to detect the encoding explicitly with iconv-lite, since Node expects UTF-8 by default.

const axios = require("axios");
const cheerio = require("cheerio");
const iconv = require("iconv-lite");
const chardet = require("chardet");

(async () => {
  const url = "https://example.com";
  // Get raw bytes
  const { data } = await axios.get(url, {
    responseType: "arraybuffer",
    headers: { "User-Agent": "Mozilla/5.0" },
  });

  // Detect the encoding and decode
  const encoding = chardet.detect(data) || "utf-8";
  const html = iconv.decode(Buffer.from(data), encoding);

  const $ = cheerio.load(html);

  // Text
  console.log("Title:", $("title").text().trim());

  // Links
  const links = new Set();
  $("a[href]").each((_, el) => {
    const href = ($(el).attr("href") || "").trim();
    if (/^(mailto:|tel:|javascript:|#)/.test(href)) return;
    links.add(new URL(href, url).href); // relative -> absolute
  });
  console.log("Links found:", links.size);
})();

PHP — DOMDocument

The built-in DOMDocument parses HTML out of the box. For non-ASCII text it's important to feed it the right encoding — the most reliable trick is shown below.

<?php
$url = "https://example.com";
$html = file_get_contents($url);

// If the page is in a legacy encoding, convert to UTF-8 before parsing
$encoding = mb_detect_encoding($html, ["UTF-8", "Windows-1252", "ISO-8859-1"], true);
if ($encoding && $encoding !== "UTF-8") {
    $html = mb_convert_encoding($html, "UTF-8", $encoding);
}

$dom = new DOMDocument();
libxml_use_internal_errors(true); // silence errors from messy HTML
// Hack so DOMDocument doesn't mangle UTF-8: state the encoding explicitly
$dom->loadHTML('<?xml encoding="UTF-8">' . $html);
libxml_clear_errors();

$xpath = new DOMXPath($dom);

// Text
$titleNode = $xpath->query("//title")->item(0);
echo "Title: " . trim($titleNode->textContent) . PHP_EOL;

// Links
$links = [];
foreach ($xpath->query("//a[@href]") as $a) {
    $href = trim($a->getAttribute("href"));
    if (preg_match('/^(mailto:|tel:|javascript:|#)/', $href)) continue;
    $links[$href] = true; // dedupe by key
}
echo "Links found: " . count($links) . PHP_EOL;

(On PHP 8.4+ you can also use the new standards-compliant Dom\HTMLDocument class, which handles encodings more gracefully and doesn't need the XML-prolog hack.)

Go — net/http + goquery

goquery mirrors the jQuery API. For encodings there's the golang.org/x/net/html/charset package, which reads the charset from headers and meta tags itself.

package main

import (
    "fmt"
    "net/http"
    "net/url"

    "github.com/PuerkitoBio/goquery"
    "golang.org/x/net/html/charset"
)

func main() {
    target := "https://example.com"
    resp, err := http.Get(target)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    // Automatically converts the stream to UTF-8 using the charset from the header/meta
    utf8Reader, err := charset.NewReader(resp.Body, resp.Header.Get("Content-Type"))
    if err != nil {
        panic(err)
    }

    doc, err := goquery.NewDocumentFromReader(utf8Reader)
    if err != nil {
        panic(err)
    }

    // Text
    fmt.Println("Title:", doc.Find("title").First().Text())

    // Links
    base, _ := url.Parse(target)
    links := map[string]bool{}
    doc.Find("a[href]").Each(func(_ int, s *goquery.Selection) {
        href, _ := s.Attr("href")
        if u, err := base.Parse(href); err == nil {
            links[u.String()] = true
        }
    })
    fmt.Println("Links found:", len(links))
}

Java — Jsoup

Jsoup is one of the most convenient libraries: loading, parsing, selectors, and encoding detection in one package.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import java.util.HashSet;
import java.util.Set;

public class Parser {
    public static void main(String[] args) throws Exception {
        String url = "https://example.com";
        // Jsoup detects the encoding from headers and meta tags itself
        Document doc = Jsoup.connect(url)
                .userAgent("Mozilla/5.0")
                .get();

        // Text
        System.out.println("Title: " + doc.title());

        // Links (absUrl resolves them to absolute for you)
        Set<String> links = new HashSet<>();
        for (Element a : doc.select("a[href]")) {
            String href = a.absUrl("href");
            if (href.isBlank()) continue;
            links.add(href);
        }
        System.out.println("Links found: " + links.size());
    }
}

7. Quick recommendations

• Don't parse HTML with regex — use a real parser (lxml, cheerio, goquery, Jsoup, DOMDocument).
• Detect the encoding from the bytes, not the string, and convert everything to UTF-8 as early as possible.
• Always resolve relative links to absolute (urljoin, new URL(base), absUrl).
• For dynamic sites, be ready to use a headless browser or work with the internal API.
• Respect robots.txt, add pauses, and don't take down someone else's server.

If HTML parsing is only one piece of a bigger project — or the target site fights back harder than you'd like — our team runs this end to end as a managed data extraction service, delivering clean, structured output instead of raw markup.

See also

If you're still choosing a stack for your task, we have two deeper pieces that continue this topic:

• Best programming language for web scraping — a comparison of Python, Node.js, PHP, Go, and other languages in terms of development speed, performance, and scraping ecosystem.
• Web scraping libraries — an overview of specific tools (BeautifulSoup, Scrapy, cheerio, goquery, Jsoup, and more), their strengths and weaknesses, and where each one fits.