C++ Web Scraping: The Complete Guide

Web scraping means automatically fetching web pages and pulling structured data out of them. Technically the job splits into two independent stages:

1. The network stage — download HTML/JSON over HTTP(S).
2. The parsing stage — turn that raw text into a tree (DOM) and extract the nodes you need.

People reach for C++ web scraping not for convenience — a scraper is far quicker to write in Python or Go — but for performance and control: tens of thousands of connections on a single core, minimal memory use, predictable latency, no garbage-collector pauses, and easy integration into an existing C/C++ backend. If you already run a C++ stack or need a high-throughput crawler, a C++ web scraper earns its keep.

Before you write any code, keep the legal and ethical side in mind: respect robots.txt, don't hammer a site (rate-limit yourself), and read the target's terms of service and the data-protection law that applies to you (GDPR in the EU/UK, the CCPA in California). The block-evasion techniques below (proxies, Tor) are tools, not an invitation to break a platform's rules.

Fetching a Page: HTTP Clients

This is the foundation of the whole scraper. C++ gives you several options, from the low-level classic to convenient "Python-like" wrappers.

libcurl — the industry standard

libcurl is the de-facto standard for network requests in C/C++. It supports HTTP/1.1, HTTP/2, HTTP/3, HTTPS, proxies, SOCKS5, cookies, compression, and timeouts — literally everything a scraper needs, in one library. The downside is a verbose, callback-based C API.

#include <curl/curl.h>
#include <string>

// Callback: curl invokes it as data arrives; we accumulate it into a string.
static size_t write_cb(char* ptr, size_t size, size_t nmemb, void* userdata) {
    auto* out = static_cast<std::string*>(userdata);
    out->append(ptr, size * nmemb);
    return size * nmemb;
}

std::string fetch(const std::string& url) {
    CURL* curl = curl_easy_init();
    std::string body;

    curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
    curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_cb);
    curl_easy_setopt(curl, CURLOPT_WRITEDATA, &body);
    curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);     // follow redirects
    curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, "");    // gzip/deflate/br automatically
    curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/5.0 (compatible; MyBot/1.0)");
    curl_easy_setopt(curl, CURLOPT_TIMEOUT, 30L);

    CURLcode res = curl_easy_perform(curl);
    if (res != CURLE_OK) {
        // handle curl_easy_strerror(res)
    }
    curl_easy_cleanup(curl);
    return body;
}

Note CURLOPT_ACCEPT_ENCODING, "" — it turns on transparent gzip/deflate/brotli decompression. Without it you'll get binary garbage instead of HTML.

cpr — "Curl for People"

cpr (C++ Requests) is a modern wrapper over libcurl in the spirit of Python's Requests. It needs C++17 and is actively maintained (docs). Same capabilities, but the code is several times shorter:

#include <cpr/cpr.h>

cpr::Response r = cpr::Get(
    cpr::Url{"https://example.com"},
    cpr::Header{{"User-Agent", "MyBot/1.0"}},
    cpr::Timeout{30000}
);

r.status_code;            // 200
r.header["content-type"]; // "text/html; charset=utf-8"
r.text;                   // response body

For most projects this is the best starting point: you get all of libcurl's power (proxies, cookies, SSL, async requests) with a friendly API. Pull it in via CMake FetchContent, vcpkg, or Conan.

cpp-httplib — header-only

cpp-httplib is a single header file with no external dependencies (HTTPS needs OpenSSL). Ideal when you don't want to pull in curl. Downsides: HTTP/1.1 only, no built-in SOCKS-proxy support, and no brotli.

#include <httplib.h>

httplib::Client cli("https://example.com");
auto res = cli.Get("/");
if (res && res->status == 200) {
    std::string body = res->body;
}

Boost.Beast — low-level and async

Boost.Beast is built on top of Boost.Asio and gives you full control over HTTP/WebSocket at the socket level with an async model (coroutines, futures, callbacks). This is the path when you need tens of thousands of concurrent connections and custom network logic. The price is noticeably more code; HTTPS has to be wired up by hand through the Asio SSL stream.

Which to choose

Working with HTTPS / SSL

Today almost the entire web is HTTPS, so TLS isn't optional — it's the norm.

Via libcurl/cpr

libcurl uses a TLS layer under the hood (by default OpenSSL, but builds also ship with GnuTLS, mbedTLS, BoringSSL, Schannel on Windows, Secure Transport on macOS). The key thing is certificate verification:

// ON BY DEFAULT — change only deliberately.
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1L); // verify the certificate chain
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2L); // verify the host name
// Point to your own CA bundle if the system one isn't found:
curl_easy_setopt(curl, CURLOPT_CAINFO, "/path/to/cacert.pem");

Never disable VERIFYPEER/VERIFYHOST in production to "fix" a certificate error — it opens the door to MITM attacks. If the system root store is missing (common on Windows or in containers), download an up-to-date cacert.pem (published by the curl project) and point to it with CURLOPT_CAINFO.

In cpr the defaults are safe; if needed, tune them through cpr::SslOptions.

Fine points

• SNI (Server Name Indication) is on by default — needed for virtual hosts.
• TLS version: it's worth forcing a minimum of TLS 1.2 (CURLOPT_SSLVERSION = CURL_SSLVERSION_TLSv1_2).
• TLS fingerprinting: advanced anti-bot systems can tell clients apart by the JA3/JA4 fingerprint of the TLS handshake. A plain libcurl fingerprint differs from a browser's — that's a big topic of its own (right down to patched curl builds that mimic a browser's ClientHello). See TLS fingerprinting for more.

Response Status and Headers

A scraper has to react to HTTP codes: 200 — OK, 301/302 — redirect, 403/429 — blocked / rate-limited, 5xx — server error (retry later).

Response code and headers in libcurl

long http_code = 0;
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &http_code);

char* content_type = nullptr;
curl_easy_getinfo(curl, CURLINFO_CONTENT_TYPE, &content_type);

// Capture the full set of headers with a separate callback:
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, header_cb);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, &headers_map);

Useful getinfo fields: CURLINFO_RESPONSE_CODE, CURLINFO_CONTENT_TYPE, CURLINFO_EFFECTIVE_URL (final URL after redirects), CURLINFO_REDIRECT_COUNT, CURLINFO_TOTAL_TIME, CURLINFO_SIZE_DOWNLOAD.

In cpr — already parsed for you

cpr::Response r = cpr::Get(cpr::Url{"https://example.com"});
r.status_code;               // 200
r.reason;                    // "OK"
r.header["content-type"];    // convenient header map
r.url;                       // final URL
r.elapsed;                   // request time

Why the parser cares: from Content-Type: text/html; charset=windows-1252 you extract the encoding (see the encodings section), and from Retry-After on a 429 you learn how long to wait before retrying.

HTML Parsing Libraries

You've downloaded the HTML — now you need to build a DOM and extract nodes by selector. Don't parse HTML with regular expressions — real-world markup with unclosed tags, nesting, and comments breaks any regex; use a proper parser.

lexbor — the modern pick #1

lexbor (GitHub) is a fast, WHATWG-compliant HTML5 parser written in pure C with no external dependencies. It supports CSS selectors, encoding detection from the byte stream, and DOM manipulation. It's the de-facto successor to myhtml/Modest and currently the best choice for new code (it's what powers the built-in DOM parser in PHP 8.4, incidentally).

#include <lexbor/html/parser.h>
#include <lexbor/dom/interfaces/element.h>

lxb_html_document_t* doc = lxb_html_document_create();
lxb_html_document_parse(doc, (const lxb_char_t*)html.data(), html.size());
// then walk the DOM or query via the selectors module (CSS selectors)
lxb_html_document_destroy(doc);

For a convenient C++ wrapper over lexbor there are third-party projects (for example, sprexer).

libxml2 + XPath

libxml2 is a mature, time-tested library. Its htmlReadMemory() parses "dirty" HTML, and XPath gives you powerful queries (no CSS selectors out of the box, but XPath is more expressive). An excellent choice when your data maps cleanly onto XPath expressions.

#include <libxml/HTMLparser.h>
#include <libxml/xpath.h>

htmlDocPtr doc = htmlReadMemory(html.data(), html.size(), nullptr, "UTF-8",
                                HTML_PARSE_RECOVER | HTML_PARSE_NOERROR | HTML_PARSE_NOWARNING);
xmlXPathContextPtr ctx = xmlXPathNewContext(doc);
xmlXPathObjectPtr res = xmlXPathEvalExpression((const xmlChar*)"//a/@href", ctx);
// iterate res->nodesetval->nodeTab

Gumbo — a classic, but archived

Gumbo from Google was long the standard for HTML5 parsing in C/C++, but the repository is now archived (read-only since January 2026). Development continues in a community fork on Codeberg. For new code, prefer lexbor; Gumbo is worth knowing because you'll meet it in plenty of existing projects.

Other tools

• htmlcxx — a simple C++ HTML/CSS parser; handy for light tasks, but long unmaintained.
• pugixml and RapidXML — for strict XML (RSS, sitemaps, SOAP), not arbitrary HTML.
• JSON: many sites serve data through an API or inline JSON. Reach for nlohmann/json (convenience) or RapidJSON (speed).

Which to choose

Character Encodings and Non-ASCII Text

A classic pain point: a page comes back as garbled "mojibake." The cause is almost always the same — an encoding mismatch. Most modern pages are UTF-8, but you'll still hit legacy single-byte encodings in the wild: windows-1252 and ISO-8859-1 (Western European), Shift_JIS and GBK (Japanese/Chinese), windows-1251 (Cyrillic), and more. Inside your program keep everything in UTF-8, converting on the way in.

How to detect the source encoding

Sources, in priority order:

1. The HTTP header Content-Type: text/html; charset=windows-1252.
2. HTML meta: <meta charset="..."> or <meta http-equiv="Content-Type" content="...; charset=...">.
3. A BOM at the start of the file (for UTF-8/16).
4. Content heuristics (if nothing else is available).

Handily, lexbor can detect the encoding from the byte stream itself — which removes most of the problem at the parsing stage.

Converting to UTF-8

Option A — iconv (GNU libiconv), available almost everywhere:

#include <iconv.h>
// windows-1252 -> UTF-8
iconv_t cd = iconv_open("UTF-8", "WINDOWS-1252");
// ... iconv(cd, &in, &inleft, &out, &outleft) ...
iconv_close(cd);

Option B — ICU (International Components for Unicode) — the most powerful and reliable route: a huge list of encodings, Unicode normalization, and a charset detector (ucsdet_*):

#include <unicode/ucnv.h>
icu::UnicodeString us(raw.data(), raw.size(), "windows-1252");
std::string utf8;
us.toUTF8String(utf8);

Option C — UTF8-CPP (utfcpp) — a lightweight, header-only library for validating/iterating/converting text that's already UTF-8/UTF-16/UTF-32 (it won't transcode legacy 8-bit encodings, but it's indispensable for working correctly with UTF-8 itself).

Practical gotchas

• Don't print UTF-8 to a Windows console without SetConsoleOutputCP(CP_UTF8) — you'll see garbage even though the data is correct.
• On Windows, use the wide API (std::wstring/UTF-16) for non-ASCII filenames.
• std::string stores bytes, not "characters"; to count actual characters, count code points (utfcpp/ICU), not .size().
• Always enforce the invariant: "on input — detect and transcode to UTF-8; everywhere downstream — UTF-8 only."

Cookies and Sessions

Many sites require sessions: login, cart, anti-bot checks, pagination behind authentication. Cookies must be accepted, stored, and sent back.

libcurl has a built-in "cookie engine":

// Enable the engine and persist cookies to a file between runs:
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookies.txt"); // read (empty string just enables the in-memory engine)
curl_easy_setopt(curl, CURLOPT_COOKIEJAR,  "cookies.txt"); // write on cleanup

// Set a specific cookie manually:
curl_easy_setopt(curl, CURLOPT_COOKIE, "session=abc123; lang=en");

Within a single "session," reuse the same CURL handle (or one cpr::Session) — that way cookies, keep-alive connections, and TLS sessions carry over between requests, which is both faster and more correct from the site's point of view.

In cpr:

cpr::Session session;
session.SetUrl(cpr::Url{"https://example.com/login"});
session.SetCookies(cpr::Cookies{{"session", "abc123"}});
cpr::Response r = session.Get();
cpr::Cookies received = r.cookies; // cookies set by the server

Gotcha: for a correct redirect that carries cookies across subdomains, enable the cookie engine before the request and don't recreate the handle on every step.

Proxies

Proxies help you get around geo-restrictions, spread load, and lower the chance of an IP block. libcurl supports HTTP, HTTPS, and SOCKS5 proxies out of the box.

// HTTP proxy with auth:
curl_easy_setopt(curl, CURLOPT_PROXY, "http://user:pass@proxyhost:8080");

// SOCKS5 (with DNS resolved proxy-side — important for anonymity):
curl_easy_setopt(curl, CURLOPT_PROXY, "socks5h://proxyhost:1080");

In cpr:

cpr::Response r = cpr::Get(
    cpr::Url{"https://example.com"},
    cpr::Proxies{{"https", "http://user:pass@proxyhost:8080"}}
);

Proxy rotation

For large-scale scraping you keep a pool of proxies and rotate them: round-robin, random, or by "health" (banned → temporarily exclude). The simplest strategy is a map of {proxy → error count / cooldown} and picking a live proxy before each request. It makes sense to keep a separate cookie jar and User-Agent per proxy so the "identities" don't cross over.

The socks5h:// prefix (with the h) means DNS resolution goes through the proxy, not locally — otherwise your real DNS query gives you away. This is critical for proxies and especially for Tor (below).

Scraping Through Tor

Tor is a special case of a SOCKS5 proxy: a local Tor daemon exposes SOCKS5 usually on 127.0.0.1:9050 (Tor Browser on 9150). Just point your requests there:

// All traffic through the Tor network; 'h' resolves DNS inside Tor (required!):
curl_easy_setopt(curl, CURLOPT_PROXY, "socks5h://127.0.0.1:9050");

Changing "identity" (a new exit node)

Through Tor's ControlPort (port 9051) you can request a new circuit — effectively a new IP — by sending the NEWNYM signal. To do that, open a TCP socket to the control port and send commands per the Tor Control Protocol:

AUTHENTICATE "password"
SIGNAL NEWNYM
QUIT

This lets you rotate the exit IP between "waves" of requests. In practice, bear in mind:

• Between NEWNYM signals Tor imposes a short cooldown — don't fire the signal too often.
• Tor is slow: high latency, narrow bandwidth. For bulk scraping it's an anonymity tool, not a performance one.
• Many sites block known Tor exit nodes wholesale (403/CAPTCHA); handling that is its own topic — see solving CAPTCHAs.
• Don't route through Tor any task where you're logging in under a real account anyway — anonymity is lost at the application layer.

The code layer is the same as for proxies (above); only the proxy address changes, plus the logic for talking to the control port.

Multithreading and curl_multi

Scraping is an I/O-bound task: most time is spent waiting on the network. Parallelism delivers a multiplied speedup. There are two fundamentally different approaches.

Thread pool + one handle per thread

The classic: a pool of worker threads, a shared thread-safe URL queue, and each thread with its own CURL handle.

#include <thread>
#include <queue>
#include <mutex>

std::queue<std::string> urls;
std::mutex m;

void worker() {
    CURL* curl = curl_easy_init();   // OWN handle per thread
    for (;;) {
        std::string url;
        {
            std::lock_guard<std::mutex> lk(m);
            if (urls.empty()) break;
            url = urls.front(); urls.pop();
        }
        std::string html = fetch_with(curl, url);
        // ... parse ...
    }
    curl_easy_cleanup(curl);
}

int main() {
    curl_global_init(CURL_GLOBAL_ALL); // ONCE, before starting threads!
    std::vector<std::thread> pool;
    for (int i = 0; i < 16; ++i) pool.emplace_back(worker);
    for (auto& t : pool) t.join();
    curl_global_cleanup();
}

Critical: curl_global_init() is called once in the main thread before workers start; a CURL handle cannot be shared between threads — each thread gets its own (see curl: threadsafe).

curl_multi — many connections in one thread

curl_multi drives hundreds or thousands of parallel transfers in one thread via multiplexing (epoll/poll). It's more memory-efficient than "one thread per connection" and scales beautifully. The catch is more complex code (a curl_multi_perform loop plus event handling). You can combine the two: several threads, each with its own multi stack.

Ready-made concurrency primitives

• std::thread / std::async — the basic STL tools.
• oneTBB (Intel Threading Building Blocks) — high-level parallel algorithms and pipelines (parallel_pipeline maps nicely onto "download → parse → store"), plus thread-safe containers.
• OpenMP — simple loop parallelization via pragmas; more often useful for CPU-heavy parsing than for network waits.

Recommendation

For most projects: a thread pool (16–64) + one handle per thread. When you hit memory or thread-count limits at thousands of connections, move to curl_multi.

Storing URLs and Queues

(This section is an overview — it's really crawler architecture.) As soon as your parser starts following links, you face the problem of managing the frontier — the queue of not-yet-visited URLs — and deduplication.

Key sub-problems:

• The work queue (frontier). In the simplest case, a std::queue in memory. For durability and distribution, an external broker: RabbitMQ (client rabbitmq-c) or Redis as a queue/set (client hiredis).
• URL deduplication. So you don't fetch the same thing twice: a std::unordered_set of normalized URLs in memory; at large volumes, a Bloom filter (compact, at the cost of rare false positives) or keys in Redis.
• URL normalization. Canonicalizing (scheme, host case, sorting the query, dropping #fragment, resolving relative links). Boost.URL helps here.
• Persistence. Crawl state and results go to a database: SQLite for a single machine, PostgreSQL/ClickHouse at scale.
• Crawl policy. BFS/DFS, per-domain priorities, per-host rate limiting and polite delays, depth limits.

A typical pipeline: frontier → fetcher (pool/multi) → parser → link extractor → normalization → dedup → back to frontier, with the extracted data going to storage.

What Else to Account For

Topics people often forget, but without which a "production" scraper breaks fast:

• robots.txt and politeness. Parse and respect robots.txt (there's an official parser, google/robotstxt); add delays and cap RPS per domain. It's both ethical and reduces ban risk.
• Anti-bot defenses and fingerprints. Sites analyze the User-Agent, the set and order of headers, TLS fingerprint (JA3/JA4), HTTP/2 framing, and behavior. At minimum, set plausible headers (Accept, Accept-Language, User-Agent) and don't "knock" identically every time. CAPTCHAs and JS challenges are a whole separate layer.
• JavaScript rendering. libcurl/lexbor see only the original HTML, without running JS. For SPA sites (React/Vue) where content loads via scripts, you need either a headless browser (driving Chromium over the Chrome DevTools Protocol from C++), or — often simpler — find and call the same internal JSON API the frontend uses.
• Content compression. Turn on Accept-Encoding (gzip/deflate/brotli) — it cuts traffic several-fold; libcurl decompresses transparently with CURLOPT_ACCEPT_ENCODING. For manual decompression: zlib and brotli.
• Retries and backoff. The network is unstable. Implement retry with exponential backoff and jitter, honor the Retry-After header on 429/503, and set reasonable timeouts (CURLOPT_TIMEOUT, CURLOPT_CONNECTTIMEOUT).
• Memory and resources. Close handles, free DOM trees (*_destroy), and watch Content-Length / a size cap (CURLOPT_MAXFILESIZE) so you don't accidentally pull a gigabyte file into memory.
• Logging and monitoring. Track response codes, throughput, and the ban rate per domain — otherwise you won't know when the scraper has "quietly died."

Pros and Cons

Upsides of doing it in C++

• Performance and low latency. Near-metal speed, minimal per-request overhead.
• Memory efficiency. You can drive thousands of connections on modest resources (especially with curl_multi), with no GC pauses.
• Full control. Precise tuning of TLS, sockets, timeouts, and memory — where higher-level languages hide the details.
• A mature ecosystem of C libraries. libcurl, OpenSSL, libxml2, ICU, lexbor — an industrial, proven foundation.
• Easy integration into an existing C/C++ backend without cross-language bridges.

Downsides

• Slower, costlier development. What takes an evening in Python needs more code and more care in C++.
• Manual resource management. Memory leaks, dangling handles, threading races — your responsibility.
• Callback-based C APIs (libcurl, libxml2) are verbose; wrappers (cpr) only partly save you.
• Weak "batteries" for JS sites. No native equivalent of Selenium/Playwright; driving a headless browser from C++ is painful.
• Fewer ready-made scraping frameworks than in Python (no out-of-the-box Scrapy) — more is written by hand.

Bottom line: C++ is justified when scale, speed, and resource consumption are critical (high-load crawlers, a product on a C++ stack). For one-off tasks and prototypes, Python or Go is faster.

Recommended Stack

A balanced set for a production C++ scraper:

Start small — cpr + lexbor + a single thread — and add complexity (proxies, multithreading, queues) only under real load.

Building and maintaining a high-performance C++ crawler is a genuine engineering investment — proxies, anti-bot handling, encodings, and scale all add up. If you'd rather have the data than own the infrastructure, scraping.pro can build it for you as a custom data extraction service or deliver clean, ready-to-use feeds via data as a service. You focus on what the data is worth; we handle how it's collected.