Rust Web Scraping: The Complete Guide

Current as of 2026. Crate versions used here: reqwest 0.13, scraper 0.27, tokio 1.x, encoding_rs 0.8. One important change: starting with reqwest 0.13, the default TLS backend is rustls (pure Rust) rather than system OpenSSL/native-tls.

Contents

1. Why use Rust for scraping
2. Fetching a page (the HTTP client)
3. Libraries for parsing content
4. Fixing character-encoding problems
5. Concurrency and async
6. Using proxies
7. Scraping through Tor
8. HTTPS / TLS
9. Working with cookies
10. Response status and headers
11. URL storage and queues
12. Extras people forget
13. Architecture of a real crawler
14. Pros and cons of doing it in Rust
15. Legal and ethical considerations

1. Why use Rust for scraping

Web scraping is the automated retrieval of HTML/JSON/XML from pages and the extraction of structured data from it. Every scraper is built from two big pieces:

• a network layer that downloads the page (the HTTP client);
• a parsing layer that turns raw HTML into the fields you actually want (a parser plus selectors).

On top of that you add proxies, concurrency, anti-bot handling, a link queue, and so on. The appeal of web scraping with Rust is that you get C-like speed and a tiny memory footprint while working safely with concurrency — exactly what matters when you're pulling millions of pages.

Here's the starter Cargo.toml we'll grow feature by feature:

[package]
name = "parser-demo"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest = { version = "0.13", features = ["json", "gzip", "brotli"] }
tokio = { version = "1", features = ["full"] }
scraper = "0.27"
encoding_rs = "0.8"
anyhow = "1"          # ergonomic error handling

Official crate pages for the basics: reqwest, tokio, scraper, encoding_rs, anyhow.

2. Fetching a page (the HTTP client)

The Rust ecosystem has several HTTP clients. For scraping, 99% of the time you reach for reqwest.

Docs: docs.rs/reqwest.

2.1 The simplest request (async)

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let body = reqwest::get("https://example.com")
        .await?      // wait for the response
        .text()      // read the body as a string
        .await?;
    println!("{body}");
    Ok(())
}

2.2 The right way — a reusable Client

reqwest::get spins up a brand-new client every time. That's expensive: you lose the connection pool (keep-alive). Create one Client and clone it — internally it's an Arc, so the clone is cheap.

use std::time::Duration;
use reqwest::Client;

fn build_client() -> anyhow::Result<Client> {
    let client = Client::builder()
        // pretend to be an ordinary browser
        .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) \
                     AppleWebKit/537.36 (KHTML, like Gecko) \
                     Chrome/124.0 Safari/537.36")
        .timeout(Duration::from_secs(30))          // overall request timeout
        .connect_timeout(Duration::from_secs(10))  // connection-setup timeout
        .gzip(true)                                // auto-decompress gzip
        .brotli(true)                              // auto-decompress brotli
        .build()?;
    Ok(client)
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = build_client()?;
    let resp = client
        .get("https://example.com")
        .header("Accept-Language", "en-US,en;q=0.9")
        .send()
        .await?;

    println!("Status: {}", resp.status());
    let html = resp.text().await?;
    println!("HTML length: {}", html.len());
    Ok(())
}

2.3 The synchronous option (no tokio)

If you'd rather not drag in an async runtime for a small script, there's blocking:

reqwest = { version = "0.13", features = ["blocking"] }

fn main() -> anyhow::Result<()> {
    let body = reqwest::blocking::get("https://example.com")?.text()?;
    println!("{body}");
    Ok(())
}

You can't call the blocking client from inside an async runtime — it will panic. Pick one or the other.

3. Libraries for parsing content

Once the HTML is downloaded, you need to take it apart. Rule number one: don't parse HTML with regular expressions. HTML isn't regular; that approach breaks on the first unescaped quote. Regexes are only appropriate for pulling small bits out of text you've already located.

3.1 scraper + CSS selectors

Docs and examples: docs.rs/scraper.

use scraper::{Html, Selector};

fn parse_articles(html: &str) -> anyhow::Result<()> {
    let document = Html::parse_document(html);

    // Compile selectors once (outside the loop).
    let item_sel  = Selector::parse("article.post").unwrap();
    let title_sel = Selector::parse("h2.title > a").unwrap();
    let date_sel  = Selector::parse("time.published").unwrap();

    for item in document.select(&item_sel) {
        let title = item
            .select(&title_sel)
            .next()
            .map(|e| e.text().collect::<String>().trim().to_string())
            .unwrap_or_default();

        // link from the href attribute
        let link = item
            .select(&title_sel)
            .next()
            .and_then(|e| e.value().attr("href"))
            .unwrap_or("");

        // date from the datetime attribute
        let date = item
            .select(&date_sel)
            .next()
            .and_then(|e| e.value().attr("datetime"))
            .unwrap_or("");

        println!("{title} | {date} | {link}");
    }
    Ok(())
}

Handy scraper techniques:

• element.text().collect::<String>() — collect all the text inside (including nested).
• element.value().attr("href") — grab an attribute.
• element.html() / element.inner_html() — get the source HTML.
• Selectors support [attr="value"], :nth-child, >, descendant combinators, and so on.

3.2 JSON from an API

Often it's easier to take data not from the HTML but from the hidden JSON API the page itself calls. Open DevTools -> the Network tab -> find the request that returns JSON. This is more reliable than parsing markup will ever be. Deserialize with serde + serde_json.

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct Product {
    id: u64,
    name: String,
    price: f64,
}

async fn fetch_products(client: &reqwest::Client) -> anyhow::Result<Vec<Product>> {
    let products = client
        .get("https://shop.example.com/api/products")
        .send()
        .await?
        .json::<Vec<Product>>()   // deserialize straight into structs
        .await?;
    Ok(products)
}

3.3 Sitemaps and RSS via quick-xml

A sitemap (sitemap.xml) is the best way to learn every URL on a site without crawling links. Parse it as ordinary XML with quick-xml (or a purpose-built crate like sitemap).

4. Fixing character-encoding problems

This is a classic headache with legacy sites. Plenty of older pages — and some government and enterprise systems — still serve content in Windows-1252, ISO-8859-1, Shift_JIS, or GBK instead of UTF-8.

Why it breaks

The resp.text() method decides on an encoding from the Content-Type: text/html; charset=... header. If the header carries no charset and it's only declared in the HTML (<meta charset="windows-1252">), reqwest assumes UTF-8 by default — and you get mojibake (café instead of café, or worse).

The fix: read the bytes and decode them yourself

Grab the raw bytes with .bytes() and decode with the right encoding using encoding_rs (the same engine that ships in Firefox).

use encoding_rs::{WINDOWS_1252, UTF_8};

async fn get_text_win1252(client: &reqwest::Client, url: &str) -> anyhow::Result<String> {
    let resp = client.get(url).send().await?;
    let bytes = resp.bytes().await?;

    // Decode as Windows-1252.
    let (text, _enc, had_errors) = WINDOWS_1252.decode(&bytes);
    if had_errors {
        eprintln!("Warning: there were errors while decoding");
    }
    Ok(text.into_owned())
}

Detecting the encoding automatically

Better than hard-coding an encoding is to detect it. The algorithm:

1. First check charset in the Content-Type header.
2. If it's missing, look for <meta charset=...> / <meta http-equiv="Content-Type"> in the first few kilobytes of HTML.
3. If that's absent too, guess statistically with the chardetng crate.

use encoding_rs::Encoding;
use reqwest::header::CONTENT_TYPE;

async fn get_text_smart(client: &reqwest::Client, url: &str) -> anyhow::Result<String> {
    let resp = client.get(url).send().await?;

    // 1) try to take charset from the header
    let header_charset = resp
        .headers()
        .get(CONTENT_TYPE)
        .and_then(|v| v.to_str().ok())
        .and_then(|ct| ct.split("charset=").nth(1))
        .map(|s| s.trim().to_string());

    let bytes = resp.bytes().await?;

    // 2) if the header has none, look inside <meta> (simplified: first 1024 bytes)
    let charset = header_charset.or_else(|| {
        let head = String::from_utf8_lossy(&bytes[..bytes.len().min(1024)]);
        head.to_lowercase()
            .split("charset=")
            .nth(1)
            .map(|s| s.trim_matches(|c: char| !c.is_ascii_alphanumeric() && c != '-')
                      .to_string())
    });

    // 3) pick an encoding (default to UTF-8)
    let enc = charset
        .as_deref()
        .and_then(|name| Encoding::for_label(name.as_bytes()))
        .unwrap_or(UTF_8);

    let (text, _, _) = enc.decode(&bytes);
    Ok(text.into_owned())
}

Alternative: reqwest's resp.text_with_charset("windows-1252") uses the given encoding as a fallback if no charset arrived in the header. Simpler, but it doesn't cover the "header says UTF-8 but it's really 1252" case.

5. Concurrency and async

Scraping is almost always I/O-bound: the CPU sits idle while network packets fly. So in Rust the winner here isn't threads but async on tokio: thousands of simultaneous requests across one or two OS threads.

Keep two jobs distinct:

• Downloading (I/O-bound) -> async/tokio, lots of concurrent connections.
• Parsing (CPU-bound; html5ever hits the CPU) -> at scale, offload to rayon or tokio::task::spawn_blocking so you don't block the async runtime.

5.1 Bounded concurrency — buffer_unordered

The most idiomatic approach: turn a stream of URLs into a stream of futures, and let buffer_unordered(N) run at most N at a time (from the futures crate).

use futures::stream::{self, StreamExt};

async fn crawl_many(client: &reqwest::Client, urls: Vec<String>) {
    let concurrency = 20; // no more than 20 requests at once

    let results = stream::iter(urls)
        .map(|url| {
            let client = client.clone(); // cheap clone (Arc inside)
            async move {
                match client.get(&url).send().await {
                    Ok(resp) => {
                        let status = resp.status();
                        let body = resp.text().await.unwrap_or_default();
                        (url, status.as_u16(), body.len())
                    }
                    Err(e) => {
                        eprintln!("Error {url}: {e}");
                        (url, 0, 0)
                    }
                }
            }
        })
        .buffer_unordered(concurrency)
        .collect::<Vec<_>>()
        .await;

    for (url, status, len) in results {
        println!("{status} {len:>8} {url}");
    }
}

5.2 Bounding with a Semaphore

When tasks are spawned via tokio::spawn, hold the limit with a semaphore:

use std::sync::Arc;
use tokio::sync::Semaphore;

async fn crawl_with_semaphore(client: reqwest::Client, urls: Vec<String>) {
    let sem = Arc::new(Semaphore::new(20)); // at most 20 "in flight"
    let mut handles = Vec::new();

    for url in urls {
        let client = client.clone();
        let sem = sem.clone();
        handles.push(tokio::spawn(async move {
            let _permit = sem.acquire().await.unwrap(); // wait for a free slot
            let _ = client.get(&url).send().await;
            // the permit is released when it drops out of scope
        }));
    }

    for h in handles {
        let _ = h.await;
    }
}

5.3 CPU-bound parsing with rayon

If you've already downloaded thousands of HTML pages and need to parse them fast, that's a job for every core (rayon):

use rayon::prelude::*;

fn parse_all(pages: Vec<String>) -> Vec<usize> {
    pages
        .par_iter()                      // parallel iterator
        .map(|html| {
            let doc = scraper::Html::parse_document(html);
            doc.select(&scraper::Selector::parse("a").unwrap()).count()
        })
        .collect()
}

6. Using proxies

You need proxies to (a) avoid IP bans during large-scale scraping and (b) get around geo-restrictions. reqwest supports HTTP, HTTPS, and SOCKS5 proxies. For a deeper look, see our guide to rotating proxies for scraping.

For SOCKS, enable the feature:

reqwest = { version = "0.13", features = ["socks"] }

6.1 One proxy per client

use reqwest::{Client, Proxy};

fn client_with_proxy() -> anyhow::Result<Client> {
    let proxy = Proxy::all("http://proxy.example.com:8080")?
        .basic_auth("user", "password"); // if authentication is required

    let client = Client::builder()
        .proxy(proxy)
        .build()?;
    Ok(client)
}

Proxy::http(...), Proxy::https(...), and Proxy::all(...) set the proxy for the matching schemes. SOCKS5:

let proxy = reqwest::Proxy::all("socks5://127.0.0.1:1080")?;

6.2 Rotating a proxy pool

One Client is tied to one proxy. To rotate, it's easier to keep one client per proxy and pick them round-robin:

use std::sync::atomic::{AtomicUsize, Ordering};
use reqwest::{Client, Proxy};

struct ProxyPool {
    clients: Vec<Client>,
    idx: AtomicUsize,
}

impl ProxyPool {
    fn new(proxies: &[&str]) -> anyhow::Result<Self> {
        let clients = proxies
            .iter()
            .map(|p| {
                Client::builder()
                    .proxy(Proxy::all(*p)?)
                    .build()
                    .map_err(Into::into)
            })
            .collect::<anyhow::Result<Vec<_>>>()?;
        Ok(Self { clients, idx: AtomicUsize::new(0) })
    }

    /// Returns the next client, round-robin.
    fn next(&self) -> &Client {
        let i = self.idx.fetch_add(1, Ordering::Relaxed) % self.clients.len();
        &self.clients[i]
    }
}

Residential/mobile proxies with provider-side rotation usually give you a single "gateway" address — in that case you don't need rotation on your end; one client is enough.

7. Scraping through Tor

Tor gives you anonymity and a free "rotation" of IPs (a new circuit -> a new exit node). There are two ways to do it.

7.1 The simple path: external Tor + SOCKS5

Run system Tor (the tor daemon or Tor Browser), which brings up a SOCKS5 proxy on 127.0.0.1:9050 (Tor Browser uses 9150). From there it's just an ordinary SOCKS proxy:

use reqwest::{Client, Proxy};

fn tor_client() -> anyhow::Result<Client> {
    // IMPORTANT: socks5h (with the h), not socks5.
    // 'h' = resolve DNS on the proxy side (inside Tor),
    // otherwise you leak DNS and .onion won't work.
    let proxy = Proxy::all("socks5h://127.0.0.1:9050")?;
    let client = Client::builder()
        .proxy(proxy)
        .build()?;
    Ok(client)
}

Confirm the traffic is going through Tor:

async fn check_tor(client: &reqwest::Client) -> anyhow::Result<()> {
    let txt = client
        .get("https://check.torproject.org/api/ip")
        .send().await?
        .text().await?;
    println!("{txt}"); // {"IsTor":true,"IP":"..."}
    Ok(())
}

Switching circuits (a new IP) is done through Tor's control port (usually 9051): you send a NEWNYM signal. You do it either by hand over the control-port protocol or with a wrapper crate. After NEWNYM you usually wait a moment (Tor rate-limits circuit changes to roughly once every 10 seconds).

7.2 Embedded Tor: arti

Arti is a pure-Rust implementation of Tor from the Tor Project itself. You can embed Tor right inside your application with no external daemon. The high-level client API lives in the arti-client crate (docs.rs).

arti-client = "..."   # check the current version: cargo add arti-client
tor-rtcompat = "..."

use arti_client::{TorClient, TorClientConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = TorClientConfig::default();
    // Bring up the embedded Tor client and wait for bootstrap.
    let tor = TorClient::create_bootstrapped(config).await?;

    // From here you can open anonymous TCP streams (AsyncRead/AsyncWrite)
    // and send HTTP over them by hand or via hyper.
    let mut _stream = tor.connect(("example.com", 80)).await?;
    // ... send the HTTP request over the stream ...
    Ok(())
}

There's also a glue crate, artiqwest, that routes HTTP requests through arti with a reqwest-like API (get/post), including .onion and websockets.

Downsides of arti: the API isn't stabilized yet (breaking changes are possible before 1.x), and it doesn't fully support every C-Tor feature. On the plus side, no external process and easier deployment.

Disclaimer: Tor exit nodes are overloaded, slow, and often banned on popular sites. Tor is great for anonymity and reaching .onion, but poor as a "free pool of fast proxies."

8. HTTPS / TLS

With reqwest 0.13 HTTPS works out of the box — the default backend is rustls (pure Rust, no system OpenSSL required). Usually there's nothing to configure.

8.1 Choosing a TLS backend

# rustls (default) — cross-platform, no OpenSSL needed
reqwest = { version = "0.13" }

# or system TLS (schannel on Windows, Secure Transport on macOS, OpenSSL on Linux)
reqwest = { version = "0.13", default-features = false, features = ["native-tls"] }

# or OpenSSL compiled in statically (handy for building a self-contained binary)
reqwest = { version = "0.13", default-features = false, features = ["native-tls-vendored"] }

TLS backends: rustls, native-tls, openssl.

8.2 Ignoring certificate errors (dangerous!)

Sometimes you need to scrape a site with a self-signed or expired certificate. You can turn off verification — but only for testing and trusted hosts, since it removes MITM protection:

let client = reqwest::Client::builder()
    .danger_accept_invalid_certs(true) // unsafe
    .build()?;

8.3 A custom root cert / client cert

use reqwest::{Certificate, Identity};

// Add a corporate/self-signed CA:
let ca = Certificate::from_pem(&std::fs::read("my-ca.pem")?)?;

// Client certificate (mTLS):
let id = Identity::from_pem(&std::fs::read("client.pem")?)?;

let client = reqwest::Client::builder()
    .add_root_certificate(ca)
    .identity(id)
    .build()?;

9. Working with cookies

Cookies are needed for sessions, authentication, and getting past some defenses. reqwest can store and re-send them automatically between requests.

Enable the feature:

reqwest = { version = "0.13", features = ["cookies"] }

9.1 Automatic cookie store

let client = reqwest::Client::builder()
    .cookie_store(true) // enable automatic cookie storage
    .build()?;

// 1) log in — the server returns Set-Cookie, and the client remembers it
client.post("https://site.example/login")
    .form(&[("user", "alice"), ("pass", "secret")])
    .send().await?;

// 2) subsequent requests automatically go out with those cookies
let dashboard = client.get("https://site.example/dashboard")
    .send().await?
    .text().await?;

9.2 A custom cookie jar (reading values / reuse)

When you need to read/set cookies by hand or carry them between sessions:

use std::sync::Arc;
use reqwest::cookie::Jar;
use reqwest::Url;

let jar = Arc::new(Jar::default());

// Seed a cookie manually up front:
let url: Url = "https://site.example/".parse()?;
jar.add_cookie_str("session=abc123; Domain=site.example; Path=/", &url);

let client = reqwest::Client::builder()
    .cookie_provider(jar.clone()) // use our jar
    .build()?;

// after some requests you can read the accumulated cookies back out of the jar

9.3 Cookies by hand via a header

If you'd rather not turn on the automation, you can just pass cookies as a header:

let resp = client.get(url)
    .header(reqwest::header::COOKIE, "session=abc123; lang=en")
    .send().await?;

10. Response status and headers

Before you parse HTML you almost always want to check the page actually arrived cleanly (200) and isn't a 404/403/429/5xx.

use reqwest::StatusCode;
use reqwest::header::{CONTENT_TYPE, CONTENT_LENGTH, LOCATION, RETRY_AFTER};

async fn fetch(client: &reqwest::Client, url: &str) -> anyhow::Result<Option<String>> {
    let resp = client.get(url).send().await?;

    let status = resp.status();
    println!("HTTP {} ({})", status.as_u16(), status.canonical_reason().unwrap_or(""));

    // Convenient status-category checks:
    if status.is_success() {            // 2xx
        // read the headers we care about
        let headers = resp.headers();

        if let Some(ct) = headers.get(CONTENT_TYPE).and_then(|v| v.to_str().ok()) {
            println!("Content-Type: {ct}");
            // parse HTML only; skip images
            if !ct.contains("text/html") {
                return Ok(None);
            }
        }
        if let Some(len) = headers.get(CONTENT_LENGTH) {
            println!("Content-Length: {len:?}");
        }

        let body = resp.text().await?;
        return Ok(Some(body));
    }

    if status.is_redirection() {        // 3xx
        if let Some(loc) = resp.headers().get(LOCATION).and_then(|v| v.to_str().ok()) {
            println!("Redirect to: {loc}");
        }
    }

    if status == StatusCode::TOO_MANY_REQUESTS {  // 429
        // the server is asking us to wait
        if let Some(ra) = resp.headers().get(RETRY_AFTER).and_then(|v| v.to_str().ok()) {
            println!("We're being throttled. Retry-After: {ra} sec");
        }
    }

    Ok(None)
}

Useful methods:

• resp.status() -> StatusCode; there's .is_success(), .is_client_error(), .is_server_error(), .is_redirection().
• resp.error_for_status() — turns 4xx/5xx into an Err, handy with ?.
• resp.headers() -> HeaderMap, iterates like a map.
• resp.url() — the final URL after redirects.
• resp.content_length() — body length, when known.

By default reqwest follows redirects itself (up to 10). Tune it with .redirect(reqwest::redirect::Policy::none()) or .limited(n).

11. URL storage and queues

Any crawler is essentially a loop: "pull a URL off the queue -> download it -> extract new links -> put them back." You need two structures:

• a frontier (queue) — what to fetch next;
• a visited/seen set — so you don't fetch the same thing twice.

11.1 In memory (for small jobs)

use std::collections::{VecDeque, HashSet};

struct Frontier {
    queue: VecDeque<String>,
    seen: HashSet<String>,
}

impl Frontier {
    fn new() -> Self {
        Self { queue: VecDeque::new(), seen: HashSet::new() }
    }
    /// Add a URL if we haven't seen it.
    fn push(&mut self, url: String) {
        if self.seen.insert(url.clone()) {  // insert returns false if it was already there
            self.queue.push_back(url);
        }
    }
    fn pop(&mut self) -> Option<String> {
        self.queue.pop_front()
    }
}

For concurrent access from several async tasks, build the queue on channels: tokio::sync::mpsc, flume, or crossbeam-channel. Workers read from the channel and write new links back.

11.2 Persistence (for big, long crawls)

At millions of URLs you'll run out of memory, and if the process crashes you lose all progress. So you move the queue and the "seen" set into external storage:

Deduplication at scale: keeping every URL in a HashSet is expensive. People use:

• URL normalization (strip #fragment, sort query params, lowercase the host) via the url crate — otherwise one page shows up under several URLs;
• a hash of the URL (say xxhash-rust / blake3) instead of the string itself;
• a Bloom filter (bloomfilter) — a compact probabilistic structure for "maybe seen / definitely not seen."

This section is a flyover. In practice the choice depends on scale: for a couple thousand pages an in-memory HashSet is fine; for an industrial crawler it's a Redis queue + Bloom filter + URL normalization.

12. Extras people forget

The things that didn't make the original checklist but without which a real scraper breaks or gets banned.

12.1 Politeness, robots.txt, and rate limiting

• robots.txt — the file where a site states what may and may not be crawled. Ethical (and sometimes legally safer) scraping respects it. Crates: texting_robots, robotstxt.
• Delays between requests to the same domain — so you don't knock the site over or earn a ban. The simplest option is tokio::time::sleep. The professional one is the governor rate limiter (token bucket):

use std::num::NonZeroU32;
use governor::{Quota, RateLimiter};

// no more than 5 requests per second
let limiter = RateLimiter::direct(Quota::per_second(NonZeroU32::new(5).unwrap()));

// before each request:
limiter.until_ready().await;
// client.get(...).send().await?;

12.2 Retries and backoff

Networks are flaky: timeouts, 503s, dropped connections. You need retries with exponential backoff (1s -> 2s -> 4s...). The easiest route is the reqwest-middleware + reqwest-retry crates:

reqwest-middleware = "0.5"
reqwest-retry = "0.9"

use reqwest_middleware::ClientBuilder;
use reqwest_retry::{RetryTransientMiddleware, policies::ExponentialBackoff};

let retry_policy = ExponentialBackoff::builder().build_with_max_retries(3);
let client = ClientBuilder::new(reqwest::Client::new())
    .with(RetryTransientMiddleware::new_with_policy(retry_policy))
    .build();
// from here client.get(...).send().await retries automatically

12.3 JavaScript pages (headless browsers)

reqwest + scraper only see the initial HTML. If content is drawn in by JavaScript (a React/Vue SPA), it won't be in that HTML. Options:

1. Find the hidden API (see §3.2) — nearly always the best path: faster, more reliable, easier.
2. Drive a real browser (which renders the JS):

A browser costs tens of times more in resources, so use it only where there's genuinely no way around the JS. For the trade-offs, see our overview of headless browser scraping.

12.4 User-Agent, headers, and anti-bot defenses

Sites tell bots from humans. Minimal camouflage:

• a plausible User-Agent (not reqwest/0.13!);
• a realistic header set: Accept, Accept-Language, Accept-Encoding, Referer, Sec-Fetch-*;
• rotation of User-Agent and proxies;
• human-like delays.

Serious defenses (Cloudflare, DataDome, PerimeterX) additionally inspect the TLS fingerprint (JA3/JA4) and the order of HTTP/2 headers. Plain reqwest emits a "Rust-shaped" fingerprint that differs from Chrome. To get around it there are crates that imitate a browser's fingerprint using curl-impersonate, for example rquest. It's an arms race — no guarantees. When you hit a wall, solving CAPTCHAs and browser fingerprinting are usually the deciding factors.

12.5 Error handling and logging

• Errors: anyhow for applications (ergonomic ? and context), thiserror for libraries (your own error types). Don't panic on every 404 — treat it as an ordinary result.
• Logging/tracing: tracing + tracing-subscriber (or log + env_logger). Logs show you where the slowdowns and bans are.

use anyhow::Context;

let html = client.get(url).send().await
    .with_context(|| format!("failed to download {url}"))?
    .text().await
    .context("failed to read the body")?;

13. Architecture of a real crawler

The blueprint of a production scraper that pulls together everything above:

Key principles:

• one shared Client (connection pool), cloned into the workers;
• concurrency capped by a semaphore, per-domain speed capped by a rate limiter;
• every network call wrapped in retry/backoff;
• the queue and the seen set are the single source of truth about progress.

If you'd rather not build it all by hand, there are ready-made crawler frameworks such as spider.

14. Pros and cons of doing it in Rust

Pros of a Rust implementation

• Performance: speed close to C/C++. At high volumes Rust outruns Python (requests/BeautifulSoup) and Go several times over on CPU and memory.
• Memory: minimal footprint, no GC pauses — crucial for millions of pages and long-running crawls.
• Fearless concurrency: the type system and borrow checker catch data races at compile time. A huge win for a concurrent scraper.
• Reliability: explicit error handling (Result), Option instead of null — fewer production crashes.
• A single static binary: easy to deploy, no interpreter or dependencies to drag along.
• A mature async ecosystem: tokio + reqwest is production-grade.

Cons

• Learning curve: the borrow checker, lifetimes, and async take longer to learn than a "write it in an evening" Python scraper.
• Development speed: a prototype in Python/scrapy comes together faster. For a one-off, Rust can be overkill.
• Dynamic content: fewer out-of-the-box options for JS rendering and anti-bot evasion than Python (which has Playwright, Scrapy, undetected-chromedriver, and so on).
• Compilation: long build times, especially with heavy dependencies.
• Fewer ready-made frameworks: Python has a full-blown scrapy; in Rust you usually assemble the pipeline from bricks yourself (though spider and others exist).

Bottom line: Rust pays off when the scraping is large-scale, ongoing, and resource-sensitive (millions of pages, hard speed/memory requirements, a long-lived service). For a one-time "scrape 500 pages," Python is usually faster in total effort.

15. Legal and ethical considerations

Technically you can do a lot — but that doesn't mean you should. In brief:

• robots.txt and ToS: respect a site's robots.txt and terms of service.
• Load: don't take someone's server down — cap your request rate and scrape off-peak.
• Personal data: collecting personal data is regulated by law (GDPR in the EU, the CCPA in California, and similar laws elsewhere). Tread carefully.
• Copyright: content may be protected; wholesale copying/republication can be unlawful. On the case law, US courts have generally held that scraping publicly available data isn't a CFAA violation (see hiQ Labs v. LinkedIn), but that's a narrow question, not blanket permission.
• Identify yourself: it's reasonable to put a contact in your User-Agent so a site admin can reach you rather than ban blindly.

This is general orientation, not legal advice — when in doubt, consult a lawyer.

Prefer to skip the build and just get the data? scraping.pro runs custom scraping and data extraction as a done-for-you service, including large-scale crawls, data as a service feeds, and ongoing price and review monitoring. We handle the proxies, anti-bot, and maintenance so you can work with clean, structured output.

A quick crate cheat sheet