C# Web Scraping: Complete Guide from Simple to Advanced

A hands-on reference for C# web scraping on .NET. The examples target modern .NET 8 (LTS) and newer and idiomatic C# (async/await, HttpClient, record, and so on). Where a feature first shipped in a specific version, that's called out.

Table of contents

1. What web scraping is and what it's made of
2. Project setup and the packages you need
3. Fetching a page: HttpClient
4. Request headers, User-Agent, and compression (gzip/br)
5. Libraries for parsing the content
6. Fixing broken characters (encodings)
7. Reading the response status and other headers
8. Working with cookies
9. Working with HTTPS/SSL
10. Using proxies
11. Scraping through Tor
12. Concurrency and parallelism
13. Resilience: timeouts, retries, politeness, robots.txt
14. Storing URLs and queues (frontier, deduplication)
15. JavaScript content: headless browsers
16. Saving your results
17. Main pros and cons of a C# implementation
18. Legal and ethical considerations

1. What web scraping is and what it's made of

Web scraping is the automated fetching of web pages and extraction of structured data from them. Logically, any scraper is made up of four parts:

1. Downloader (fetcher) — downloads the HTML for a URL.
2. Content parser — turns HTML into a tree you can pull data from with selectors.
3. Extraction and data normalization — pull out the fields you need, clean them, and coerce them to types.
4. Scheduler (frontier) — manages the URL queue, deduplication, request rate, and retries.

A good scraper is much more than "download and parse." Around 80% of the difficulty is reliability: encodings, timeouts, retries, ban avoidance, rate limiting, and queue management. Most of this article is about exactly that.

2. Project setup and the packages you need

dotnet new console -n Scraper
cd Scraper

# HTML parsing — pick one or use both
dotnet add package HtmlAgilityPack
dotnet add package AngleSharp

# Support for legacy encodings (windows-1252, windows-1251, etc.) — required on .NET Core+
dotnet add package System.Text.Encoding.CodePages

# Resilience (retries, circuit breaker)
dotnet add package Microsoft.Extensions.Http.Polly

# Optional — a headless browser for JS pages
dotnet add package Microsoft.Playwright

Official resources for these packages:

• HtmlAgilityPack — GitHub · NuGet
• AngleSharp — site and docs · GitHub · NuGet
• System.Text.Encoding.CodePages — NuGet · docs
• Polly / Microsoft.Extensions.Http.Polly — docs · GitHub · NuGet
• Microsoft.Playwright (.NET) — docs · NuGet

3. Fetching a page: HttpClient

In modern .NET, the one right tool is HttpClient. The old WebClient and HttpWebRequest are considered legacy and you shouldn't reach for them in new code.

The golden rule: reuse HttpClient

HttpClient is designed to be long-lived. Creating a new instance per request (using var client = new HttpClient()) is a classic mistake: it leads to socket exhaustion (ports get stuck in TIME_WAIT). Use a single shared instance per application, or use IHttpClientFactory. Microsoft's HttpClient usage guidelines cover this in detail.

using System.Net;
using System.Net.Http;

// One handler + one client for the whole app (or a DI singleton)
var handler = new SocketsHttpHandler
{
    AutomaticDecompression = DecompressionMethods.All, // gzip, deflate, brotli
    PooledConnectionLifetime = TimeSpan.FromMinutes(2), // guards against stale DNS
    MaxConnectionsPerServer = 20,
    AllowAutoRedirect = true,
    MaxAutomaticRedirections = 10
};

var http = new HttpClient(handler)
{
    Timeout = TimeSpan.FromSeconds(30)
};

http.DefaultRequestHeaders.UserAgent.ParseAdd(
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " +
    "(KHTML, like Gecko) Chrome/124.0 Safari/537.36");

// Simplest possible fetch
string html = await http.GetStringAsync("https://example.com");

Prefer HttpRequestMessage and HttpResponseMessage

GetStringAsync is convenient, but it hides the status code, headers, and encoding. For a real scraper, take the full response so you control everything:

using var request = new HttpRequestMessage(HttpMethod.Get, "https://example.com");
request.Headers.Referrer = new Uri("https://google.com");

using var response = await http.SendAsync(
    request, HttpCompletionOption.ResponseHeadersRead);

response.EnsureSuccessStatusCode(); // throws on 4xx/5xx (optional)

byte[] bytes = await response.Content.ReadAsByteArrayAsync();
// bytes -> we decode to a string ourselves (see the encodings section)

HttpCompletionOption.ResponseHeadersRead returns control as soon as the headers arrive, without waiting for the whole body. Useful for large responses and streaming.

4. Request headers, User-Agent, and compression

Many sites block requests that lack "human" headers. A baseline set worth sending:

http.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 ... Chrome/124.0 ...");
http.DefaultRequestHeaders.Accept.ParseAdd("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
http.DefaultRequestHeaders.AcceptLanguage.ParseAdd("en-US,en;q=0.9");
http.DefaultRequestHeaders.AcceptEncoding.ParseAdd("gzip, deflate, br");

Compression matters: don't declare Accept-Encoding: gzip, br by hand unless you've enabled AutomaticDecompression. Otherwise the server sends a compressed body and you get garbage. The right way is to set AutomaticDecompression = DecompressionMethods.All on the handler (as in section 3) — then .NET adds the header and decompresses for you. Brotli (br) has been supported since .NET Core 3.0.

Rotating the User-Agent and sending a realistic header order is a common trick against simple anti-bot systems, but it's no silver bullet against serious defenses (see sections 13 and 18). Pairing it with rotating proxies is far more effective.

5. Libraries for parsing the content

Once the HTML is downloaded, you need to parse it into a tree and pull data out with selectors. The two main players in .NET are HtmlAgilityPack and AngleSharp.

5.1 HtmlAgilityPack (XPath)

A time-tested classic. It works with XPath and forgives dirty, invalid HTML.

using HtmlAgilityPack;

var doc = new HtmlDocument();
doc.LoadHtml(html);

// Page title
string? title = doc.DocumentNode
    .SelectSingleNode("//title")?.InnerText.Trim();

// All links
foreach (var a in doc.DocumentNode.SelectNodes("//a[@href]") ?? Enumerable.Empty<HtmlNode>())
{
    string href = a.GetAttributeValue("href", "");
    string text = HtmlEntity.DeEntitize(a.InnerText).Trim();
    Console.WriteLine($"{text} -> {href}");
}

// Search by class via XPath
var prices = doc.DocumentNode
    .SelectNodes("//span[contains(@class,'price')]");

⚠️ SelectNodes returns null when nothing matches (not an empty collection) — always null-check or use ?? Enumerable.Empty<...>(). Call HtmlEntity.DeEntitize to turn &,  , and friends back into normal characters.

5.2 AngleSharp (CSS selectors, a real DOM)

A modern library that implements the W3C standards. It parses HTML the way a browser does and supports CSS selectors (querySelector / querySelectorAll), just like JavaScript. Often more comfortable, especially if you come from front-end work.

using AngleSharp;
using AngleSharp.Dom;

var config = Configuration.Default;
var context = BrowsingContext.New(config);
var document = await context.OpenAsync(req => req.Content(html));

// CSS selectors, just like in the browser
string? title = document.QuerySelector("title")?.TextContent.Trim();

var cards = document.QuerySelectorAll("div.product-card");
foreach (var card in cards)
{
    string? name  = card.QuerySelector("h2.name")?.TextContent.Trim();
    string? price = card.QuerySelector(".price")?.TextContent.Trim();
    string? link  = card.QuerySelector("a")?.GetAttribute("href");
    Console.WriteLine($"{name} | {price} | {link}");
}

AngleSharp does more: it can load a whole page by URL, parse forms, and work with the CSSOM. It's a full DOM engine, not just an HTML parser.

5.3 Fizzler, regex, and when to pick what

• Fizzler — adds CSS selectors on top of HtmlAgilityPack (.QuerySelectorAll(...)) when you want CSS but want to stay on HAP.
• Regex over HTML is an anti-pattern. HTML is not a regular language; regex breaks on nesting, attributes in arbitrary order, and comments. Regex is only appropriate for cleaning up already-extracted text (for example, pulling a number out of "Price: $1,299").

What to choose:

Tip: very often the data is already on the page as JSON inside <script type="application/ld+json"> or in an SPA's state. Parsing that JSON is more robust than parsing the markup.

6. Fixing broken characters (encodings)

This is the most common headache on sites that don't use UTF-8. The symptom is mojibake — garbled characters instead of readable text (привет or éèÃ). The cause is almost always the wrong encoding when decoding bytes into a string.

While UTF-8 dominates today, you'll still hit legacy single-byte encodings in the wild: windows-1252 (Western European), ISO-8859-1, windows-1251 (Cyrillic), Shift_JIS (Japanese), and so on.

Step 1. Register the code-pages provider

On .NET Core / .NET 5+, the old single-byte encodings are not enabled by default. Without them, Encoding.GetEncoding(1252) throws. Add the System.Text.Encoding.CodePages package (see CodePagesEncodingProvider) and register it once at startup:

using System.Text;

// At the very start of the program (Main / startup)
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);

var win1252 = Encoding.GetEncoding(1252);   // or GetEncoding("windows-1252")

Step 2. Don't use GetStringAsync blindly

GetStringAsync decodes the body based on the Content-Type; charset=... header. If the site lies in the header or omits the charset, you get mush. The robust path is to download the bytes and determine the encoding yourself.

static async Task<string> GetHtmlAsync(HttpClient http, string url)
{
    using var resp = await http.GetAsync(url);
    byte[] bytes = await resp.Content.ReadAsByteArrayAsync();

    // 1) charset from the HTTP header
    string? charset = resp.Content.Headers.ContentType?.CharSet;

    // 2) if the header has none — look in <meta charset> / <meta http-equiv>
    if (string.IsNullOrEmpty(charset))
        charset = SniffCharsetFromMeta(bytes);

    Encoding enc;
    try
    {
        enc = string.IsNullOrEmpty(charset)
            ? Encoding.UTF8
            : Encoding.GetEncoding(charset.Trim('"', '\''));
    }
    catch
    {
        enc = Encoding.UTF8; // fallback
    }

    return enc.GetString(bytes);
}

// Rough charset detection from the first bytes (meta tag)
static string? SniffCharsetFromMeta(byte[] bytes)
{
    // meta is always in the ASCII-compatible part; read the first ~2 KB as latin1
    string head = Encoding.GetEncoding("ISO-8859-1")
        .GetString(bytes, 0, Math.Min(bytes.Length, 2048));

    var m = System.Text.RegularExpressions.Regex.Match(
        head,
        @"charset\s*=\s*[""']?\s*([a-zA-Z0-9\-]+)",
        System.Text.RegularExpressions.RegexOptions.IgnoreCase);

    return m.Success ? m.Groups[1].Value : null;
}

This is one of the rare cases where regex over HTML is fine — only to pull the encoding name out of the meta tag, nothing more.

Step 3. Let the parser handle it (often easiest)

Both AngleSharp and HtmlAgilityPack can detect the encoding from the bytes themselves — if you feed them a stream/bytes rather than an already-decoded string.

// HtmlAgilityPack: detects the encoding from <meta> itself
using var resp = await http.GetAsync(url);
await using var stream = await resp.Content.ReadAsStreamAsync();

var doc = new HtmlDocument
{
    OptionReadEncoding = true // read encoding from the document
};
doc.Load(stream); // detectEncodingFromByteOrderMarks = true by default

// AngleSharp: pass the stream and it sorts out the encoding
using var resp = await http.GetAsync(url);
await using var stream = await resp.Content.ReadAsStreamAsync();

var context = BrowsingContext.New(Configuration.Default);
var document = await context.OpenAsync(req => req.Content(stream));

The practical algorithm: register CodePages → hand the bytes/stream to the parser → if you still get mojibake, check the charset in the header and meta tag and decode manually with the right encoding.

7. Reading the response status and other headers

HttpResponseMessage gives you full access to the status and headers — you need this for retry logic, redirect handling, and detecting bans.

using var resp = await http.GetAsync(url);

int statusCode = (int)resp.StatusCode;       // 200, 404, 503...
bool ok        = resp.IsSuccessStatusCode;   // true for 2xx
var reason     = resp.ReasonPhrase;          // "OK", "Not Found"

// Response headers
if (resp.Headers.TryGetValues("Server", out var server))
    Console.WriteLine("Server: " + string.Join(",", server));

// Content headers
string? contentType   = resp.Content.Headers.ContentType?.MediaType; // text/html
long?   contentLength = resp.Content.Headers.ContentLength;

// Useful for a scraper
var retryAfter = resp.Headers.RetryAfter;     // on 429/503 — when to retry
var location   = resp.Headers.Location;       // where it redirects (if AllowAutoRedirect=false)

switch (statusCode)
{
    case 200: /* parse */ break;
    case 301 or 302: /* redirect */ break;
    case 403: /* possibly a ban / needs cookies or UA */ break;
    case 404: /* no page — drop it from the queue */ break;
    case 429: /* too many requests — slow down, see Retry-After */ break;
    case >= 500: /* server error — retry later */ break;
}

The split matters: general headers live in resp.Headers, while body-related ones (Content-Type, Content-Length, Content-Encoding) live in resp.Content.Headers. If you look for Content-Type in resp.Headers, it won't be there.

8. Working with cookies

Cookies are needed for sessions, authentication, and passing "checks." In .NET they're handled by a CookieContainer attached to the handler.

var cookies = new CookieContainer();

var handler = new SocketsHttpHandler
{
    CookieContainer = cookies,
    UseCookies = true // enabled by default
};
var http = new HttpClient(handler);

// Requests automatically send and store cookies from this container
await http.GetAsync("https://example.com/login");

// You can set a cookie manually (e.g. a session token)
cookies.Add(new Uri("https://example.com"),
    new Cookie("session_id", "abc123") { Path = "/" });

// Read the current cookies for a domain
foreach (Cookie c in cookies.GetCookies(new Uri("https://example.com")))
    Console.WriteLine($"{c.Name} = {c.Value}");

Fine points:

• One container = one session. For parallel scraping with different "identities," create a separate handler + container per session/proxy.
• To disable cookies (so each request is "clean"), set UseCookies = false.
• You can persist/restore a session between runs by serializing the cookies (name, value, domain, path, expiry) to JSON. This is the backbone of scraping behind a login.

9. Working with HTTPS/SSL

By default HttpClient establishes the TLS connection and validates the server certificate for you. Usually there's nothing to configure. You only need to intervene in rare cases.

Ignoring certificate errors (careful!)

Sometimes a site has a broken/self-signed certificate and you still need to download it. You can disable validation like this, but only deliberately — you lose protection against MITM:

var handler = new SocketsHttpHandler
{
    SslOptions = new System.Net.Security.SslClientAuthenticationOptions
    {
        // WARNING: accepts any certificate. Debugging / trusted tasks only.
        RemoteCertificateValidationCallback = (sender, cert, chain, errors) => true
    }
};

(The classic HttpClientHandler equivalent is ServerCertificateCustomValidationCallback, with a ready-made stub HttpClientHandler.DangerousAcceptAnyServerCertificateValidator.)

Controlling the TLS version

var handler = new SocketsHttpHandler
{
    SslOptions = new System.Net.Security.SslClientAuthenticationOptions
    {
        EnabledSslProtocols = System.Security.Authentication.SslProtocols.Tls12
                            | System.Security.Authentication.SslProtocols.Tls13
    }
};

Don't disable certificate validation "just in case" in production. It opens the door to traffic tampering. Use it surgically, and only where you truly need it.

10. Using proxies

Proxies help you (a) get around IP-based blocks and (b) spread load and cut ban risk by rotating addresses.

Basic HTTP proxy setup

var proxy = new WebProxy("http://proxy-host:8080")
{
    Credentials = new NetworkCredential("user", "password") // if auth is needed
};

var handler = new SocketsHttpHandler
{
    Proxy = proxy,
    UseProxy = true
};
var http = new HttpClient(handler);

SOCKS proxies (native in .NET 6+)

Since .NET 6, the socks4, socks4a, and socks5 schemes are supported right in WebProxy — no third-party libraries needed:

var proxy = new WebProxy("socks5://127.0.0.1:1080");
var handler = new SocketsHttpHandler { Proxy = proxy, UseProxy = true };

Rotating proxies

The simplest strategy is a proxy pool: give each proxy its own HttpClient (the handler with the proxy is reused!) and take them round-robin or at random. Proxies that error or time out get temporarily "penalized."

public sealed class ProxyPool
{
    private readonly HttpClient[] _clients;
    private int _index;

    public ProxyPool(IEnumerable<string> proxyUrls)
    {
        _clients = proxyUrls.Select(url =>
        {
            var handler = new SocketsHttpHandler
            {
                Proxy = new WebProxy(url),
                UseProxy = true,
                AutomaticDecompression = DecompressionMethods.All
            };
            return new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(30) };
        }).ToArray();
    }

    public HttpClient Next()
    {
        int i = Interlocked.Increment(ref _index);
        return _clients[(i & int.MaxValue) % _clients.Length];
    }
}

Important: don't create a new handler-with-proxy per request — that brings us right back to socket exhaustion. Create one HttpClient per proxy and reuse it. For advice on choosing between datacenter and residential IPs, see our guide to proxies for scraping.

11. Scraping through Tor

Tor gives you anonymity and free IP rotation. It's essentially a local SOCKS5 proxy.

Connecting to Tor as SOCKS5

After installing Tor (the tor daemon or Tor Browser), a SOCKS5 proxy comes up on the machine, by default at 127.0.0.1:9050 (Tor Browser uses 9150).

var handler = new SocketsHttpHandler
{
    Proxy = new WebProxy("socks5://127.0.0.1:9050"), // .NET 6+
    UseProxy = true,
    AutomaticDecompression = DecompressionMethods.All
};
var http = new HttpClient(handler);

string html = await http.GetStringAsync("https://check.torproject.org");

Changing IP (new circuit) via the control port

The main trick is that you can request a new circuit (a new exit IP) with the SIGNAL NEWNYM command on the control port (9051 by default) — see the Tor control-protocol spec. Enable it in torrc:

ControlPort 9051
# and authentication, e.g. a hashed password:
HashedControlPassword 16:...   # from `tor --hash-password "mypass"`

Requesting a new circuit over raw TCP:

using System.Net.Sockets;
using System.Text;

static async Task NewTorIdentityAsync(string password,
    string host = "127.0.0.1", int controlPort = 9051)
{
    using var client = new TcpClient();
    await client.ConnectAsync(host, controlPort);
    await using var stream = client.GetStream();
    using var reader = new StreamReader(stream, Encoding.ASCII);
    using var writer = new StreamWriter(stream, Encoding.ASCII) { AutoFlush = true };

    await writer.WriteLineAsync($"AUTHENTICATE \"{password}\"");
    var authResp = await reader.ReadLineAsync(); // expect "250 OK"

    await writer.WriteLineAsync("SIGNAL NEWNYM");
    var sigResp = await reader.ReadLineAsync();   // "250 OK"
}

Keep in mind: Tor is slow, and many sites throttle traffic from exit nodes. There's a limit between NEWNYM calls (MaxCircuitDirtiness, ~10 sec), so you can't rotate IP instantly on every request. For high-speed scraping, commercial proxies are usually more practical; Tor is for anonymity.

12. Concurrency and parallelism

In network scraping, the bottleneck is waiting for a response, not the CPU. So you don't want classic "multithreading" — you want asynchronous parallelism with a cap on concurrent requests. Firing thousands of requests at once is a bad idea: you'll clog the network, exhaust connections, and get banned.

Approach 1: Parallel.ForEachAsync (.NET 6+) — the simplest

var urls = new List<string> { /* ... */ };
var results = new System.Collections.Concurrent.ConcurrentBag<string>();

await Parallel.ForEachAsync(
    urls,
    new ParallelOptions { MaxDegreeOfParallelism = 8 }, // no more than 8 at once
    async (url, ct) =>
    {
        try
        {
            string html = await http.GetStringAsync(url, ct);
            results.Add(Parse(html));
        }
        catch (Exception ex)
        {
            Console.Error.WriteLine($"FAIL {url}: {ex.Message}");
        }
    });

Approach 2: SemaphoreSlim + Task.WhenAll — flexible control

var gate = new SemaphoreSlim(initialCount: 8); // at most 8 in parallel

async Task<string?> FetchAsync(string url)
{
    await gate.WaitAsync();
    try
    {
        return await http.GetStringAsync(url);
    }
    catch { return null; }
    finally { gate.Release(); }
}

string?[] pages = await Task.WhenAll(urls.Select(FetchAsync));

Approach 3: System.Threading.Channels — a producer/consumer pipeline

For a long-running crawler this is the best pattern: one URL queue, several consumer workers. It pairs well with section 14.

using System.Threading.Channels;

var channel = Channel.CreateBounded<string>(new BoundedChannelOptions(1000)
{
    SingleReader = false,
    SingleWriter = false
});

// Start N workers
int workers = 8;
var consumers = Enumerable.Range(0, workers).Select(_ => Task.Run(async () =>
{
    await foreach (string url in channel.Reader.ReadAllAsync())
    {
        try
        {
            string html = await http.GetStringAsync(url);
            var newLinks = ExtractLinks(html);
            foreach (var link in newLinks)
                await channel.Writer.WriteAsync(link); // add new URLs to the queue
        }
        catch { /* log + retry */ }
    }
})).ToArray();

// Seed with starting URLs
foreach (var seed in seeds)
    await channel.Writer.WriteAsync(seed);

// channel.Writer.Complete(); // when we decide crawling is done
await Task.WhenAll(consumers);

Tune the degree of parallelism to the specific site: 4–16 is a typical range. Hundreds of simultaneous requests to a single domain is a DoS and an almost-guaranteed ban.

13. Resilience: timeouts, retries, politeness, robots.txt

This wasn't in the original checklist, but a production scraper can't live without it.

Retries with exponential backoff (Polly)

The Polly library provides declarative retry policies, circuit breakers, and timeouts. It's especially elegant with IHttpClientFactory:

using Polly;
using Polly.Extensions.Http;

var retryPolicy = HttpPolicyExtensions
    .HandleTransientHttpError()                 // 5xx, 408
    .OrResult(r => (int)r.StatusCode == 429)    // too many requests
    .WaitAndRetryAsync(
        retryCount: 4,
        sleepDurationProvider: attempt =>
            TimeSpan.FromSeconds(Math.Pow(2, attempt))     // 2,4,8,16 s
            + TimeSpan.FromMilliseconds(Random.Shared.Next(0, 1000)) // jitter
    );

// Registration via DI:
// services.AddHttpClient("scraper").AddPolicyHandler(retryPolicy);

Politeness (rate limiting) and robots.txt

• A delay between requests to the same domain (say 0.5–2 sec) reduces load on the site and the risk of a ban. .NET 7+ ships System.Threading.RateLimiting.
• robots.txt is the file with rules for bots (Disallow, Crawl-delay). It isn't legally binding everywhere, but ignoring it is bad form and invites conflict. Respect Crawl-delay and disallowed sections.

// Simplest "polite" per-domain delay
var lastHit = new System.Collections.Concurrent.ConcurrentDictionary<string, DateTime>();

async Task PolitelyAsync(Uri uri, TimeSpan minDelay)
{
    string host = uri.Host;
    if (lastHit.TryGetValue(host, out var prev))
    {
        var wait = minDelay - (DateTime.UtcNow - prev);
        if (wait > TimeSpan.Zero) await Task.Delay(wait);
    }
    lastHit[host] = DateTime.UtcNow;
}

Timeouts and cancellation

Beyond HttpClient.Timeout, use a CancellationToken (shared across the whole crawler) so you can cleanly shut the scraper down on Ctrl+C and not hang the process on dead connections.

When a site fights back with CAPTCHAs or aggressive fingerprinting, retries and politeness alone won't save you — that's where a headless browser (section 15) or a managed scraping service comes in.

14. Storing URLs and queues (frontier)

The URL queue to crawl is called the frontier. Its jobs: store what's still to be fetched, and never fetch the same thing twice.

Deduplication (the set of visited URLs)

// Thread-safe set of already-seen URLs
var visited = new System.Collections.Concurrent.ConcurrentDictionary<string, byte>();

bool TryEnqueue(string url)
{
    string norm = Normalize(url); // URL normalization is critical!
    return visited.TryAdd(norm, 0); // true if we haven't seen this URL
}

URL normalization is mandatory, otherwise example.com/p?a=1&b=2 and example.com/p?b=2&a=1 count as different pages. At minimum: lowercase the host, drop the #fragment, sort query parameters, strip a trailing slash, and normalize the scheme.

Queue storage options

A minimal SQLite frontier (for restarts)

The idea: store URLs with a status (pending / in_progress / done / failed) and depth. On start you take pending, after fetching you mark it done, and new links are added with INSERT OR IGNORE (a unique index on the URL gives you database-level deduplication).

CREATE TABLE IF NOT EXISTS frontier (
    url     TEXT PRIMARY KEY,    -- normalized URL = dedup
    status  TEXT NOT NULL DEFAULT 'pending',
    depth   INTEGER NOT NULL DEFAULT 0,
    added   TEXT NOT NULL
);

That way you can stop the crawler and resume from the same place — the queue survives a restart.

At large scale you add priorities (important pages first), a depth limit, a per-domain page cap, and the politeness policy right into the frontier.

15. JavaScript content: headless browsers

HttpClient downloads the source HTML, before any JavaScript runs. If the site is an SPA (React/Vue/Angular) and the data is loaded by scripts, it won't be in the source HTML. Options:

1. Find the API. An SPA often calls a JSON endpoint — open DevTools → Network, find the request with the data, and hit it directly with HttpClient. This is faster and more reliable than any browser.
2. A headless browser, when you can't get at the API: it actually renders the page.

Playwright for .NET (recommended)

using Microsoft.Playwright;

using var pw = await Playwright.CreateAsync();
await using var browser = await pw.Chromium.LaunchAsync(
    new() { Headless = true });
var page = await browser.NewPageAsync();

await page.GotoAsync("https://example.com/products");
await page.WaitForSelectorAsync(".product-card"); // wait for the data to appear

// Extract via Playwright locators...
var names = await page.Locator(".product-card h2").AllTextContentsAsync();

// ...or grab the finished HTML and parse it with your usual library
string renderedHtml = await page.ContentAsync();

Alternatives: Selenium WebDriver (the classic, but heavier) and PuppeteerSharp (a Puppeteer port). These days most .NET projects reach for Playwright — it's officially supported by Microsoft and more convenient. For a deeper look, see our guide to headless browser scraping.

Downsides of headless browsers: they're tens of times slower and hungrier for resources than HttpClient. Use them only when rendering is truly unavoidable.

16. Saving your results

The data has to go somewhere. Typical options:

// JSON (System.Text.Json) — handy for nested data
await using var fs = File.Create("data.json");
await System.Text.Json.JsonSerializer.SerializeAsync(fs, items,
    new System.Text.Json.JsonSerializerOptions { WriteIndented = true });

• CSV — for tabular data (the CsvHelper library).
• JSON / JSONL — for nested structures (System.Text.Json); JSONL (one object per line) is handy for streaming large volumes.
• A database (SQLite/PostgreSQL via EF Core or Dapper) — when you need queries, content-based dedup, or incremental updates.

Tip: write results as you go, streaming them out during the crawl instead of holding everything in memory — otherwise a big crawl will hit OutOfMemory.

17. Main pros and cons of a C# implementation

Pros

• Performance and async. async/await, HttpClient, Channels, and Parallel.ForEachAsync give you efficient concurrent I/O out of the box.
• A mature ecosystem. HtmlAgilityPack, AngleSharp, Playwright, Polly — all industrial quality.
• Static typing. Fewer silly bugs in large crawlers, and comfortable refactoring.
• Native SOCKS/proxy since .NET 6, plus easy TLS and cookie handling.
• Cross-platform (.NET runs on Linux/Windows/macOS and drops easily into Docker).

Cons

• Encodings. Legacy single-byte encodings aren't available out of the box — you have to remember CodePagesEncodingProvider (section 6).
• JS sites. Plain HttpClient doesn't execute JS; you need a headless browser, which is heavy and slow.
• Anti-bot systems. Cloudflare, CAPTCHAs, and fingerprinting are hard to get around; an "honest" scraper often runs into the wall of defenses.
• Fragility to markup. Any scraper breaks when a site's HTML structure changes — it needs monitoring and maintenance.
• Fewer ready-made frameworks than Python. Python has all-in-one Scrapy; in .NET you more often assemble the pipeline from building blocks yourself (though there's DotnetSpider and Abot). If you're weighing languages, compare with our Python web scraping guide.

18. Legal and ethical considerations

Being technically able to do something doesn't mean you have the right to. A brief note on what to keep in mind (this is not legal advice):

• A site's Terms of Service may expressly forbid automated collection. Violating them is grounds for a block and complaints.
• Personal data is regulated by law (GDPR in the EU/UK, CCPA in California, and similar). Collecting and storing personal data without a basis is a real risk.
• Copyright. Scraped content is often protected; republishing it may infringe.
• Load. Aggressive scraping is effectively a DoS. Respect Crawl-delay, limit your RPS, and don't take someone's server down.
• robots.txt and public APIs are the preferred path. If a site has an official API, it's almost always better to use it.

On the legal front, US case law such as hiQ Labs v. LinkedIn has shaped how courts view scraping of publicly available data under the Computer Fraud and Abuse Act (CFAA) — but the details vary by jurisdiction, and personal data is a separate question governed by GDPR/CCPA.

The basic rule: scrape politely, identifiably (an honest User-Agent where appropriate), and respect the site's limits and the law in your jurisdiction.

Prefer a done-for-you pipeline?

Building and, more importantly, maintaining a resilient C# scraper — proxies, retries, encodings, headless rendering, and anti-bot handling — is a real engineering commitment. If you'd rather just receive clean data on a schedule, scraping.pro can run it for you as a custom data extraction service or ongoing data as a service, and you consume the results through an API or file drop.

Use this document as a step-by-step plan: each section is a separate "brick" of a scraper, assembled into the overall pipeline frontier → fetcher → parser → storage.