Golang Web Scraping: The Complete Guide

Go is one of the most comfortable languages for building a scraper or crawler: static typing catches bugs at compile time, goroutines give you cheap concurrency out of the box, and the standard net/http library covers most of the networking work with zero third-party dependencies. This guide to golang web scraping walks the whole path — from fetching a single page to a distributed, concurrent crawler with proxies, Tor, and durable queues.

Everything here was tested on Go 1.22+. Third-party packages are installed with go get; the exact commands appear in each section.

Fetching a page
Libraries for parsing content
Handling non-UTF-8 encodings
Concurrency
Using proxies
Scraping through Tor
Working with HTTPS and SSL
Working with cookies
Response status and headers
Extras everyone forgets: politeness, robots.txt, User-Agent, JS rendering, retry
URL storage and queues
Pros and cons of building a scraper in Go
Conclusion

1. Fetching a page

The simplest version

package main

import (
    "fmt"
    "io"
    "net/http"
)

func main() {
    resp, err := http.Get("https://example.com")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close() // always close the body — otherwise you leak connections

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(body))
}

http.Get uses the global http.DefaultClient, which has no timeout. For a production scraper that is a non-starter: a single hung server will block a goroutine forever.

The right version: your own client with a timeout and headers

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "time"
)

func fetch(ctx context.Context, rawURL string) ([]byte, *http.Response, error) {
    client := &http.Client{
        Timeout: 15 * time.Second, // overall timeout for the whole request
    }

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, rawURL, nil)
    if err != nil {
        return nil, nil, err
    }

    // Many sites return 403 without a human-looking User-Agent.
    req.Header.Set("User-Agent",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "+
            "(KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36")
    req.Header.Set("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    req.Header.Set("Accept-Language", "en-US,en;q=0.9")

    resp, err := client.Do(req)
    if err != nil {
        return nil, nil, err
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    return body, resp, err
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    body, resp, err := fetch(ctx, "https://example.com")
    if err != nil {
        panic(err)
    }
    fmt.Println("Status:", resp.StatusCode, "| Size:", len(body))
}

The key points:

context.Context is your single cancellation mechanism. If the context expires or is canceled, the request is aborted. That saves you during graceful shutdown and whenever you cap the time budget for an entire crawl.
The client Timeout bounds everything: connection setup, sending, and reading the body. You can tune it more precisely through a custom http.Transport (see below).
defer resp.Body.Close() — an unclosed body holds a TCP connection open and prevents it from being reused from the pool. At scale that produces the classic "too many open files" error.

Fine-tuning the Transport

http.Transport is the "engine" under the client. A single transport instance is reused across requests and maintains a pool of keep-alive connections, so create it once for the whole application, not once per request.

transport := &http.Transport{
    MaxIdleConns:        100,              // total idle connections in the pool
    MaxIdleConnsPerHost: 10,               // per host
    IdleConnTimeout:     90 * time.Second, // how long an idle connection lives
    DisableCompression:  false,            // gzip is decompressed automatically
    ForceAttemptHTTP2:   true,
}
client := &http.Client{
    Transport: transport,
    Timeout:   15 * time.Second,
}

A common mistake is creating &http.Client{} (or a transport) inside the loop for every URL. That breaks the connection pool and exhausts local ports. Create the client once and pass it by reference.

2. Libraries for parsing content

Once the HTML is loaded, you have to parse it. There are three tiers of tooling — pick a golang scraping library based on whether you need to parse one page or crawl a whole site.

2.1. goquery — jQuery-style syntax (the most popular)

bash

go get github.com/PuerkitoBio/goquery

package main

import (
    "fmt"
    "net/http"

    "github.com/PuerkitoBio/goquery"
)

func main() {
    resp, _ := http.Get("https://news.ycombinator.com")
    defer resp.Body.Close()

    doc, err := goquery.NewDocumentFromReader(resp.Body)
    if err != nil {
        panic(err)
    }

    // CSS selectors, just like jQuery
    doc.Find(".titleline > a").Each(func(i int, s *goquery.Selection) {
        title := s.Text()
        href, _ := s.Attr("href")
        fmt.Printf("%d. %s — %s\n", i+1, title, href)
    })
}

goquery supports nearly all of CSS3: .class, #id, [attr=value], :first-child, :nth-of-type(n), and the >, +, ~ combinators. It is the best choice for most HTML-parsing jobs.

2.2. Colly — a full crawler framework

bash

go get github.com/gocolly/colly/v2

Colly takes care of fetching, parsing, following links, rate limiting, caching, and much more — it is not just a parser but a crawler engine.

package main

import (
    "fmt"
    "time"

    "github.com/gocolly/colly/v2"
)

func main() {
    c := colly.NewCollector(
        colly.AllowedDomains("example.com"),
        colly.MaxDepth(2),
        colly.Async(true), // asynchronous crawling
    )

    // Concurrency limit and delay — politeness built in
    c.Limit(&colly.LimitRule{
        DomainGlob:  "*",
        Parallelism: 4,
        Delay:       500 * time.Millisecond,
        RandomDelay: 500 * time.Millisecond,
    })

    // Callback for every heading found
    c.OnHTML("h1, h2", func(e *colly.HTMLElement) {
        fmt.Println("Heading:", e.Text)
    })

    // Follow every link
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Request.AbsoluteURL(e.Attr("href"))
        e.Request.Visit(link)
    })

    c.OnRequest(func(r *colly.Request) {
        fmt.Println("Fetching:", r.URL)
    })

    c.OnError(func(r *colly.Response, err error) {
        fmt.Println("Error:", r.Request.URL, err)
    })

    c.Visit("https://example.com")
    c.Wait() // wait for all async requests to finish
}

Reach for Colly when you actually need to walk a site (crawling) rather than parse a single page.

2.3. golang.org/x/net/html — the low-level tokenizer

The (semi-)standard package. It gives you maximum control and zero dependencies, but writing against it by hand is tedious — it is a streaming, token-by-token parser.

package main

import (
    "fmt"
    "strings"

    "golang.org/x/net/html"
)

func main() {
    r := strings.NewReader(`<html><body><a href="/x">Link</a></body></html>`)
    tokenizer := html.NewTokenizer(r)

    for {
        tt := tokenizer.Next()
        if tt == html.ErrorToken {
            break // end of document
        }
        if tt == html.StartTagToken {
            t := tokenizer.Token()
            if t.Data == "a" {
                for _, a := range t.Attr {
                    if a.Key == "href" {
                        fmt.Println("href:", a.Val)
                    }
                }
            }
        }
    }
}

Use it when speed on huge documents matters, or when goquery feels too heavy.

2.4. JSON and APIs instead of HTML

Often the data on a page is loaded by a separate AJAX request that returns JSON. This is the most convenient case — parsing JSON is more reliable than parsing HTML:

type Product struct {
    ID    int     `json:"id"`
    Name  string  `json:"name"`
    Price float64 `json:"price"`
}

var products []Product
resp, _ := http.Get("https://api.example.com/products")
defer resp.Body.Close()
json.NewDecoder(resp.Body).Decode(&products)

Before you parse any HTML, open the Network tab in DevTools — the JSON endpoint you need may already exist.

2.5. XPath

If you prefer XPath, there is github.com/antchfx/htmlquery:

doc, _ := htmlquery.LoadURL("https://example.com")
nodes := htmlquery.Find(doc, "//div[@class='item']/a/@href")

Comparison

Tool	When to use it	Dependencies
`goquery`	Parsing HTML with CSS selectors	1
`colly`	Crawling entire sites	a few
`x/net/html`	Maximum control / speed	semi-std
`encoding/json`	The API/AJAX returns JSON	std
`htmlquery`	You love XPath	1

3. Handling non-UTF-8 encodings

A classic headache: you fetch a page and instead of text you get Ã©Ã¨Ã« or â€œquotesâ€. The cause is that the site serves content in a legacy encoding — Windows-1252, ISO-8859-1 (Latin-1), Shift-JIS, GBK, and friends — rather than UTF-8, while Go treats every byte of a string as UTF-8 by default. This bites you on older enterprise sites, government portals, and localized pages the world over.

The universal fix: a charset detector

The golang.org/x/net/html/charset package figures out the encoding from the Content-Type header, from <meta charset>, and heuristically from the content, then hands you a reader that transcodes the stream to UTF-8 on the fly.

bash

go get golang.org/x/net/html
go get golang.org/x/text

package main

import (
    "fmt"
    "io"
    "net/http"

    "github.com/PuerkitoBio/goquery"
    "golang.org/x/net/html/charset"
)

func main() {
    resp, _ := http.Get("https://some-legacy-encoded-site.example")
    defer resp.Body.Close()

    // charset.NewReader detects the encoding and transcodes to UTF-8
    utf8Reader, err := charset.NewReader(resp.Body, resp.Header.Get("Content-Type"))
    if err != nil {
        panic(err)
    }

    doc, err := goquery.NewDocumentFromReader(utf8Reader)
    if err != nil {
        panic(err)
    }

    fmt.Println(doc.Find("title").Text()) // now the accented characters are correct
    _ = io.Discard
}

This works in about 95% of cases — make it your default.

Specifying the encoding explicitly

If you know the encoding for certain (say, the site is always Windows-1252), you can transcode by hand with golang.org/x/text/encoding:

import (
    "golang.org/x/text/encoding/charmap"
    "golang.org/x/text/transform"
)

// Windows-1252 → UTF-8
decoder := charmap.Windows1252.NewDecoder()
reader := transform.NewReader(resp.Body, decoder)

body, _ := io.ReadAll(reader)
fmt.Println(string(body))

For Latin-1 use charmap.ISO8859_1; for other Western code pages, the rest of the charmap family. For CJK encodings such as Shift-JIS or GBK, reach for the golang.org/x/text/encoding/japanese and .../simplifiedchinese sub-packages instead.

The reverse task — sending non-UTF-8 text

If you need to POST text in a legacy encoding (for example, to an old Latin-1 form):

encoder := charmap.Windows1252.NewEncoder()
encoded, _ := encoder.String("Café résumé")
// encoded is now Windows-1252 bytes — send it in the request body

If you see garbled characters only in the Windows console but the file output is fine, the problem is your terminal's code page, not the parser. Run chcp 65001 to switch cmd to UTF-8.

4. Concurrency

This is where Go shines. Goroutines are thousands of times cheaper than OS threads, and channels give you safe data exchange without explicit mutexes.

4.1. The naive (wrong) approach

// Bad: one goroutine per URL, with no limits
for _, url := range urls {
    go fetch(url) // 100,000 URLs → 100,000 concurrent requests → the server dies or bans you
}

Without a concurrency limit you will either flatten the target server, exhaust your file-descriptor limit, or get banned instantly.

4.2. Worker pool — the canonical pattern

Create a fixed number of workers that pull jobs from a channel. Concurrency is bounded by the number of workers.

package main

import (
    "fmt"
    "io"
    "net/http"
    "sync"
    "time"
)

type Result struct {
    URL    string
    Status int
    Size   int
    Err    error
}

func worker(id int, client *http.Client, jobs <-chan string, results chan<- Result, wg *sync.WaitGroup) {
    defer wg.Done()
    for url := range jobs {
        resp, err := client.Get(url)
        if err != nil {
            results <- Result{URL: url, Err: err}
            continue
        }
        body, _ := io.ReadAll(resp.Body)
        resp.Body.Close()
        results <- Result{URL: url, Status: resp.StatusCode, Size: len(body)}
    }
}

func main() {
    urls := []string{
        "https://example.com",
        "https://go.dev",
        "https://news.ycombinator.com",
        // ... thousands of URLs
    }

    const numWorkers = 8

    client := &http.Client{Timeout: 10 * time.Second}
    jobs := make(chan string, 100)
    results := make(chan Result, 100)

    var wg sync.WaitGroup
    for i := 0; i < numWorkers; i++ {
        wg.Add(1)
        go worker(i, client, jobs, results, &wg)
    }

    // Feed jobs in a separate goroutine
    go func() {
        for _, u := range urls {
            jobs <- u
        }
        close(jobs) // important: close the channel so workers can exit
    }()

    // Close results once all workers are done
    go func() {
        wg.Wait()
        close(results)
    }()

    // Read the results
    for r := range results {
        if r.Err != nil {
            fmt.Printf("[err] %s: %v\n", r.URL, r.Err)
        } else {
            fmt.Printf("[ok]  %s [%d] %d bytes\n", r.URL, r.Status, r.Size)
        }
    }
}

Breaking the pattern down:

jobs is the input channel. Close it after feeding all URLs — that is the signal for workers to leave their for range loop.
sync.WaitGroup lets you wait until every worker finishes.
results is closed by a separate goroutine after wg.Wait(), otherwise the main for range results would block forever.
Concurrency is controlled by a single constant, numWorkers.

4.3. Bounding with a semaphore (errgroup)

A more modern approach is golang.org/x/sync/errgroup with a limit. It conveniently collects the first error and supports context cancellation.

bash

go get golang.org/x/sync/errgroup

package main

import (
    "context"
    "fmt"
    "net/http"

    "golang.org/x/sync/errgroup"
)

func main() {
    urls := []string{"https://example.com", "https://go.dev" /* ... */}

    g, ctx := errgroup.WithContext(context.Background())
    g.SetLimit(8) // at most 8 concurrent goroutines

    client := &http.Client{}
    for _, u := range urls {
        u := u // needed on Go < 1.22: capture the loop variable
        g.Go(func() error {
            req, _ := http.NewRequestWithContext(ctx, "GET", u, nil)
            resp, err := client.Do(req)
            if err != nil {
                return err
            }
            resp.Body.Close()
            fmt.Println(u, resp.StatusCode)
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        fmt.Println("one of the tasks failed:", err)
    }
}

On Go 1.21 and earlier, the loop variable is reused, so the u := u line is mandatory — otherwise every goroutine gets the same (last) URL. Go 1.22+ fixed this at the language level, so with a current toolchain you can drop the line, but the habit does no harm.

4.4. Protecting shared data

If workers write to a shared map (for example, the set of visited URLs), it must be protected:

var (
    visited = make(map[string]bool)
    mu      sync.Mutex
)

func markVisited(url string) bool {
    mu.Lock()
    defer mu.Unlock()
    if visited[url] {
        return false // already seen
    }
    visited[url] = true
    return true
}

Alternatives are sync.Map (good for "many reads, few writes") or sync/atomic for counters. Run your tests with the -race flag — Go's race detector finds these bugs automatically.

5. Using proxies

Proxies let you spread load, get around geo-blocks, and lower the odds of an IP ban. If your crawl is large enough to need a rotating pool, see our guide to rotating proxies.

One proxy per client

package main

import (
    "net/http"
    "net/url"
)

func clientWithProxy(proxyAddr string) (*http.Client, error) {
    // user:pass@host:port is supported
    proxyURL, err := url.Parse(proxyAddr) // e.g. "http://user:pass@1.2.3.4:8080"
    if err != nil {
        return nil, err
    }

    transport := &http.Transport{
        Proxy: http.ProxyURL(proxyURL),
    }
    return &http.Client{Transport: transport}, nil
}

The http://, https://, and socks5:// schemes are all supported. For authentication, put the username and password directly in the URL.

Rotating a pool of proxies

To spread requests across a list of proxies, swap out the transport's Proxy function — it is called on every request:

package main

import (
    "math/rand"
    "net/http"
    "net/url"
    "sync/atomic"
)

type ProxyRotator struct {
    proxies []*url.URL
    counter uint64
}

func NewProxyRotator(addrs []string) *ProxyRotator {
    r := &ProxyRotator{}
    for _, a := range addrs {
        if u, err := url.Parse(a); err == nil {
            r.proxies = append(r.proxies, u)
        }
    }
    return r
}

// Round-robin proxy selection
func (r *ProxyRotator) Next(_ *http.Request) (*url.URL, error) {
    if len(r.proxies) == 0 {
        return nil, nil // no proxy
    }
    i := atomic.AddUint64(&r.counter, 1)
    return r.proxies[i%uint64(len(r.proxies))], nil
}

func main() {
    rotator := NewProxyRotator([]string{
        "http://user:pass@10.0.0.1:8080",
        "http://user:pass@10.0.0.2:8080",
        "socks5://10.0.0.3:1080",
    })

    transport := &http.Transport{
        Proxy: rotator.Next, // next proxy on each request
    }
    client := &http.Client{Transport: transport}

    _ = client
    _ = rand.Int
}

In practice, keep "health" data alongside each proxy: an error counter and the time of its last ban. A dead proxy is temporarily dropped from the rotation. Teams usually write a small wrapper that checks a proxy against a known-good endpoint before using it.

6. Scraping through Tor

Tor is a free anonymizing network that is available locally as a SOCKS5 proxy (by default at 127.0.0.1:9050). Scraping through Tor is useful for anonymity and automatic IP rotation, but it is slow, and many sites block Tor exit nodes.

Setup

Start a Tor daemon. The easiest way is Docker:

bash

docker run -d --name tor -p 9050:9050 -p 9051:9051 dperson/torproxy

Or install the system tor package (apt install tor, brew install tor) — it will bring up SOCKS5 on 9050 by itself.

An HTTP client over Tor (SOCKS5)

bash

go get golang.org/x/net/proxy

package main

import (
    "fmt"
    "io"
    "net/http"

    "golang.org/x/net/proxy"
)

func torClient() (*http.Client, error) {
    // Connect to the local Tor SOCKS5 endpoint
    dialer, err := proxy.SOCKS5("tcp", "127.0.0.1:9050", nil, proxy.Direct)
    if err != nil {
        return nil, err
    }

    transport := &http.Transport{
        Dial: dialer.Dial, // all traffic goes through Tor
    }
    return &http.Client{Transport: transport}, nil
}

func main() {
    client, err := torClient()
    if err != nil {
        panic(err)
    }

    // check.torproject.org confirms we are on Tor
    resp, err := client.Get("https://check.torproject.org/api/ip")
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    fmt.Println(string(body)) // {"IsTor":true,"IP":"..."}
}

Changing IP (a new circuit) via the control port

Tor can build a new circuit on command — that is, change your exit IP. You do it through the control port (9051) by sending the NEWNYM signal:

package main

import (
    "fmt"
    "net/textproto"
)

func newTorIdentity(controlPassword string) error {
    conn, err := textproto.Dial("tcp", "127.0.0.1:9051")
    if err != nil {
        return err
    }
    defer conn.Close()

    // Authenticate (the password must be configured in torrc)
    if _, _, err := conn.Cmd(`AUTHENTICATE "%s"`, controlPassword); err != nil {
        return err
    }
    conn.ReadResponse(250)

    // Signal a new circuit
    id, _ := conn.Cmd("SIGNAL NEWNYM")
    conn.StartResponse(id)
    defer conn.EndResponse(id)
    _, msg, err := conn.ReadResponse(250)
    fmt.Println("Tor replied:", msg)
    return err
}

For the control port to work, torrc must set ControlPort 9051 and a password hash (HashedControlPassword, generated with tor --hash-password YOUR_PASSWORD).

Tor gives you anonymity, not invisibility. It is slow, exit nodes are frequently blacklisted, and hammering a site through Tor is poor form toward a volunteer network. For bulk collection, commercial residential proxies are a better fit.

7. Working with HTTPS and SSL

Good news: for HTTPS you usually do not need to do anything — Go verifies certificates automatically using the system's trusted root store.

When you do need to intervene

1. Self-signed certificates (test/internal sites). Turning off verification solves the problem but opens a hole for MITM — for tests only:

import "crypto/tls"

transport := &http.Transport{
    TLSClientConfig: &tls.Config{
        InsecureSkipVerify: true, // NOT for production!
    },
}
client := &http.Client{Transport: transport}

2. The correct way — add a specific root certificate to the trust pool:

package main

import (
    "crypto/tls"
    "crypto/x509"
    "net/http"
    "os"
)

func clientWithCustomCA(caCertPath string) (*http.Client, error) {
    caCert, err := os.ReadFile(caCertPath)
    if err != nil {
        return nil, err
    }

    caPool := x509.NewCertPool()
    caPool.AppendCertsFromPEM(caCert)

    transport := &http.Transport{
        TLSClientConfig: &tls.Config{
            RootCAs:    caPool,
            MinVersion: tls.VersionTLS12, // no lower than TLS 1.2
        },
    }
    return &http.Client{Transport: transport}, nil
}

3. Controlling the TLS version and cipher suites. Sometimes a site requires a specific configuration, or you want to imitate a particular browser:

tlsConfig := &tls.Config{
    MinVersion: tls.VersionTLS12,
    MaxVersion: tls.VersionTLS13,
}

TLS fingerprinting. Advanced anti-bot systems (Cloudflare, Akamai) identify bots by the "fingerprint" of the TLS handshake (JA3/JA4) — the stock Go client has a recognizable one. To disguise yourself as a real browser, teams use github.com/refraction-networking/utls, which can forge the ClientHello to look like Chrome or Firefox. When a site still throws a challenge, that is usually the point where you also need solving CAPTCHAs. This is advanced anti-bot evasion territory.

8. Working with cookies

Cookies are needed for sessions, authentication, and passing checks. Go can manage them automatically through cookiejar.

Storing cookies automatically

package main

import (
    "fmt"
    "net/http"
    "net/http/cookiejar"

    "golang.org/x/net/publicsuffix"
)

func main() {
    // The public-suffix list is needed to handle domains correctly
    jar, err := cookiejar.New(&cookiejar.Options{
        PublicSuffixList: publicsuffix.List,
    })
    if err != nil {
        panic(err)
    }

    client := &http.Client{Jar: jar}

    // First request: the server sets a cookie (e.g. a session)
    client.Get("https://example.com/login")

    // The second request automatically sends the stored cookies
    resp, _ := client.Get("https://example.com/dashboard")
    defer resp.Body.Close()

    // Inspect what is in the jar for a specific host
    u, _ := resp.Request.URL.Parse("https://example.com")
    for _, c := range jar.Cookies(u) {
        fmt.Printf("%s = %s\n", c.Name, c.Value)
    }
}

With a Jar, the client behaves like a browser: it accepts Set-Cookie from responses and replays them on subsequent requests to the same domain.

Setting cookies manually

When you already have a session token (copied from the browser) and just need to inject the session:

req, _ := http.NewRequest("GET", "https://example.com/account", nil)
req.AddCookie(&http.Cookie{Name: "session_id", Value: "abc123xyz"})
req.AddCookie(&http.Cookie{Name: "csrf_token", Value: "tok456"})

resp, _ := client.Do(req)

A typical login flow (POST authentication)

import (
    "net/url"
    "strings"
)

form := url.Values{}
form.Set("username", "user")
form.Set("password", "pass")

req, _ := http.NewRequest("POST", "https://example.com/login",
    strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

// If the client has a Jar, the session cookie is saved automatically,
// and every subsequent request is authenticated.
resp, _ := client.Do(req)

Persisting the cookie jar between program runs is done by hand: iterate over jar.Cookies(url), serialize to a JSON file, and reload it at startup via jar.SetCookies(url, cookies). The standard cookiejar is not persistent, but ready-made wrappers exist (github.com/juju/persistent-cookiejar).

9. Response status and headers

After a request runs, the *http.Response object holds all the metadata.

resp, err := client.Do(req)
if err != nil {
    // A network error (DNS, timeout, connection refused).
    // IMPORTANT: when err != nil, resp == nil — do not touch resp.Body!
    return err
}
defer resp.Body.Close()

// Status
fmt.Println(resp.StatusCode) // 200, 404, 503 ...
fmt.Println(resp.Status)     // "200 OK", "404 Not Found"

// Individual headers (case-insensitive)
fmt.Println(resp.Header.Get("Content-Type"))   // text/html; charset=utf-8
fmt.Println(resp.Header.Get("Content-Length"))
fmt.Println(resp.Header.Get("Server"))
fmt.Println(resp.Header.Get("Set-Cookie"))

// A single header can have several values
for _, v := range resp.Header.Values("Set-Cookie") {
    fmt.Println("cookie:", v)
}

// All headers at once
for name, values := range resp.Header {
    fmt.Printf("%s: %v\n", name, values)
}

Handling statuses well

switch {
case resp.StatusCode == http.StatusOK: // 200
    // parse the body

case resp.StatusCode == http.StatusTooManyRequests: // 429
    // we're being throttled — read the Retry-After header and wait
    retryAfter := resp.Header.Get("Retry-After")
    fmt.Println("Rate limited, waiting:", retryAfter)

case resp.StatusCode >= 500: // 5xx — server error
    // worth retrying later

case resp.StatusCode == http.StatusNotFound: // 404
    // the page doesn't exist — stop trying

case resp.StatusCode >= 300 && resp.StatusCode < 400: // 3xx
    // redirect; by default the Go client follows up to 10 redirects
}

Controlling redirects

By default the client follows redirects. To disable or intercept them:

client := &http.Client{
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
        // Forbid redirects — the 3xx response is returned "as is"
        return http.ErrUseLastResponse
    },
}

Remember the distinction: a network error (err != nil) means the request never arrived or never came back (DNS, timeout). An HTTP error (resp.StatusCode == 404/500) is a valid response from the server, and in that case err == nil. Always check the status code separately from err.

10. Extras everyone forgets

These sections were not in the original checklist, but without them a production scraper falls apart.

10.1. Politeness and rate limiting

Do not bombard a server with requests — it is both a load on someone else's infrastructure and a fast ban. Throttle the frequency with golang.org/x/time/rate:

import "golang.org/x/time/rate"

// 2 requests per second, burst up to 5
limiter := rate.NewLimiter(rate.Limit(2), 5)

func politeGet(ctx context.Context, client *http.Client, url string) (*http.Response, error) {
    if err := limiter.Wait(ctx); err != nil { // blocks until it's allowed
        return nil, err
    }
    return client.Get(url)
}

10.2. robots.txt

Good manners (and sometimes a legal necessity) mean respecting robots.txt, where a site states what may be crawled. The github.com/temoto/robotstxt package helps parse it:

import "github.com/temoto/robotstxt"

resp, _ := http.Get("https://example.com/robots.txt")
data, _ := io.ReadAll(resp.Body)
robots, _ := robotstxt.FromBytes(data)

if robots.TestAgent("/private/page", "MyBot") {
    // allowed — scrape it
} else {
    // disallowed by robots.txt
}

10.3. Rotating the User-Agent

The same User-Agent on thousands of requests is a dead giveaway for a bot. Keep a list and pick one at random:

var userAgents = []string{
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) ... Chrome/133.0 ...",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ... Safari/605.1 ...",
    "Mozilla/5.0 (X11; Linux x86_64) ... Firefox/134.0",
}

req.Header.Set("User-Agent", userAgents[rand.Intn(len(userAgents))])

10.4. Retry with exponential backoff

Networks are unreliable — transient failures (5xx, timeouts) should be retried with a growing pause:

func fetchWithRetry(ctx context.Context, client *http.Client, url string, maxRetries int) (*http.Response, error) {
    var lastErr error
    for attempt := 0; attempt <= maxRetries; attempt++ {
        if attempt > 0 {
            // 1s, 2s, 4s, 8s... + a little randomness (jitter)
            backoff := time.Duration(1<<uint(attempt-1)) * time.Second
            jitter := time.Duration(rand.Intn(500)) * time.Millisecond
            select {
            case <-time.After(backoff + jitter):
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }

        req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
        resp, err := client.Do(req)
        if err != nil {
            lastErr = err
            continue // retry on a network error
        }
        if resp.StatusCode >= 500 || resp.StatusCode == 429 {
            resp.Body.Close()
            lastErr = fmt.Errorf("status %d", resp.StatusCode)
            continue // retry on 5xx/429
        }
        return resp, nil // success
    }
    return nil, fmt.Errorf("retries exhausted: %w", lastErr)
}

10.5. Scraping JavaScript-rendered pages

If the content is drawn by JavaScript (a React/Vue SPA), http.Get returns an almost-empty HTML shell. Then you need a headless browser driving a real Chrome over the CDP protocol:

github.com/chromedp/chromedp — controlling Chrome from Go;
github.com/go-rod/rod — a higher-level alternative.

import "github.com/chromedp/chromedp"

ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()

var html string
chromedp.Run(ctx,
    chromedp.Navigate("https://spa-example.com"),
    chromedp.WaitVisible(".content"),      // wait for JS to render
    chromedp.OuterHTML("html", &html),     // grab the finished DOM
)
// then feed html into goquery

The downside: a headless browser is heavy (memory, CPU) and slow. Before you pull in Chrome, check whether the data is available through a JSON API (see section 2.4) — often the browser is unnecessary.

10.6. Legal and ethical considerations

Scraping is a gray area. The basic guardrails: respect a site's robots.txt and Terms of Service; do not collect personal data without a lawful basis (GDPR in the EU/UK, CCPA in California); do not create excessive load; and do not pass off someone else's content as your own. What is technically possible and what is legally permissible are not the same thing. If you want a deeper primer, see our overview of the legal side of web scraping.

11. URL storage and queues

A crawler needs two structures: a queue of not-yet-visited URLs (the frontier) and a set of already-visited ones (so you do not go in circles).

11.1. In-memory (the simplest option)

For small jobs, a channel as the queue and a map as the visited set are enough:

type Crawler struct {
    queue   chan string
    visited map[string]bool
    mu      sync.Mutex
}

func (c *Crawler) enqueue(url string) {
    c.mu.Lock()
    defer c.mu.Unlock()
    if c.visited[url] {
        return // already seen — skip
    }
    c.visited[url] = true
    select {
    case c.queue <- url:
    default: // queue is full — drop it or stash it separately
    }
}

The problems with in-memory: everything is lost on restart, and on millions of URLs the map eats all your RAM.

11.2. Deduplicating at scale: the Bloom filter

Storing tens of millions of strings in a map is expensive. A Bloom filter is a probabilistic structure that uses little memory and answers quickly with "definitely not seen" or "possibly seen" (with a small false-positive rate):

import "github.com/bits-and-blooms/bloom/v3"

// ~10M elements, 1% error rate
filter := bloom.NewWithEstimates(10_000_000, 0.01)

if filter.TestString(url) {
    // possibly already seen — skip (with a small risk of rarely missing a new one)
} else {
    filter.AddString(url)
    // definitely new — enqueue
}

11.3. External queues (production, distributed)

When the crawler must survive restarts and run across several machines, the queue is moved into external storage:

Storage	Role	Notes
Redis	queue (`LPUSH`/`BRPOP`) + visited set (`SET`/`SADD`)	fast, atomic, ideal for distributed workers
RabbitMQ / Kafka	task queue	reliable delivery, ack/nack, redelivery of failures
PostgreSQL / SQLite	persistent frontier	convenient for storing URL + metadata + status
BadgerDB / bbolt	embedded KV store	no separate server, everything in one binary

An example Redis-backed queue:

import "github.com/redis/go-redis/v9"

rdb := redis.NewClient(&redis.Options{Addr: "localhost:6379"})

// Add a URL to the queue only if it hasn't been visited (atomic via SET)
func enqueue(ctx context.Context, url string) error {
    // SADD returns 1 if the element is new
    added, err := rdb.SAdd(ctx, "visited", url).Result()
    if err != nil {
        return err
    }
    if added == 1 {
        return rdb.LPush(ctx, "frontier", url).Err()
    }
    return nil // duplicate
}

// Take a URL off the queue (blocking)
func dequeue(ctx context.Context) (string, error) {
    res, err := rdb.BRPop(ctx, 5*time.Second, "frontier").Result()
    if err != nil {
        return "", err
    }
    return res[1], nil // res[0] is the key name, res[1] is the value
}

This setup lets you run dozens of workers on different machines: they all pull jobs from the shared Redis queue and write results to a shared database without duplicating work.

11.4. Priorities and crawl strategy

BFS (a plain FIFO queue) crawls the site "wide" and is usually preferable for crawling.
DFS (a LIFO stack) dives deep along a single branch.
A priority queue (container/heap or a Redis sorted set) crawls the important pages first (by "depth" or by expected value).

12. Pros and cons of building a scraper in Go

Pros

Concurrency out of the box. Goroutines and channels make a concurrent crawler natural and cheap. Thousands of parallel requests — without the pain of thread pools.
Performance. It compiles to native code that is fast and memory-thrifty. A go web scraper outruns the Python equivalent several times over in throughput.
A single binary. go build produces a self-contained executable with no dependencies or interpreter — deployment is trivial, and it is great for Docker and cron.
A strong standard library. net/http, crypto/tls, cookiejar, context, and encoding/json cover almost everything without third-party packages.
Static typing. Many errors are caught at compile time rather than at runtime in the middle of a multi-hour crawl.
A built-in race detector (-race) — indispensable in concurrent code.
A mature ecosystem: Colly, goquery, and chromedp are time-tested tools.

Cons

Verbose HTML parsing. Compared with Python (requests + BeautifulSoup in five lines), Go needs more code and explicit error handling.
Weaker for headless browsers. Python with Playwright/Selenium is richer; chromedp/rod are good, but the ecosystem is more modest.
Anti-detect is harder. Forging a TLS fingerprint (utls) and emulating a browser take more effort than the ready-made solutions in other stacks.
Manual error handling. The constant if err != nil gets tiring, though it does keep you disciplined.
Fewer ready-made ML/NLP tools for post-processing the extracted text — Python still leads there. A common pattern: collect data in Go, do analytics in Python.
A learning curve for concurrency. Channels, deadlocks, and races are powerful but demand understanding; a newcomer can easily write a goroutine leak.

When to choose Go — and when not to

Scenario	Recommendation
High-load crawler, millions of pages	Go — ideal
Long-running scraping service	Go
A one-off "grab this table" script	Python is faster to write
Heavy JS rendering, complex anti-detect	Often easier in Python + Playwright
Scraping plus immediate ML analysis	Python is closer to the data

13. Conclusion

The minimal shape of a production web scraping with golang setup looks like this:

One reusable http.Client with a timeout, a tuned Transport, and (optionally) a cookiejar.
A worker pool or errgroup with a limit — for controlled concurrency.
goquery (or Colly for crawling) — for data extraction, with charset.NewReader for correct handling of legacy encodings.
Proxies/Tor + User-Agent rotation — when you need anonymity or to get around bans.
Rate limiting and respect for robots.txt — so you neither flatten the server nor earn a ban.
Retry with backoff — for resilience against transient failures.
An external queue (Redis) and deduplication (a Bloom filter) — when the job outgrows RAM.

Go gives you an excellent balance of performance, reliability, and deployment simplicity. The single best piece of advice: always start by checking whether the site has an open JSON API. If it does, half the problems in this article (encodings, JS rendering, brittle selectors) disappear on their own.

If you would rather not build and babysit all of this yourself, scraping.pro runs it as a done-for-you service — from a one-off custom data extraction job to a fully managed data-as-a-service feed with monitoring and delivery into your systems. You bring the target and the schema; we handle the proxies, the anti-bot layer, and the uptime.

Custom Solutions

Price & Assortment Monitoring

Company & B2B Databases, Chain Monitoring

Content, Search, Ads & Review Monitoring

Media & News Monitoring

Domain Monitoring

Classifieds & Jobs

Golang Web Scraping: Complete Guide from Simple to Advanced

Contents

1. Fetching a page

The simplest version

The right version: your own client with a timeout and headers

Fine-tuning the Transport

2. Libraries for parsing content

2.1. goquery — jQuery-style syntax (the most popular)

2.2. Colly — a full crawler framework

2.3. golang.org/x/net/html — the low-level tokenizer

2.4. JSON and APIs instead of HTML

2.5. XPath

Comparison

3. Handling non-UTF-8 encodings

The universal fix: a charset detector

Specifying the encoding explicitly

The reverse task — sending non-UTF-8 text

4. Concurrency

4.1. The naive (wrong) approach

4.2. Worker pool — the canonical pattern

4.3. Bounding with a semaphore (errgroup)

4.4. Protecting shared data

5. Using proxies

One proxy per client

Rotating a pool of proxies

6. Scraping through Tor

Setup

An HTTP client over Tor (SOCKS5)

Changing IP (a new circuit) via the control port

7. Working with HTTPS and SSL

When you do need to intervene

8. Working with cookies

Storing cookies automatically

Setting cookies manually

A typical login flow (POST authentication)

9. Response status and headers

Handling statuses well

Controlling redirects

10. Extras everyone forgets

10.1. Politeness and rate limiting

10.2. robots.txt

10.3. Rotating the User-Agent

10.4. Retry with exponential backoff

10.5. Scraping JavaScript-rendered pages

10.6. Legal and ethical considerations

11. URL storage and queues

11.1. In-memory (the simplest option)

11.2. Deduplicating at scale: the Bloom filter

11.3. External queues (production, distributed)

11.4. Priorities and crawl strategy

12. Pros and cons of building a scraper in Go

Pros

Cons

When to choose Go — and when not to

13. Conclusion

Read also