Ruby Web Scraping: A Complete Guide from Simple to Advanced

This is a practical, end-to-end guide to web scraping with Ruby — from downloading a single page to multithreaded collection through rotating proxies and TOR. Every example is working code, written for Ruby 3.x.

Contents

1. Introduction: what scraping is and what to think about first
2. Fetching the page
3. Libraries for parsing content
4. Handling text encoding problems
5. Response status and headers
6. Working with HTTPS / SSL
7. Working with cookies
8. Using proxies
9. Scraping through TOR
10. Multithreading and concurrency
11. JavaScript pages (headless browsers)
12. Anti-bot defenses, User-Agent, delays, retries
13. Storing URLs and working with queues
14. Ready-made scraping frameworks
15. Pros and cons of Ruby for scraping
16. Conclusion and checklist

1. Introduction {#introduction}

Web scraping is the automated collection of data from web pages. The process almost always breaks down into two distinct tasks that are important not to conflate:

1. Fetching — getting the raw server response (the HTTP client's job).
2. Extraction — pulling the data you need out of that response (the HTML/JSON parser's job).

Ruby has dedicated tools for each task, and a good Ruby scraper usually combines one HTTP client with one parser.

What to think about before you write any code

Before you scrape, it pays to consider a few things — it will save you time and headaches:

• robots.txt. The file at https://example.com/robots.txt describes what the site owner allows bots to crawl. It is not always legally binding, but ignoring it is bad form and a risk.
• Terms of Service (ToS). Some sites explicitly prohibit automated collection. That is a legal question, not a technical one. In the US, the hiQ v. LinkedIn line of cases and the Computer Fraud and Abuse Act (CFAA) are the usual reference points for scraping public data.
• Server load. A scraper can easily turn into a DoS attack. Add delays and don't hammer a server with hundreds of concurrent connections unless you truly need to.
• Is there an API? Often it is simpler and more defensible to use an official API or an internal JSON endpoint than to parse HTML.
• Personal data. Collecting and storing personal data is regulated by law (GDPR in the EU/UK, CCPA in California, and similar frameworks elsewhere).

A quick robots.txt check with the webrobots gem:

require 'open-uri'
require 'webrobots'   # gem install webrobots

robots = WebRobots.new('MyScraperBot/1.0')
url = 'https://example.com/some/page'

if robots.allowed?(url)
  puts "Allowed to scrape"
else
  puts "robots.txt disallows this path"
end

If any of this feels like more than you want to build and maintain in-house, note up front that scraping.pro runs this as a done-for-you data extraction service — but this guide will show you how to do it yourself in Ruby.

2. Fetching the page {#fetching-the-page}

This is the foundation. Let's look at HTTP clients from the simplest to the most flexible.

2.1. open-uri — the quickest way

open-uri is part of the standard library. It's ideal for "grab this page in one line."

require 'open-uri'

html = URI.open('https://example.com').read
puts html

With headers and timeouts:

require 'open-uri'

html = URI.open(
  'https://example.com',
  'User-Agent' => 'Mozilla/5.0 (compatible; MyBot/1.0)',
  open_timeout: 5,
  read_timeout: 10
).read

Pros: nothing to install, minimal code. Cons: awkward for POST, response headers, redirects, and error handling (it raises OpenURI::HTTPError on 404/500).

2.2. Net::HTTP — standard library, full control

Net::HTTP is also built into Ruby. It's verbose, but it gives you access to everything.

require 'net/http'
require 'uri'

uri = URI('https://example.com/search?q=ruby')

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = (uri.scheme == 'https')
http.open_timeout = 5
http.read_timeout = 10

request = Net::HTTP::Get.new(uri)
request['User-Agent'] = 'MyBot/1.0'

response = http.request(request)

puts response.code        # "200"
puts response.body        # response body
puts response['Content-Type']

A POST request:

uri = URI('https://example.com/login')
res = Net::HTTP.post_form(uri, 'user' => 'admin', 'pass' => 'secret')
puts res.body

Pros: no dependencies, full control over request/response. Cons: verbose, manual redirect handling, not the most pleasant API.

2.3. http.rb (the http gem) — modern and ergonomic

The http.rb gem offers a clean, chainable API. It's one of the best "default" choices.

require 'http'   # gem install http

response = HTTP
  .headers('User-Agent' => 'MyBot/1.0')
  .timeout(connect: 5, read: 10)
  .follow                       # follow redirects automatically
  .get('https://example.com')

puts response.status            # 200
puts response.to_s              # body
puts response.headers['Content-Type']

POST with JSON:

response = HTTP.post(
  'https://api.example.com/items',
  json: { name: 'Widget', qty: 3 }
)
data = response.parse           # parses the JSON automatically

2.4. Faraday — a client with middleware

Faraday is a "wrapper over wrappers." Its key feature is the middleware stack: you can layer in logging, retries, JSON parsing, and error handling as composable middleware.

require 'faraday'         # gem install faraday
require 'faraday/retry'   # gem install faraday-retry

conn = Faraday.new(url: 'https://example.com') do |f|
  f.request :retry, max: 3, interval: 0.5   # automatic retries
  f.response :raise_error                    # 4xx/5xx -> exception
  f.options.timeout = 10
  f.headers['User-Agent'] = 'MyBot/1.0'
end

response = conn.get('/data')
puts response.status
puts response.body

Faraday shines when your scraper grows into a full application: one place to configure every request.

2.5. Typhoeus — when you need speed and parallelism

Typhoeus wraps libcurl. Its main advantage is Hydra, which issues many requests in parallel (see the concurrency section).

require 'typhoeus'   # gem install typhoeus

response = Typhoeus.get(
  'https://example.com',
  headers: { 'User-Agent' => 'MyBot/1.0' },
  timeout: 10,
  followlocation: true
)

puts response.code
puts response.body
puts response.headers['Content-Type']

Which one to choose

3. Libraries for parsing content {#parsing-content}

You've got the HTML — now extract the data.

3.1. Nokogiri — the de facto standard

Nokogiri parses HTML and XML and supports both CSS selectors and XPath. It's the main tool for 95% of jobs, and Nokogiri web scraping is practically synonymous with scraping in Ruby.

require 'nokogiri'
require 'open-uri'

html = URI.open('https://example.com').read
doc = Nokogiri::HTML(html)

# CSS selectors
title = doc.css('h1.title').text.strip
links = doc.css('a').map { |a| a['href'] }

# one element vs. all
first = doc.at_css('div.price')        # first match (or nil)
all   = doc.css('div.item')            # NodeSet of all matches

# XPath (more powerful for complex conditions)
prices = doc.xpath('//div[@class="price"]/text()').map(&:to_s)

# Attributes and nesting
doc.css('article.post').each do |post|
  title = post.at_css('h2')&.text&.strip
  date  = post.at_css('time')&.[]('datetime')
  body  = post.at_css('.content')&.text&.strip
  puts "#{date} — #{title}"
end

CSS vs. XPath — when to use which:

• CSS — shorter and more readable for simple selections: div.item > a.link.
• XPath — more powerful: search by text, by parent, by position:

# a link whose text contains "Download"
doc.xpath('//a[contains(text(), "Download")]')
# an element whose ancestor is a div with id="main"
doc.xpath('//div[@id="main"]//span[@class="price"]')
# select by index
doc.xpath('(//tr)[3]')

3.2. Parsing JSON — don't forget it

Very often the data isn't in the HTML at all but in JSON — an internal API the page loads over AJAX. Opening the Network tab in your browser and finding the JSON endpoint is frequently easier than parsing HTML.

require 'json'
require 'http'

raw = HTTP.get('https://api.example.com/products?page=1').to_s
data = JSON.parse(raw, symbolize_names: true)

data[:products].each do |p|
  puts "#{p[:name]}: #{p[:price]}"
end

3.3. Other parsers

• Oga — a pure-Ruby alternative to Nokogiri (no C extensions). Slower, but easier to install. Useful where Nokogiri is hard to build.
• Loofah (built on Nokogiri) — for cleaning/sanitizing HTML.
• Regular expressions — avoid parsing HTML with regexes. They break on any markup change. A regex is only appropriate for extracting small bits from text you've already selected (a phone number, a price out of a string, and so on).

# OK: pull a number out of already-selected text
price_text = doc.at_css('.price').text     # "$1,299"
price = price_text.gsub(/[^\d]/, '').to_i   # 1299

4. Text encoding problems {#encoding}

The single most common headache in real-world scraping is garbled text — "mojibake" instead of readable characters. The cause is that an HTTP response is just bytes, and Ruby has to interpret their encoding correctly (UTF-8, Windows-1252, ISO-8859-1, Shift_JIS, Windows-1251, and so on). This bites hardest on older sites and non-English pages, where legacy encodings are still common.

4.1. Where the problem comes from

Ruby stores an encoding with every string. If the bytes are Windows-1252 but Ruby thinks they're UTF-8, you get garbage.

str = response.body
puts str.encoding          # e.g. ASCII-8BIT or UTF-8
puts str.valid_encoding?   # false -> something is off

4.2. Detecting the encoding and converting

You can learn the encoding from:

1. the Content-Type: text/html; charset=windows-1252 header;
2. a <meta charset="..."> tag inside the HTML;
3. heuristics (the rchardet / charlock_holmes libraries).

Manual conversion (when you know the source encoding):

# from Windows-1252 to UTF-8
utf8 = body.force_encoding('Windows-1252').encode('UTF-8')

force_encoding only changes the encoding "label" without re-encoding the bytes, whereas encode actually transcodes the bytes to the target encoding. Order matters: first tell Ruby the truth about the source bytes, then transcode.

Safe conversion that replaces broken characters:

utf8 = body.encode(
  'UTF-8',
  'Windows-1252',
  invalid: :replace,
  undef:   :replace,
  replace: '?'
)

4.3. Nokogiri and encodings — the right way

The best approach is to pass the encoding straight to Nokogiri — it will handle the conversion:

require 'nokogiri'

# if you know the encoding
doc = Nokogiri::HTML(body, nil, 'Windows-1252')

# Nokogiri can read <meta charset> on its own if you don't get in the way:
doc = Nokogiri::HTML(body)   # often enough
puts doc.css('h1').text       # already in UTF-8

4.4. Automatic encoding detection

When a site doesn't declare its charset honestly, charlock_holmes (built on ICU) helps:

require 'charlock_holmes'   # gem install charlock_holmes

detection = CharlockHolmes::EncodingDetector.detect(body)
puts detection[:encoding]    # => "windows-1252"
puts detection[:confidence]  # => 90

utf8 = body.encode('UTF-8', detection[:encoding],
                   invalid: :replace, undef: :replace)

4.5. A universal helper

def to_utf8(body, content_type = nil)
  # 1. try the header
  if content_type && content_type =~ /charset=([\w-]+)/i
    enc = $1
    return body.encode('UTF-8', enc, invalid: :replace, undef: :replace)
  end

  # 2. already valid UTF-8?
  test = body.dup.force_encoding('UTF-8')
  return test if test.valid_encoding?

  # 3. auto-detect
  require 'charlock_holmes'
  det = CharlockHolmes::EncodingDetector.detect(body)
  body.encode('UTF-8', det[:encoding] || 'UTF-8',
              invalid: :replace, undef: :replace)
end

Rule of thumb: keep your entire internal pipeline in UTF-8. Convert at the boundary, right after fetching, and never think about it again.

5. Response status and headers {#status-and-headers}

Before you parse the body, you need to confirm the response is even valid. Ignoring the HTTP status is a classic beginner mistake — you end up parsing an error page and treating it as data.

5.1. Reading the status and headers

require 'http'

resp = HTTP.get('https://example.com')

puts resp.status              # 200 (status object)
puts resp.status.to_i         # 200 (integer)
puts resp.status.success?     # true for 2xx
puts resp.status.redirect?    # true for 3xx

# headers
puts resp.headers['Content-Type']
puts resp.headers['Content-Length']
puts resp.headers['Server']
puts resp.content_type.mime_type   # "text/html"

In Net::HTTP:

res = Net::HTTP.get_response(URI('https://example.com'))
puts res.code            # "200"
puts res.message         # "OK"
puts res['Set-Cookie']
res.each_header { |k, v| puts "#{k}: #{v}" }

5.2. What to do with different statuses

case resp.code
when 200      then process(resp.to_s)
when 301, 302 then follow_redirect(resp.headers['Location'])
when 404      then log("page not found")
when 403, 429 then back_off    # blocked / rate-limited — slow down
when 500..599 then retry_later # server error — retry later
end

The ones that matter most:

• 429 Too Many Requests — you're going too fast. Check the Retry-After header.
• 403 Forbidden — often anti-bot. Rotate your User-Agent / proxy.
• 3xx + Location — a redirect; decide whether to follow it.

5.3. Useful headers

• Content-Type — content type and encoding.
• Set-Cookie — cookies (see section 7).
• Location — where a redirect points.
• Retry-After — how long to wait before retrying.
• ETag / Last-Modified — for caching and conditional requests (If-None-Match / If-Modified-Since → 304 Not Modified, saving bandwidth).

6. HTTPS / SSL {#https-ssl}

Most clients work with HTTPS out of the box and verify certificates — which is correct and secure.

# http.rb, Faraday, Typhoeus, open-uri — HTTPS works automatically
HTTP.get('https://example.com')

In Net::HTTP you have to enable SSL explicitly:

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_PEER   # default: verify the cert

Disabling certificate verification — be careful!

Sometimes a site has a self-signed or broken certificate. Disabling verification opens the door to a man-in-the-middle attack, so do it only deliberately:

# Net::HTTP
http.verify_mode = OpenSSL::SSL::VERIFY_NONE   # NOT for production

# http.rb
ctx = OpenSSL::SSL::SSLContext.new
ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE
HTTP.get('https://self-signed.example.com', ssl_context: ctx)

# Typhoeus
Typhoeus.get('https://example.com', ssl_verifypeer: false)

The "certificate verify failed" problem

A common error on a fresh Ruby/Windows install — there's no up-to-date set of root certificates. Fixes:

• update the certifi gem / the system ca-certificates;
• point to a bundle: http.ca_file = '/path/to/cacert.pem';
• on macOS/Linux, updating OpenSSL is usually enough.

You can also pin a specific TLS version if the server is finicky:

http.min_version = OpenSSL::SSL::TLS1_2_VERSION

7. Cookies {#cookies}

Cookies are needed for sessions: login, shopping cart, "human" behavior. The server sends them in Set-Cookie, and the client has to return them in Cookie on subsequent requests.

7.1. Manually

require 'http'

# got the cookies
resp = HTTP.get('https://example.com/login')
cookies = resp.cookies   # HTTP::CookieJar

# send them on the next request
resp2 = HTTP.cookies(cookies).get('https://example.com/account')

7.2. Persisting a session across requests (http.rb)

require 'http'
require 'http-cookie'

jar = HTTP::CookieJar.new

# log in
login = HTTP.post('https://example.com/login',
                  form: { user: 'me', pass: 'secret' })
login.cookies.each { |c| jar.add(c) }

# use the session
page = HTTP.cookies(jar).get('https://example.com/dashboard')

7.3. Cookies in Net::HTTP

res = Net::HTTP.get_response(URI('https://example.com'))
cookie = res['Set-Cookie']

req = Net::HTTP::Get.new(URI('https://example.com/next'))
req['Cookie'] = cookie

7.4. Mechanize — cookies "for free"

For complex sessions, the easiest option is Mechanize (section 14): it automatically maintains a cookie jar across requests, just like a browser.

require 'mechanize'

agent = Mechanize.new
agent.get('https://example.com/login') do |page|
  form = page.forms.first
  form.field_with(name: 'user').value = 'me'
  form.field_with(name: 'pass').value = 'secret'
  form.submit
end
# cookies are already stored; the agent remembers the session
dashboard = agent.get('https://example.com/dashboard')

7.5. Saving cookies to disk

So you don't have to log in on every run:

agent.cookie_jar.save('cookies.yml')   # save
agent.cookie_jar.load('cookies.yml')   # restore

8. Proxies {#proxies}

Proxies let you bypass IP-based blocks, spread load, hide the source, and scrape from a specific geo-location. For large-scale scraping you typically use a pool of rotating proxies.

8.1. Proxies across the different clients

# open-uri
URI.open('https://example.com',
         proxy: 'http://user:pass@1.2.3.4:8080').read

# Net::HTTP
proxy = Net::HTTP::Proxy('1.2.3.4', 8080, 'user', 'pass')
proxy.start('example.com', 443, use_ssl: true) do |http|
  http.get('/')
end

# http.rb
HTTP.via('1.2.3.4', 8080, 'user', 'pass').get('https://example.com')

# Faraday
Faraday.new('https://example.com',
            proxy: 'http://user:pass@1.2.3.4:8080').get

# Typhoeus
Typhoeus.get('https://example.com',
             proxy: 'http://1.2.3.4:8080',
             proxyuserpwd: 'user:pass')

8.2. Rotating proxies

class ProxyPool
  def initialize(proxies)
    @proxies = proxies
    @index = 0
    @mutex = Mutex.new
  end

  def next_proxy
    @mutex.synchronize do
      proxy = @proxies[@index]
      @index = (@index + 1) % @proxies.size
      proxy
    end
  end
end

pool = ProxyPool.new([
  'http://1.1.1.1:8080',
  'http://2.2.2.2:8080',
  'http://3.3.3.3:8080'
])

10.times do
  host, port = pool.next_proxy.sub('http://', '').split(':')
  resp = HTTP.via(host, port.to_i).get('https://example.com')
  puts resp.status
end

8.3. Types of proxies

• HTTP/HTTPS — ordinary web proxies.
• SOCKS5 — low-level, proxy any traffic (needed for TOR, see below).
• Datacenter vs. Residential — datacenter proxies are cheaper but easier to ban; residential proxies (routed through real ISPs) cost more but "look like real users."

8.4. Handling dead proxies

Proxies drop out constantly. Wrap the request in a retry that rotates the proxy:

def fetch_with_proxy(url, pool, attempts: 3)
  attempts.times do
    proxy = pool.next_proxy
    begin
      host, port = proxy.sub(%r{^https?://}, '').split(':')
      resp = HTTP.timeout(connect: 5, read: 10)
                 .via(host, port.to_i)
                 .get(url)
      return resp if resp.status.success?
    rescue HTTP::Error, Errno::ECONNREFUSED, IO::TimeoutError => e
      warn "Proxy #{proxy} failed: #{e.message}"
      next
    end
  end
  nil
end

9. Scraping through TOR {#tor}

TOR gives you free anonymity and "endless" IP rotation. Technically, TOR is a local SOCKS5 proxy (by default 127.0.0.1:9050).

9.1. Installation and startup

# Linux
sudo apt install tor
sudo systemctl start tor

# macOS
brew install tor
brew services start tor

# check: TOR listens on 9050 (SOCKS) and optionally 9051 (control)

9.2. Requests through TOR

Since TOR is SOCKS5, you need a client that supports SOCKS. The most convenient is socksify:

require 'socksify'        # gem install socksify
require 'socksify/http'
require 'net/http'
require 'uri'

uri = URI('https://check.torproject.org')

Net::HTTP.SOCKSProxy('127.0.0.1', 9050).start(uri.host, uri.port, use_ssl: true) do |http|
  res = http.get(uri.path)
  puts res.body.include?('Congratulations') ? 'Through TOR ✓' : 'Not TOR ✗'
end

With Typhoeus over SOCKS (http.rb has no native SOCKS support):

require 'typhoeus'
resp = Typhoeus.get('https://check.torproject.org',
                    proxy: 'socks5://127.0.0.1:9050')
puts resp.code

9.3. Getting a new circuit (new IP) via the control port

To get a new IP, send the NEWNYM command to the control port (9051). First configure it in /etc/tor/torrc:

ControlPort 9051
HashedControlPassword 16:...   # generate with: tor --hash-password "yourpass"

Then:

require 'socket'

def tor_new_identity(password, host: '127.0.0.1', port: 9051)
  sock = TCPSocket.new(host, port)
  sock.puts %(AUTHENTICATE "#{password}")
  raise 'auth failed' unless sock.gets.start_with?('250')
  sock.puts 'SIGNAL NEWNYM'
  sock.gets
ensure
  sock&.close
end

# change identity every N requests
tor_new_identity('yourpass')
sleep 5   # give TOR time to build a new circuit

9.4. TOR's limitations

• Slow. Traffic goes through three relays — latency is high.
• Many sites block TOR exit nodes (exit-node lists are public).
• Not for large-scale collection — it overloads the volunteer-run TOR network. For high volume, use commercial residential proxies instead.

10. Multithreading and concurrency {#concurrency}

Fetching pages is an I/O-bound task: the program spends most of its time waiting on the network. That means concurrency delivers a huge win, and Ruby's GIL (Global VM Lock) does not get in the way here: during network waits a thread releases the GIL, and others keep working.

10.1. Simple threads (Thread)

require 'http'

urls = %w[https://example.com/1 https://example.com/2 https://example.com/3]

threads = urls.map do |url|
  Thread.new do
    resp = HTTP.get(url)
    [url, resp.status.to_i]
  end
end

results = threads.map(&:value)
results.each { |url, code| puts "#{code} #{url}" }

Downside: without a cap on the thread count, you can easily open 1000 connections at once and get banned or crash. You need a pool.

10.2. A bounded thread pool (queue)

require 'thread'
require 'http'

def crawl(urls, pool_size: 10)
  queue   = Queue.new
  results = Queue.new
  urls.each { |u| queue << u }

  workers = Array.new(pool_size) do
    Thread.new do
      until queue.empty?
        url = queue.pop(true) rescue break
        begin
          resp = HTTP.timeout(10).get(url)
          results << [url, resp.status.to_i, resp.to_s]
        rescue => e
          results << [url, :error, e.message]
        end
      end
    end
  end

  workers.each(&:join)
  Array.new(results.size) { results.pop }
end

crawl(urls, pool_size: 10).each { |url, code, _| puts "#{code} #{url}" }

10.3. concurrent-ruby — the industrial approach

The concurrent-ruby gem gives you ready-made pools and futures — no need to roll your own.

require 'concurrent-ruby'   # gem install concurrent-ruby
require 'http'

pool = Concurrent::FixedThreadPool.new(10)

futures = urls.map do |url|
  Concurrent::Future.execute(executor: pool) do
    HTTP.timeout(10).get(url).to_s
  end
end

futures.each { |f| puts f.value&.length }   # .value blocks until ready
pool.shutdown
pool.wait_for_termination

10.4. Typhoeus::Hydra — parallelism on libcurl

The most efficient option for purely network-bound parallelism: one thread, but libcurl drives many connections at once (multiplexing).

require 'typhoeus'

hydra = Typhoeus::Hydra.new(max_concurrency: 20)

requests = urls.map do |url|
  req = Typhoeus::Request.new(url, followlocation: true, timeout: 10)
  req.on_complete do |response|
    puts "#{response.code} #{url}"
    # parse response.body here
  end
  hydra.queue(req)
  req
end

hydra.run   # runs all requests in parallel

10.5. async (Fibers) — the modern alternative

The async gem uses fibers for thousands of simultaneous connections at almost no overhead.

require 'async'
require 'async/http/internet'

Async do
  internet = Async::HTTP::Internet.new
  tasks = urls.map do |url|
    Async do
      response = internet.get(url)
      puts "#{response.status} #{url}"
      response.read   # be sure to read/close
    end
  end
  tasks.each(&:wait)
ensure
  internet&.close
end

Which one to choose

• Up to a few dozen URLs — plain Thread + Queue.
• Production code — concurrent-ruby.
• Maximum speed, thousands of requests — Typhoeus::Hydra or async.

Important: Nokogiri parsing is CPU-bound, and here the GIL does get in the way. If your bottleneck is HTML parsing (not the network), true CPU parallelism requires processes (the Parallel gem, fork) or a GIL-free runtime like JRuby/TruffleRuby.

require 'parallel'   # gem install parallel
# 4 processes, genuinely parallel (they sidestep the GIL)
results = Parallel.map(urls, in_processes: 4) do |url|
  doc = Nokogiri::HTML(HTTP.get(url).to_s)
  doc.at_css('h1')&.text
end

11. JavaScript pages {#javascript}

Many modern sites render their content in the browser via JavaScript. In the raw HTML the server returns, the data you want simply isn't there. You then have two options.

11.1. Find the API (preferred)

Open DevTools → Network → XHR/Fetch. Usually the JS pulls data from a JSON API. Scraping that API directly is faster and more stable than driving a browser.

11.2. A headless browser

If you can't find the API, spin up a real browser without a UI and take the fully rendered DOM. This is the same approach we cover in more depth in headless browser scraping.

Ferrum — controls Chrome over CDP, pure Ruby, no Selenium:

require 'ferrum'   # gem install ferrum (requires an installed Chrome/Chromium)

browser = Ferrum::Browser.new(headless: true, timeout: 20)
page = browser.create_page
page.go_to('https://spa-site.example.com')
page.network.wait_for_idle   # wait for loading to finish

html = page.body             # the rendered DOM
doc  = Nokogiri::HTML(html)
puts doc.css('.dynamic-item').map(&:text)

browser.quit

Watir / Selenium — heavier, cross-browser, with a rich API for clicks/forms. See our guide to Selenium scraping for the details.

playwright-ruby-client — a modern alternative to Selenium, and a good fit if you already know Playwright.

Headless browsers are far slower and hungrier on memory. Use them only when there's no other way to get the rendered content.

12. Anti-bot defenses, delays, retries {#anti-bot}

For a scraper to run for a long time without getting banned, it has to behave "politely" and human-like.

12.1. Rate limiting

urls.each do |url|
  fetch(url)
  sleep(rand(1.0..3.0))   # a random pause — less bot-like
end

12.2. Rotating the User-Agent

USER_AGENTS = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ...',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 ...',
  'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ...'
]

HTTP.headers('User-Agent' => USER_AGENTS.sample).get(url)

12.3. Retries with exponential backoff

def fetch_with_retry(url, max: 4)
  attempt = 0
  begin
    attempt += 1
    resp = HTTP.timeout(10).get(url)
    raise "HTTP #{resp.code}" if resp.code >= 500 || resp.code == 429
    resp
  rescue => e
    if attempt < max
      delay = 2**attempt + rand   # 2, 4, 8... + jitter
      warn "Attempt #{attempt} failed (#{e.message}), waiting #{delay.round}s"
      sleep delay
      retry
    else
      raise
    end
  end
end

12.4. Other cloaking techniques

• send realistic headers (Accept, Accept-Language, Referer);
• keep a cookie session (like a browser);
• rotate proxies (section 8);
• respect Retry-After on 429;
• avoid a too-regular, "machine-like" request rhythm.

When the defense is a full CAPTCHA challenge, header tuning won't be enough — you'll need a solving service or a headless browser in the loop.

Ethics note: aggressively bypassing protections can violate ToS and the law. Stay within what's reasonable and legal.

13. Storing URLs and queues {#queues}

As soon as a scraper grows beyond "one script, one page," you face the problem of managing many URLs: what's already downloaded, what's queued, what failed.

13.1. The simplest option — an in-memory Set

Fine for one-off jobs. The key is deduplication, so you don't download the same thing twice.

require 'set'

visited = Set.new
queue   = ['https://example.com']

until queue.empty?
  url = queue.shift
  next if visited.include?(url)
  visited << url

  doc = Nokogiri::HTML(HTTP.get(url).to_s)
  # push new links onto the queue
  doc.css('a').each do |a|
    link = a['href']
    queue << link if link&.start_with?('https://example.com') &&
                     !visited.include?(link)
  end
end

Downside: it's lost on restart, doesn't scale, and doesn't work across processes.

13.2. A Redis queue — for serious scraping

Redis gives you a persistent queue shared across many workers/machines, plus built-in deduplication via sets (SADD).

require 'redis'   # gem install redis

redis = Redis.new

# add a URL (if we haven't seen it)
def enqueue(redis, url)
  # SADD returns 1 if the url is new
  if redis.sadd('seen', url) == 1
    redis.rpush('queue', url)
  end
end

# a worker takes the next URL (blocking)
def dequeue(redis)
  _list, url = redis.blpop('queue', timeout: 5)
  url
end

enqueue(redis, 'https://example.com')

while (url = dequeue(redis))
  process(url)
  # extracted links -> back onto the queue
end

Advantages: multiple workers on different machines pull from one queue; state survives restarts; it's easy to add "retry" and "failed" queues.

13.3. Ready-made job queue systems

For production scrapers you typically use a background job processor, where "download a page" is a job:

• Sidekiq (on Redis) — the most popular, multithreaded.
• GoodJob / Solid Queue (on PostgreSQL) — no separate Redis needed. Solid Queue is now the default in Rails 8.

# A Sidekiq worker example
class ScrapeWorker
  include Sidekiq::Job
  sidekiq_options retry: 3, queue: 'scraping'

  def perform(url)
    resp = HTTP.timeout(10).get(url)
    return unless resp.status.success?
    doc = Nokogiri::HTML(resp.to_s)
    save(doc)
    # spawn new jobs
    doc.css('a').each { |a| ScrapeWorker.perform_async(a['href']) if internal?(a['href']) }
  end
end

This gives you retries, priorities, monitoring, and horizontal scaling out of the box.

13.4. Storing results

The data itself goes into a database (PostgreSQL, SQLite, MongoDB) or files (CSV/JSON/Parquet). A minimal SQLite example:

require 'sequel'   # gem install sequel sqlite3

DB = Sequel.sqlite('scraped.db')
DB.create_table?(:pages) do
  primary_key :id
  String :url, unique: true
  String :title
  Integer :status
  DateTime :fetched_at
end

DB[:pages].insert_conflict(:replace).insert(
  url: url, title: title, status: 200, fetched_at: Time.now
)

13.5. Other things to remember for large crawls

• URL normalization (strip # anchors, extra params, canonicalize) — otherwise duplicates leak into the queue.
• Crawl depth and domain scoping, so you don't wander off across the whole internet.
• A Bloom filter to dedupe millions of URLs without storing every string.
• Priorities (important pages first).
• Checkpoints — so you can resume after a crash.

14. Ready-made frameworks {#frameworks}

You don't have to build everything by hand. Some tools cover the common cases.

14.1. Mechanize — "a browser without a UI"

Mechanize maintains a cookie session on its own, follows links, fills and submits forms, and follows redirects. It's ideal for scraping behind a login.

require 'mechanize'   # gem install mechanize

agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'

page = agent.get('https://example.com')
search = page.form_with(id: 'search') do |f|
  f.q = 'ruby scraping'
end.submit

search.links.each { |link| puts link.href }

Mechanize uses Nokogiri under the hood, so the same selectors are available (page.css(...)).

14.2. Kimurai — a full-blown spider framework

Kimurai is Ruby's answer to Python's Scrapy: routes, parse methods, built-in headless-browser support (Selenium/Ferrum), pipelines, and exports. If you know Scrapy, the mental model will feel familiar.

require 'kimurai'   # gem install kimurai

class NewsSpider < Kimurai::Base
  @name = 'news_spider'
  @engine = :mechanize           # or :selenium_chrome for JS
  @start_urls = ['https://example.com/news']

  def parse(response, url:, data: {})
    response.css('article.post').each do |post|
      item = {
        title: post.css('h2').text.strip,
        link:  post.css('a').first['href']
      }
      # go to the article page
      request_to :parse_article, url: item[:link], data: item
    end

    # pagination
    if (next_page = response.at_css('a.next'))
      request_to :parse, url: absolute_url(next_page['href'], base: url)
    end
  end

  def parse_article(response, url:, data: {})
    data[:body] = response.css('.content').text.strip
    save_to 'results.json', data, format: :json
  end
end

NewsSpider.crawl!

Note: recent versions of the original Kimurai have shifted toward an AI-assisted DSL. If you want "classic" Kimurai with ordinary selectors, look at the maintained fork Tanakai — the API is nearly identical.

14.3. Vessel / Wombat and others

• Vessel — a lightweight spider on top of Ferrum (fast, Chrome-based).
• Wombat — declarative field extraction via a DSL.
• Spidr — a simple site crawler.

When to reach for a framework

• One-off script → http.rb + Nokogiri by hand.
• Scraping behind a login, forms → Mechanize.
• A large, structured crawl with pagination/pipelines → Kimurai.

15. Pros and cons of Ruby for scraping {#pros-cons}

Advantages of scraping in Ruby

• Nokogiri — one of the best HTML/XML parsers anywhere; CSS + XPath out of the box.
• Expressive syntax — a Ruby scraper reads almost like pseudocode and is quick to write.
• A rich ecosystem — Mechanize, Kimurai, Ferrum, Typhoeus, Sidekiq, and so on cover practically any task.
• Excellent Rails integration — perfect when the data flows straight into a web app.
• I/O concurrency works well — for network tasks the GIL is no obstacle; threads/fibers deliver high concurrency.
• Mature queue and background-job tools (Sidekiq) — easy to take to production scale.

Downsides and gotchas

• The GIL limits CPU parallelism — if your bottleneck is HTML parsing rather than the network, you need processes or JRuby. For heavy CPU-bound parsing out of the box, Ruby trails Go/Rust.
• Interpreter speed is lower than compiled languages; it shows on huge volumes.
• JS-heavy sites require a headless browser — slow and resource-hungry (though that's true in any language).
• Nokogiri is a C extension — occasionally painful to build on unusual systems (though these days it usually installs cleanly).
• Scrapers are inherently fragile — any parser breaks when a site changes its markup; not Ruby-specific, but it needs maintenance.
• The ecosystem is smaller than Python's — Python (Scrapy, BeautifulSoup, requests) has more ready-made solutions and tutorials aimed specifically at scraping.

When Ruby is a good choice

When you're already in a Ruby/Rails stack, want readable maintainable code, your volumes are moderate, and your bottleneck is the network (I/O) rather than the CPU. For extreme volumes and pure CPU-bound parsing, look at Go/Rust or Python + Scrapy.

16. Conclusion {#conclusion}

Web scraping with Ruby is built from two bricks — an HTTP client and a parser — and everything else (encodings, proxies, threads, queues) grows around them as the task scales.

A quick tool-selection cheat sheet

Start simple (http + Nokogiri) and add complexity only when you genuinely need it — that's the guiding principle of a good scraper.

Let us run it for you

Building a scraper is one thing; keeping a fleet of them alive through markup changes, IP bans, and CAPTCHAs is another. If you'd rather have clean data delivered than maintain the plumbing, scraping.pro offers custom scraping and data as a service: you describe the sites and fields, we handle proxies, rendering, CAPTCHA solving, scheduling, and monitoring, and you get structured data on a schedule. It's the same architecture described above, run and maintained for you.

Official resources and documentation

Ruby standard library:

• OpenURI — fetch a page in one line
• Net::HTTP — the built-in HTTP client
• URI — parse and build URLs
• JSON — parse JSON
• Thread / Queue — threads and queues

HTTP clients: http.rb, Faraday (+ faraday-retry), Typhoeus, Mechanize.

Parsers: Nokogiri, Oga, Loofah.

Encodings: charlock_holmes (ICU-based detection).

Proxies / TOR: socksify, the Tor Project.

Concurrency: concurrent-ruby, async (+ async-http), parallel.

Headless browsers: Ferrum, Watir, Selenium, playwright-ruby-client.

Queues and storage: redis-rb, Sidekiq, Solid Queue, Sequel.

Frameworks: Kimurai / Tanakai, Vessel, Wombat, Spidr.

Other: webrobots — robots.txt parsing.