Linux is the natural home for web scraping. It's what almost every server runs, it schedules jobs with a single cron line, it runs headless browsers cheaply, and its shell lets you pipe small sharp tools together into a working scraper without writing a program. This guide is a practical toolkit for building a Linux web scraper — the command-line fetchers and parsers, the full stacks, how to schedule and run jobs unattended, and the terminal tips that make it all smoother.
Whether you want a throwaway one-liner or a production pipeline, everything here runs from the terminal. If you're new to the concept itself, start with what is web scraping; this article assumes you know the idea and want the Linux tooling.
Fetching pages from the terminal: curl and wget
The two tools you'll reach for first are already on nearly every Linux box.
curl is the Swiss-army knife for single requests — APIs, form posts, custom headers, cookies, auth:
# GET with a browser-like User-Agent, following redirects, gzip-aware
curl -sSL --compressed \
-H "User-Agent: Mozilla/5.0" \
"https://example.com/products" -o page.html
# POST a login form and keep the session cookies
curl -c cookies.txt -d "user=me&pass=secret" https://example.com/login
curl -b cookies.txt https://example.com/dashboard
wget is better for recursive downloads and mirroring an entire site or directory tree:
# Mirror a site into a local folder (assets included, links rewritten)
wget --mirror --convert-links --adjust-extension \
--page-requisites --no-parent https://example.com/
# Grab a whole directory over FTP into the current folder
wget -r -nH --cut-dirs=3 ftp://user:pass@ftp.example.com/a/b/c/
Rule of thumb: curl for one request you shape precisely, wget for pulling many files. Both take proxy settings (curl --proxy, or the http_proxy/https_proxy environment variables) for rotating your IP.
Parsing HTML on the command line
Fetching is half the job; the other half is pulling fields out of the HTML. You don't always need a full programming language — a few CLI parsers turn the shell into a scraper.
- pup — CSS selectors for HTML, like
jqfor markup.curl ... | pup 'h2.title text{}'. - htmlq — the same idea, Rust-based:
curl ... | htmlq --text 'a.link'. - xmllint — ships with libxml2; supports XPath:
xmllint --html --xpath '//h1/text()' page.html 2>/dev/null. - jq — the standard for JSON, which is what you get when you hit an API instead of HTML.
- grep / ripgrep (rg), sed, awk — quick line-oriented extraction and cleanup when the target is simple.
A complete scraper in one pipe — fetch a page, select product titles with a CSS selector, clean whitespace:
curl -sSL "https://example.com/shop" \
| pup 'div.product h2 text{}' \
| sed 's/^[[:space:]]*//; s/[[:space:]]*$//'
And for an API, no HTML parser needed at all:
curl -sSL "https://api.example.com/items?page=1" \
| jq -r '.items[] | [.id, .name, .price] | @csv'
Hitting the JSON endpoint directly like this is faster and far more robust than parsing rendered HTML — always check for one first (see scraping hidden APIs). CSS-selector fundamentals carry over from CSS selectors for scraping; messier text jobs fall to grep, sed, and awk.
Full scraping stacks on Linux
For anything beyond a one-liner, move to a real language runtime — all first-class on Linux.
Python is the default. requests/httpx to fetch, BeautifulSoup or selectolax to parse, and Scrapy when you need a full crawling framework with queues, retries, and pipelines built in. Always work inside a virtual environment:
python3 -m venv venv && source venv/bin/activate
pip install httpx selectolax
See the Python web scraping guide for the ecosystem.
Node.js pairs got/axios with Cheerio (jQuery-style parsing) and is a natural fit if your team writes JavaScript.
Headless browsers handle sites that render content with JavaScript — a plain curl gets an empty shell there. Playwright (Python or Node) is the current default; it runs Chromium/Firefox/WebKit headlessly and installs cleanly on Linux:
pip install playwright
playwright install --with-deps chromium # pulls the browser + system libs
The --with-deps flag is the Linux-specific bit: it installs the shared libraries headless Chromium needs, which is the usual cause of "it works on my laptop but not the server." More on when you need this in headless browser scraping.
Running headless browsers on a GUI-less server
Servers have no display, which trips up browser automation. Three ways through it:
- True headless mode — Playwright and Puppeteer run headless by default; no display needed. This is the modern default.
- xvfb — for a tool that insists on a display, wrap it in a virtual framebuffer:
xvfb-run -a python scraper.py. - Docker — the cleanest option for reproducibility. Official Playwright/Puppeteer images bundle the browser and every system dependency, so the container runs identically on your laptop and the server.
On locked-down environments you may also need --no-sandbox for Chromium (understand the security trade-off before using it in production).
Scheduling: cron and systemd timers
Unattended scheduling is where Linux shines. The classic tool is cron — one line per job:
# crontab -e
# Run the scraper every day at 3:15 AM, appending output to a log
15 3 * * * /home/user/venv/bin/python /home/user/scraper.py >> /home/user/scrape.log 2>&1
The five fields are minute, hour, day-of-month, month, day-of-week. A few patterns you'll reuse: */30 * * * * (every 30 minutes), 0 */6 * * * (every 6 hours), 0 9 * * 1 (9 AM every Monday). Always use absolute paths (cron runs with a minimal environment and a bare PATH) and redirect both stdout and stderr to a log so failures are visible.
The modern alternative is a systemd timer, which gives you logging via journalctl, dependency ordering, and easy catch-up if the machine was off. A timer is two small unit files (scraper.service + scraper.timer) — worth it for anything important, since debugging a silent cron job is a rite of passage nobody enjoys.
Terminal tips that make scraping smoother
A grab bag of the practical stuff, modernized for scraper work:
Make scripts executable and safe to run:
chmod +x scraper.sh # add execute permission
chmod 750 scraper.sh # owner rwx, group r-x, others none
chown -R youruser:yourgroup /opt/scraper # fix ownership after deploy
Octal permissions in one glance: 7 = read+write+execute, 6 = read+write, 5 = read+execute, 4 = read-only, 0 = none — summed per user/group/other.
Keep long jobs alive after you log out:
nohup python scraper.py & # survives the terminal closing; output -> nohup.out
tmux new -s scrape # or run inside tmux/screen and detach with Ctrl-b d
For a big crawl that runs for hours, tmux (or screen) is the friendly choice — reattach later with tmux attach -t scrape to check progress.
Find and manage things quickly:
find / -name "chromedriver" 2>/dev/null # locate a binary
du -sh ./output/* # see what your scraper is filling up
tail -f scrape.log # watch a running job's log live
Rotate logs and prune old data so a long-running scraper doesn't fill the disk — logrotate for logs, and a find ./data -mtime +30 -delete line in cron to age out old files.
Proxies, politeness, and rate limiting
A scraper that hammers a site from one IP gets blocked and is a bad neighbor. On the CLI:
- Proxies —
curl --proxy http://user:pass@ip:port, or sethttps_proxyfor the whole session. Rotate through a pool for volume, ideally residential IPs, which draw less scrutiny than datacenter ranges. - Slow down — add
sleepbetween requests in a loop, and jitter it (sleep $((RANDOM % 3 + 1))) so the rhythm isn't robotic. - Set a real User-Agent and reuse cookies/sessions where the site expects them.
- Respect robots.txt and rate limits — being polite keeps you unblocked and on the right side of the rules; see is web scraping legal.
Docker for reproducible scrapers
Once a scraper matters, put it in a container. A small Dockerfile pins the language runtime, system libraries, and browser so the job runs identically everywhere — no "works on my machine," and trivial to deploy to any Linux server or scheduler. Combined with a cron/systemd trigger or a CI schedule, this is the standard shape of a production Linux scraper in 2026.
FAQ
What's the best Linux tool for quick scraping?
For one-off jobs, curl piped into pup, htmlq, or jq gets you data in a single command line. For anything repeatable, move to Python (httpx + selectolax, or Scrapy) or add Playwright for JavaScript-rendered sites.
How do I schedule a scraper on Linux?
cron for simple recurring jobs (crontab -e, one line with absolute paths and a log redirect), or a systemd timer when you want proper logging and catch-up behavior.
Can I run a headless browser on a server with no display?
Yes. Playwright and Puppeteer run truly headless by default. If a tool demands a display, wrap it in xvfb-run; for reproducibility, use the official Docker images that bundle the browser and its dependencies.
Do I need proxies to scrape from Linux?
Not for small, polite jobs. For volume, or any site with anti-bot defenses, rotate proxies (curl --proxy or the https_proxy env var), pace requests, and set a real User-Agent to avoid blocks.
The Linux toolkit gets you scraping fast, but running crawls reliably at scale — proxies, headless rendering, scheduling, retries, and upkeep as sites change — is ongoing work. If you'd rather receive clean, structured data than operate the pipeline, scraping.pro offers data as a service: we run the scrapers on our infrastructure and deliver results to your database or as files on your schedule.