Plenty of modern sites render their content with JavaScript, so a plain HTTP request returns an empty shell. To get the real data you need a browser that executes that JavaScript — and on a server with no monitor, that browser has to run without a graphical display. Headless Firefox scraping is exactly that: driving a real Firefox engine on a headless Linux box, controlled from Python with Selenium.
This tutorial takes you from a bare Debian/Ubuntu server to a working scraper. If you have read older guides that rely on Xvfb and PyVirtualDisplay to fake a screen, good news — that whole layer is obsolete. Firefox has shipped a native headless mode for years, and Selenium now fetches its own driver, so the setup in 2026 is far shorter than it used to be.
Why headless Firefox instead of plain requests
If a page's data is already in the initial HTML, you do not need a browser at all — use requests and an HTML parser, which is far faster and lighter. Reach for a headless browser only when:
- content is injected by JavaScript after load (infinite scroll, SPAs, dashboards),
- you need to click, type, or wait for elements before the data appears, or
- the site's behavior depends on real browser rendering.
Firefox is a solid choice here: it is stable on Linux, honors a real rendering engine, and its geckodriver is well maintained. Chrome/Chromium works the same way if you prefer it.
What you no longer need: Xvfb
The classic recipe simulated a virtual X display with Xvfb and wrapped it in PyVirtualDisplay so a normal (non-headless) Firefox thought it had a screen. That still works, but it is extra moving parts. Since Firefox 55, the browser runs truly headless on its own with a single flag. Skip Xvfb unless you specifically need to run a non-headless browser on a display-less machine (occasionally useful for evading bot detection that checks for headless signals).
Step 1: Install Firefox
On Debian or Ubuntu, install the packaged Firefox (the ESR build is a stable pick for servers):
sudo apt-get update
sudo apt-get install -y firefox-esr
The old sourceforge/ubuntuzilla repositories and the iceweasel removal step from 2014-era tutorials are no longer relevant. Confirm it installed:
firefox --version
Step 2: Install Python and Selenium
Use Python 3 (Python 2 is long dead). Create a virtual environment so system packages stay clean:
sudo apt-get install -y python3 python3-venv python3-pip
python3 -m venv venv
source venv/bin/activate
pip install selenium
Step 3: The driver — geckodriver
Selenium talks to Firefox through geckodriver. Here is the big time-saver: since Selenium 4.6, Selenium Manager is built in and downloads the correct geckodriver automatically the first time you run your script. In most cases you do not have to install anything.
If you are on a locked-down box with no outbound access for the manager, install geckodriver manually — grab the latest Linux release from the geckodriver GitHub releases page and put the binary on your PATH:
# example: adjust the version/URL to the current release
tar -xzf geckodriver-*-linux64.tar.gz
sudo mv geckodriver /usr/local/bin/
geckodriver --version
Step 4: Your first headless Firefox scraper
Here is the modern equivalent of the old Xvfb example — no virtual display, native headless mode, Python 3 syntax:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)
driver.get("https://www.example.com")
print(driver.title)
driver.quit()
That is the entire program. options.add_argument("--headless") is what replaces the old Display(visible=0) dance. Run it over SSH and it prints the page title with no GUI in sight.
Step 5: Scraping JavaScript-rendered content
Printing a title proves the plumbing works; real scraping means waiting for dynamic content and extracting it. Never use fixed time.sleep() calls — wait for the specific element instead:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("--headless")
options.add_argument("--width=1920")
options.add_argument("--height=1080")
# a realistic user agent helps blend in
options.set_preference(
"general.useragent.override",
"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0",
)
driver = webdriver.Firefox(options=options)
try:
driver.get("https://quotes.toscrape.com/js/") # a JS-rendered demo site
WebDriverWait(driver, 15).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".quote"))
)
for quote in driver.find_elements(By.CSS_SELECTOR, ".quote"):
text = quote.find_element(By.CSS_SELECTOR, ".text").text
author = quote.find_element(By.CSS_SELECTOR, ".author").text
print(f"{author}: {text}")
finally:
driver.quit()
WebDriverWait + expected_conditions pauses only until the content actually exists, which is both faster and far more reliable than guessing a sleep duration. Setting an explicit window size avoids mobile/responsive layouts that hide elements at the default headless viewport.
Running it reliably on a server
For a scraper meant to run unattended, wrap the browser in try/finally (as above) so a crash never leaves an orphaned firefox process eating memory. If you launch many browsers, watch RAM — each Firefox instance is heavy, so scrape in small pools rather than hundreds at once. To keep the job running after you disconnect from SSH, start it under tmux, nohup, or a systemd service — see the companion guide on running scrapers detached.
Avoiding blocks
A headless browser renders like a real one, but sites still detect automation. To scrape at scale without getting blocked:
- rotate your IP with rotating proxies so requests don't all originate from one server address,
- set a believable user agent and window size (as shown above),
- throttle your request rate and randomize timing,
- be ready to handle challenges with a CAPTCHA-solving service when they appear.
Some anti-bot systems specifically fingerprint headless Firefox. If you hit persistent walls, undetected browser variants or a switch to Playwright (below) can help — but respecting rate limits does more than any single trick.
Alternative: Playwright
Selenium is the veteran, but Playwright has become the go-to for new browser automation projects. It drives Firefox (and Chromium and WebKit) headlessly, auto-waits for elements by default, and installs its own browser binaries with playwright install firefox. If you are starting fresh, it is worth a look; if you already have Selenium code, the setup above is all you need.
Summary
Headless Firefox scraping on Linux is now a five-line setup: install firefox-esr, pip install selenium, and pass --headless. Selenium Manager handles geckodriver, and native headless mode retires the old Xvfb/PyVirtualDisplay stack entirely. Add explicit waits, a realistic user agent, and proxy rotation, and you have a robust scraper for JavaScript-heavy sites.
If you would rather not maintain servers, drivers, and anti-bot logic yourself, scraping.pro runs headless browser fleets as a managed web scraping service and delivers the extracted data directly.