Headless Firefox Scraping on Linux: A Tutorial

Run sudo apt-get install -y firefox-esr on a fresh Ubuntu 24.04 server and apt answers that the package has no installation candidate. Ask for plain firefox and a simulated install pulls in snapd 2.76, systemd, udev, apparmor and squashfs-tools, then installs firefox 1:1snap1-0ubuntu5, a stub whose only job is to fetch the Firefox snap. Point geckodriver at the result and your scraper can hang before it loads a page. Both commands were run on Ubuntu 24.04.4 on 13 August 2026, and the second is why this tutorial no longer starts where it used to.

Mozilla documents the hang in geckodriver's own changelog, under a heading called Known problems: "When Firefox is packaged inside a container (like the default Firefox browser shipped with Ubuntu versions since 22.04), it may see a different filesystem to the host. This can affect access to the generated profile directory, which may result in a hang when starting Firefox." That warning is still at the top of the changelog for 0.37.1.

The rest is easier than it was. Firefox runs headless natively, so the old virtual-display layer is gone, and Selenium now fetches not just the driver but the browser too. This guide takes a bare Debian or Ubuntu box to a working scraper and says which parts of the classic recipe are dead.

Where the versions stand on 13 August 2026

Everything below was checked against these, on the date given.

Two of those change what you can write. Selenium's wheel declares Requires-Python: >=3.10, so a box still on Python 3.9 gets an old Selenium silently resolved by pip. And geckodriver's support table lists 0.37.1 as working with Firefox 115 ESR and anything newer, with no upper bound. There is no version-matching puzzle here the way there is with Chrome and chromedriver.

Pick ESR if the machine is meant to be left alone. Pick stable if you care about matching what real visitors send: a scraper announcing Firefox 140 while everyone else is on 153 is a small, free signal handed to the other side.

When a browser is the wrong tool

A headless browser costs orders of magnitude more than an HTTP request, in memory and in wall-clock time. Check whether you need to pay that.

Fetch the target with curl and read what comes back. If the data is already in that HTML, use requests and a parser and stop here. If the page is empty but the network tab shows a JSON endpoint feeding it, call that endpoint directly. That is usually faster and easier to debug than driving a browser, and it is covered in more depth in JavaScript rendering libraries for scraping.

The arithmetic is worth doing once. One extra second per page across a ten-thousand-page crawl is close to three hours. A browser that takes four seconds where a request takes one costs you a day and a half on the same crawl, every time you run it.

Reach for Firefox when the content genuinely arrives after load, when you have to click or scroll before it appears, or when the site's behavior depends on real rendering. Those cases are common. They are not every case. The broader trade-offs are laid out in Selenium web scraping.

Step 1: Get a Firefox geckodriver can drive

Debian. sudo apt-get install -y firefox-esr works and gives you a normal, unconfined binary. Nothing else to do.

Ubuntu. The firefox-esr package does not exist, and firefox is a snap stub. Three ways out, in descending order of how much we would trust them on a server.

Mozilla ships its own APT repository, documented on its Linux install page. This gives you a real .deb that updates with the rest of the system:

sudo install -d -m 0755 /etc/apt/keyrings
wget -q https://packages.mozilla.org/apt/repo-signing-key.gpg -O- \
  | sudo tee /etc/apt/keyrings/packages.mozilla.org.asc > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/packages.mozilla.org.asc] https://packages.mozilla.org/apt mozilla main" \
  | sudo tee /etc/apt/sources.list.d/mozilla.list > /dev/null
printf 'Package: *\nPin: origin packages.mozilla.org\nPin-Priority: 1000\n' \
  | sudo tee /etc/apt/preferences.d/mozilla > /dev/null
sudo apt-get update && sudo apt-get install -y firefox

The pin matters. Without Pin-Priority: 1000 apt can decide the Ubuntu snap stub is the better candidate and quietly put you back where you started.

Second option: download the tarball, unpack it anywhere, and point Selenium at it with options.binary_location. This is what geckodriver's usage documentation recommends first, in these words: "Do not use container-packaged Firefox builds with geckodriver. Instead download a Firefox release from https://download.mozilla.org/?product=firefox-latest&os=linux and a geckodriver release from https://github.com/mozilla/geckodriver/releases."

Third option, if you are stuck with the snap: the same page offers a workaround. "Set the --profile-root command line option to write the profile to a directory accessible to both Firefox and geckodriver, for example a non-hidden directory under $HOME." In Python that is a service argument:

from selenium import webdriver

service = webdriver.FirefoxService(service_args=["--profile-root", "/home/scraper/ffprofiles"])

That is a workaround, not a fix. The confinement is still there, and so are the surprises about which paths Firefox can read. On a machine you control, install a real Firefox, then confirm it with firefox --version.

Step 2: Python and Selenium

Python 2 has been gone for years, and Python 3.9 is out too as far as Selenium is concerned. Check before you build the venv:

python3 --version          # needs to be 3.10 or newer
sudo apt-get install -y python3 python3-venv python3-pip
python3 -m venv venv
source venv/bin/activate
pip install "selenium>=4.47"

Pin the lower bound. pip install selenium on an old interpreter resolves to whatever last supported it, and you find out months later when a BiDi call fails with an error message that makes no sense.

Step 3: geckodriver, and why you probably will not install it

Selenium Manager has shipped inside the wheel since 4.6 and now does more than fetch drivers. Its documentation has it discovering, downloading and caching both: "the downloaded assets (drivers and browsers) are stored" in ~/.cache/selenium by default. On a box with no Firefox at all, it will go and get one.

You can watch it decide. Running the bundled binary on a container with neither piece installed produced this, verbatim:

[DEBUG] geckodriver not found in PATH
[DEBUG] firefox not found in PATH
[DEBUG] firefox not found in the system
[ERROR] error sending request for url (https://product-details.mozilla.org/1.0/firefox_versions.json)

That last line is the useful one. Selenium Manager resolves the Firefox version from the same Mozilla feed quoted in the table above, and the driver from GitHub. If your build machine cannot reach those two hosts, this is exactly what you will see, and the traceback that follows will not name either of them.

This is where the earlier version of this article was too breezy. It said "in most cases you do not have to install anything" and left it there. The second half matters: when the manager cannot reach the network, you have to stage the assets yourself, and there are environment variables built for that.

export SE_CACHE_PATH=/opt/selenium-cache   # where drivers and browsers live
export SE_OFFLINE=true                     # never try the network
export SE_AVOID_BROWSER_DOWNLOAD=true      # use the system Firefox, do not fetch one
export SE_SKIP_DRIVER_IN_PATH=true         # ignore a stray geckodriver on PATH
export SE_PROXY=http://proxy.internal:3128 # if outbound goes through a proxy

The same keys can live in an se-config.toml inside the cache directory, which is easier to bake into an image than a list of exports. Selenium 4.47.0 pins geckodriver v0.37.1 in its own build files, so that is the version to stage. Take the Linux release from the geckodriver releases page and put the binary on your PATH:

tar -xzf geckodriver-v0.37.1-linux64.tar.gz
sudo mv geckodriver /usr/local/bin/
geckodriver --version

Step 4: The smallest scraper that works

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument("-headless")

driver = webdriver.Firefox(options=options)
driver.get("https://www.example.com")
print(driver.title)
driver.quit()

That is the whole program, and it is what replaced the old Display(visible=0) dance.

One dash or two makes no difference, and it is worth knowing why. Selenium's Firefox documentation uses -headless. Plenty of code in the wild, including the previous version of this page, uses --headless. Firefox strips one leading dash, then strips a second if it is there, before matching the flag name. Both spellings land on the same switch. Firefox's --help prints it as --headless Run without a GUI, and setting it is equivalent to exporting MOZ_HEADLESS=1 before launch.

One thing that used to work and no longer does: options.headless = True was removed from the Python bindings. On 4.47.0 the attribute is absent, so assigning it creates a property nobody reads, and you get a visible browser or a crash on a display-less box.

Step 5: Waiting for content that JavaScript writes

Printing a title proves the plumbing. Real scraping means waiting for the thing you want. Fixed time.sleep() calls are the wrong tool: too short and you get an empty list, too long and you pay for it on every page of the crawl.

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("-headless")
options.add_argument("--width=1920")
options.add_argument("--height=1080")

driver = webdriver.Firefox(options=options)
try:
    driver.get("https://quotes.toscrape.com/js/")
    WebDriverWait(driver, 15).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".quote"))
    )
    for quote in driver.find_elements(By.CSS_SELECTOR, ".quote"):
        text = quote.find_element(By.CSS_SELECTOR, ".text").text
        author = quote.find_element(By.CSS_SELECTOR, ".author").text
        print(f"{author}: {text}")
finally:
    driver.quit()

quotes.toscrape.com/js/ is still up and still serves markup with no quote text in it, which is the entire point of that variant. Fetching it without a browser gives you a header, a footer and nothing in between.

--width and --height are the Firefox spelling, and they are not Chrome's. Firefox's command-line handler reads width and height as window features. Its --window-size flag exists but is documented in the source as "Width and optionally height of screenshot", tied to --screenshot rather than to the browsing window. Copying --window-size=1920,1080 from a Chrome recipe leaves you on the default headless viewport, where the page may render its mobile layout and hide the elements you were about to select.

Presence is not visibility. presence_of_element_located returns as soon as the node exists in the DOM, which on a lazily-populated list can be before it has any text. When elements that clearly exist hand back empty strings, switch to visibility_of_element_located.

Do you still need Xvfb?

Short answer: no, and the numbers say why.

The classic recipe ran a virtual X server with Xvfb, wrapped it in PyVirtualDisplay, and let an ordinary Firefox believe it had a screen. It works. It also costs a whole extra process. On Ubuntu 24.04.4, x86_64, 2 vCPU, 8 GB RAM, we started Xvfb :99 -screen 0 1920x1080x24 and measured a resident set of 69,036 KB; at 1280x720x24 it was 64,568 KB. Measured with ps -o rss= three seconds after launch, with nothing connected to the display. Your numbers will differ; the shape will not. Roughly 65 MB per worker, spent on a screen nobody looks at.

The Python wrapper has stopped moving too. PyVirtualDisplay's last release on PyPI is 3.0, from 13 February 2022. The alternative wrapper xvfbwrapper is alive, at 0.2.29 released 23 July 2026.

Two situations still justify the extra process.

You deliberately want a headful browser. Some anti-bot checks look for signals only a headless build produces. A normal Firefox inside Xvfb behaves like a desktop one on a machine with no desktop. This is the genuine use left for the old stack.

You need something headless mode refuses to do. Firefox's startup code carries the comment "Nothing can dismiss a modal dialog in headless mode", and repeats it a few hundred lines down. Native dialogs, some GPU-dependent paths and anything needing a real compositor are where headless stops matching headful. If a page works in your desktop Firefox and dies on the server, test that before blaming detection.

For everything else, -headless is one flag and Xvfb is a service to install, start, supervise and eventually debug.

What breaks at ten thousand pages

A single script that scrapes one page is forgiving. A run that opens ten thousand is not.

Orphaned processes are the classic overnight failure. A crashed script that never calls driver.quit() leaves both geckodriver and firefox alive, holding memory until the box swaps. The try/finally above is not decoration. geckodriver 0.37.0 improved its own side of this, adding a graceful Firefox shutdown with a forced kill as fallback when the driver terminates. It still cannot help you if the Python process is the one that dies.

Memory, not CPU, sets your concurrency. Each Firefox is heavy and each tab is heavier. Measure one worker against your own target pages with ps -o rss=, divide free RAM by that, then take a third of the answer as the pool size. A pool of four that never dies beats a pool of forty that gets OOM-killed at 3 a.m.

Profile directories accumulate. Every session writes a fresh profile under the system temp directory, and unclean kills leave them behind. --profile-root gives you one place to sweep.

Restart the browser every few hundred pages. Long-lived Firefox processes grow. That costs a second each time and removes a class of slow leak.

Keep the run alive after you log out. tmux, nohup and systemd all work; the practical differences are in running scrapers detached with real-time output. A unit with Restart=on-failure and a memory limit is the version that survives contact with production.

Where headless Firefox stops

CDP is gone from Firefox, permanently. Mozilla announced the deprecation in May 2024, writing that "Users of CDP in Firefox are encouraged to migrate to the W3C WebDriver BiDi protocol", then finished the job. Firefox's remote protocol preferences page now records that "With the end of the CDP support, WebDriver BiDi is the only available protocol, and the preference was removed in Firefox 141." Selenium 4.47.0 still sets remote.active-protocols to 1 in every Firefox session it creates, a harmless leftover on anything newer than 141. Tooling that drives Firefox over CDP is dead code.

There is no maintained stealth layer for Firefox. The packages people reach for tell the story by their release dates: selenium-stealth last shipped 1.0.6 in November 2020, geckodriver-autoinstaller in February 2020. The Chrome side of this ecosystem is active. The Firefox side is not.

Nobody publishes a reliable list of headless Firefox signals. Firefox does not brand its user agent when headless, unlike the old headless Chrome that put HeadlessChrome in the string; the code that builds the user agent in Firefox's networking layer contains no headless branch at all. What does give you away is navigator.webdriver, standardized and true whenever an automation session is attached, headless or not. Percentages quoted for "headless detection rates" trace back to vendors selling either the detection or the evasion, not to a published measurement.

Rate and origin do more work than any of it. Rotate addresses with residential proxies, throttle, randomize timing, and keep a captcha-solving service ready for what still gets through. Do not pin a user agent to an old version. The previous version of this page suggested Firefox/128.0, which today is twenty-five major releases behind the browser actually rendering the page. The gap between what you claim and what your TLS handshake and JavaScript engine reveal is a stronger signal than the default string ever was.

Renting the browser instead of running it

Sometimes the fleet is the problem and the data is the point. Browserbase publishes per-hour pricing for hosted browsers: free for one browser hour, $20 a month for 25 concurrent browsers and 100 browser hours with overage at $0.12 per browser hour, $99 a month for 100 concurrent and 500 hours at $0.10 over, and custom pricing above that. Their own page puts 100 hours at "roughly 3,000 page-level tasks". Prices read from that page on 13 August 2026.

Against a server you administer yourself, the arithmetic is rarely close on raw cost and often close on total cost, once you count driver upgrades, OOM kills and the night the snap package changed underneath you. Where a target fights back hard enough that the browser layer becomes a full-time job, a managed extraction service or a plain data feed is the end of that trade. Running your own on a shared host has its own constraints, worked through in headless browsers on PythonAnywhere.

Playwright, and when it is worth switching

Playwright reached 1.62.0 on 31 July 2026 and is the default choice for new browser automation work. It auto-waits on every action, drives Firefox, Chromium and WebKit through one API, and installs its own browsers with playwright install firefox.

That last point deserves a caveat this article used to skip. The Firefox Playwright installs is not Mozilla's release build. Its browser documentation states plainly: "Playwright doesn't work with the branded version of Firefox since it relies on patches." You get a build that tracks Stable closely and is not the binary your users run. For most scraping that is irrelevant. Where matching a real browser is the point, Selenium plus a stock Firefox is the more faithful stack, and that is why the setup above is worth keeping.

Summary

The five-line version holds: get a real Firefox, pip install "selenium>=4.47", pass -headless. What decides whether it survives a night is elsewhere. Avoid the Ubuntu snap, or work around it with --profile-root. Know that Selenium Manager fetches the browser too, and which two hosts it needs to do it. Wait on elements rather than on the clock, size the window with --width and --height, and let Xvfb go unless you want a headful browser on a display-less machine. Firefox and geckodriver are both actively maintained as of August 2026, with no version-matching puzzle between them. The virtual display is the only part of the classic recipe that has genuinely expired.