I have already written several articles on how to use Selenium WebDriver for web scraping and all those examples were for Windows. But what about if you want to run your WebDriver-based scraper somewhere on a headless Linux server? For example on a Virtual Private Server with SSH-only access. Here I will show you how to do it in several simple steps.

Let’s say you already have a virtual or dedicated Debian server with Python installed. The following tutorial will guide you from installing all necessary software to running your first WebDriver-based scraping program in Python. I assume that you are logged in as an administrator.

1. Install Xvfb

Since your server doesn’t have a screen to run FireFox you need to simulate one. Xvfb is a software that simulates a display doing everything in memory and not showing any screen output. You can install it with a simple command:

2. Install Firefox

If Firefox is not installed on your system you can install it in the following way:

The first command removes a native Debian browser Iceweasel (if it is installed on your system). Then we add a package repository that contains Firefox, install the corresponding key and update the local package list. After that we install Firefox with some libraries (some of them may probably be already installed on your system).

3. Install PyVirtualDisplay

PyVirtualDisplay is a Python wrapper for Xvfb. It allows you to easily work with a virtual display in Python. Installation is simple:

If you don’t have pip on your system you can install it with the following command:

4. Install Selenium

To install Selenium you can run the following:

5. Run a simple scraping program

Now we’re ready to run a simple program that uses Firefox for scraping Google’s home page title (I found this code here):

That’s it!