C# Selenium Web Scraping: A Step-by-Step Example

Browser automation tools make it possible to drive a real browser from your own code, and that turns out to be a very natural way to collect data from websites. In this guide we walk through a small, self-contained C# program that opens Chrome, signs in to a protected page, and reads text out of the area that is only visible after authentication. The approach generalizes to almost any site you can navigate by hand.

The tool we will use is Selenium WebDriver — the part of the Selenium project that controls browsers through their native automation interfaces. Because a genuine browser does the work, you do not have to reverse-engineer HTTP requests, manage cookies, or replicate a login flow at the network level. You interact with the page the same way a person would: type into fields, click buttons, and read what appears on screen.

Prerequisites

To follow along you will need:

• The .NET SDK (any currently supported version; the examples target .NET 8). You can get it from the official .NET downloads page.
• Google Chrome installed on the machine that runs the program.
• An IDE or editor such as Visual Studio, Visual Studio Code, or JetBrains Rider.

That is all. Notably, you no longer need to hunt down a matching chromedriver executable by hand — more on that in the next section.

Adding Selenium to the project

Selenium's .NET bindings are distributed as NuGet packages, so installing them is a one-line operation. From the project directory run:

dotnet add package Selenium.WebDriver
dotnet add package Selenium.Support

The first package, Selenium.WebDriver, contains the core API. The second, Selenium.Support, adds convenience helpers such as the explicit-wait utilities we use below. Both are published by the Selenium project itself; the official installation instructions live in the Selenium documentation.

A welcome change in recent versions of Selenium is Selenium Manager. Starting with Selenium 4.6, the bindings can locate the right browser driver automatically — downloading and caching a compatible chromedriver for the Chrome you have installed. This is built into the library, so for ordinary local runs you do not configure any driver path at all. The details of how this works are described in the Selenium Manager documentation.

Importing the namespaces

With the packages in place, bring in the namespaces the program relies on:

using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;

OpenQA.Selenium exposes the core types (IWebDriver, By, IWebElement, and so on), OpenQA.Selenium.Chrome provides the Chrome-specific driver, and OpenQA.Selenium.Support.UI is where the waiting helpers live.

Launching the browser

Creating a ChromeDriver instance starts a fresh Chrome window that your code now controls through the driver object:

using IWebDriver driver = new ChromeDriver();

Wrapping the driver in a using declaration guarantees the browser process is shut down when the variable goes out of scope, even if something throws along the way. Thanks to Selenium Manager, the constructor finds an appropriate driver for you; if you ever need to point at a specific driver binary or pass Chrome options, the ChromeDriver constructor and ChromeOptions class let you do that.

Navigating to the target page

The example signs in to a small public practice page designed for exactly this kind of exercise. Send the browser there with:

driver.Navigate().GoToUrl("https://the-internet.herokuapp.com/login");

The call blocks until the page has loaded, after which the login form is ready to be filled in.

Locating the page elements

To interact with the form we first need references to its fields and its submit button. Selenium locates elements through the By factory, which supports strategies such as ID, name, CSS selector, and XPath. The official reference for these strategies is the locating-elements guide.

var userNameField = driver.FindElement(By.Id("username"));
var passwordField = driver.FindElement(By.Id("password"));
var loginButton   = driver.FindElement(By.CssSelector("button[type='submit']"));

Here the two input fields are found by their id attributes, while the login button is matched with a CSS selector that targets the form's submit <button>.

Filling in and submitting the form

Once you hold references to the elements, typing and clicking are direct method calls:

userNameField.SendKeys("tomsmith");
passwordField.SendKeys("SuperSecretPassword!");
loginButton.Click();

SendKeys enters text exactly as if it were typed on a keyboard, and Click performs a real click. Submitting the form causes Chrome to load the page that lives behind the login.

Waiting for the protected content

After a click that triggers navigation, the data you want may not exist in the DOM for a few moments. Rather than guessing with a fixed delay, use an explicit wait, which polls the page until a condition is satisfied or a timeout elapses. This is both faster and more reliable than a hard-coded sleep; see the waiting strategies documentation for the full picture.

var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
var resultElement = wait.Until(d => d.FindElement(By.CssSelector(".subheader")));

The wait returns as soon as the target heading is present, or throws after ten seconds if it never appears.

Extracting and saving the data

With the element in hand, read its text and write it to a file:

var result = resultElement.Text;
File.WriteAllText("result.txt", result);

Text returns the rendered text content of the element, and File.WriteAllText (from System.IO) persists it to disk. From here you could just as easily push the value into a database, an API call, or a larger dataset.

Shutting down

When the work is finished, close the browser and release the driver:

driver.Quit();

If you used the using declaration shown earlier, this happens automatically; calling Quit explicitly is simply the clearest way to express intent in a short program.

The complete program

Putting every step together yields a compact console application:

using System;
using System.IO;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;

namespace WebDriverScraper
{
    internal class Program
    {
        static void Main()
        {
            // Start Chrome (Selenium Manager resolves the driver automatically)
            using IWebDriver driver = new ChromeDriver();

            // Open the login page
            driver.Navigate().GoToUrl("https://the-internet.herokuapp.com/login");

            // Locate the form fields and the submit button
            var userNameField = driver.FindElement(By.Id("username"));
            var passwordField = driver.FindElement(By.Id("password"));
            var loginButton   = driver.FindElement(By.CssSelector("button[type='submit']"));

            // Enter the credentials and sign in
            userNameField.SendKeys("tomsmith");
            passwordField.SendKeys("SuperSecretPassword!");
            loginButton.Click();

            // Wait for the protected content to appear, then read it
            var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
            var resultElement = wait.Until(d => d.FindElement(By.CssSelector(".subheader")));

            // Save the scraped text to a file
            File.WriteAllText("result.txt", resultElement.Text);

            // Close the browser
            driver.Quit();
        }
    }
}

Conclusion

The appeal of WebDriver is how little ceremony it demands: you press keys and click buttons much as you would by hand, and the browser quietly handles the requests, redirects, and cookies underneath. That makes it an excellent fit for scraping pages that hide behind logins, render content with JavaScript, or otherwise resist a plain HTTP approach. With the modern Selenium bindings and Selenium Manager doing the driver bookkeeping, the gap between "I can do this manually" and "I can automate this" is now very small.

To go further, the official Selenium documentation covers advanced locators, interacting with frames and windows, executing JavaScript, and configuring headless runs for unattended environments.