Fetch https://the-internet.herokuapp.com/login with HttpClient and you get the form. Post to it and you get a 302, a session cookie, and a one-time flash message that only exists on the page after the redirect. Drop any one of those three and your scraper still finishes with exit code 0, still writes a file, and still tells you nothing went wrong. A real browser carries all three without being asked, and that is the whole argument for driving one from C#.
The tool here is Selenium WebDriver, the part of the Selenium project that steers browsers through their own automation interfaces. You do not reverse-engineer the request sequence, rebuild the cookie jar, or hunt for the anti-forgery field. You type into boxes, click things, and read what appears.
Everything below was rechecked on 10 August 2026 against Selenium .NET 4.47.0, which landed on NuGet that same day, .NET 10.0, and Chrome stable 151.0.7922.77. Three claims in the earlier version of this article were either wrong at the time or have gone stale since. Each one is called out where it appears rather than quietly edited away.
What a browser costs before you write a line
A browser is not a slightly slower HTTP client. It downloads every asset the page references, builds a DOM, runs the JavaScript, and holds a process open while it does. The market prices that gap openly. ScrapingBee bills 1 credit for a plain fetch and 5 credits for a JavaScript-rendered one, per its API documentation read on 10 August 2026. Its $99 Startup plan carries a million credits a month. That is a million raw fetches, or two hundred thousand rendered ones, or roughly fifty cents per thousand rendered pages. Switch on the stealth proxy at 75 credits per call and the same $99 buys 13,333 pages, about $7.40 per thousand.
You pay that same ratio in wall-clock time when you run the browser yourself. Selenium's cost lands as seconds and RAM instead of credits, and the arithmetic is unforgiving at volume. One extra second per page across a ten-thousand-page crawl is just under three hours of runtime you did not plan for.
So the first question is not which locator to use. It is whether you need a browser at all. Open the network tab and submit the form by hand. If the page fills itself from a JSON endpoint, call that endpoint directly and skip everything below. WebDriver earns its keep on the pages where the HTML you need is assembled in the browser, or sits behind a login, or depends on a script you would otherwise have to reimplement.
Prerequisites
- The .NET SDK. The current long-term-support release is .NET 10, SDK 10.0.302, published 14 July 2026 on the .NET downloads page. The earlier version of this guide targeted .NET 8. That is the first stale claim: per Microsoft's own release index, .NET 8 and .NET 9 both leave support on 10 November 2026, three months from now. Target .NET 10 unless something pins you.
- Google Chrome on the machine that runs the program. Stable was 151.0.7922.77 when we checked the Chrome for Testing feed on 10 August 2026.
- An editor. Visual Studio, VS Code, and Rider all work; nothing here depends on the IDE.
You do not need to download a matching chromedriver by hand any more. The caveats to that live two sections down.
One package, not two
dotnet add package Selenium.WebDriverThat is the whole install. Selenium.WebDriver 4.47.0 went up on 10 August 2026, is owned by the selenium account, and has 181.8 million downloads behind it. The package ships assemblies for net8.0, netstandard2.0 and net462, so a .NET 10 project resolves the net8.0 asset without complaint.
The second correction. The earlier version of this article told you to add Selenium.Support as well, "for the explicit-wait helpers." Most C# tutorials say the same thing. It has not been true for years. WebDriverWait and DefaultWait<T> live in the main package, at dotnet/src/webdriver/Support/WebDriverWait.cs, and they were already sitting there in the selenium-3.141.59 tag from 2018. What keeps the myth alive is the namespace: the type is OpenQA.Selenium.Support.UI.WebDriverWait, so the using line reads like a second package even when there is only one.
Selenium.Support today contains SelectElement for <select> dropdowns, PopupWindowFinder, the event-firing driver wrapper, and some extension methods. Add it when you want those. Version 4.47.0 targets netstandard2.0 only and takes a hard dependency on Selenium.WebDriver 4.47.0, so the two versions move together whether you like it or not. The official install page still shows both packages in its sample .csproj, which is where a good part of the confusion comes from.
Selenium Manager, and the three places it stops helping
Since Selenium 4.6 the bindings resolve the browser driver themselves. Since 4.11 they will download Chrome itself through Chrome for Testing, since 4.12 Firefox, since 4.14 Edge. For a laptop run you configure no driver path at all.
Read the Selenium Manager documentation before you lean on it in production, because it is franker than the tutorials that cite it. The page is titled "Selenium Manager (Beta)" as of August 2026, four years after the feature shipped, and its versioning stays on major version 0 for that reason. It states plainly that it "only operates as a fallback: if no driver is provided, Selenium Manager will come to the rescue."
Three failure modes follow from that design.
It needs the network on first run. A build agent with no egress gets a driver-resolution failure rather than a browser. Set SE_PROXY for corporate proxies, or pass --offline and pre-seed the cache at ~/.cache/selenium, which also holds the se-config.toml it reads.
It resolves against whatever Chrome is installed right now. Chrome updates itself. A long-lived runner can pick up a new major version between two runs of the same job, and Selenium Manager will dutifully fetch a new driver to match, changing the browser under your test without a commit.
Pinning is still worth doing in CI. The third-party Selenium.WebDriver.ChromeDriver package by jsakamoto publishes a NuGet version per driver build, currently 151.0.7922.7700 against Chrome stable 151.0.7922.77, with 81.9 million downloads. It is not a Selenium project package, and that is the trade: you get a reproducible pin and you take on a dependency the Selenium team does not ship.
The program, step by step
Namespaces
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;OpenQA.Selenium holds IWebDriver, By, IWebElement. OpenQA.Selenium.Chrome holds the Chrome driver and ChromeOptions. OpenQA.Selenium.Support.UI holds the waits, and, as covered above, comes from the same assembly as the other two.
Starting Chrome
var options = new ChromeOptions();
// options.AddArgument("--headless=new");
using IWebDriver driver = new ChromeDriver(options);The using declaration shuts the browser process down when the variable leaves scope, including on the path where something throws. Skip it and a failed run leaves an orphaned Chrome and a chromedriver behind. Ten of those on a build agent and the next job dies for reasons that have nothing to do with its own code.
Keep the window visible while you are writing the script. Headless changes rendering behaviour often enough that debugging a headless failure against a headed assumption wastes an afternoon. Selenium's own Chrome page lists --headless=new among the arguments people actually use; switch it on after the run is stable, not before.
Opening the page
driver.Navigate().GoToUrl("https://the-internet.herokuapp.com/login");The third correction. The earlier version said this call "blocks until the page has loaded." It blocks until document.readyState reaches complete, which is not the same sentence. Selenium's page load strategy documentation says so directly: waiting for the readiness state "does not necessarily mean that the page has finished loading, especially for sites like Single Page Applications that use JavaScript to dynamically load content after the Ready State returns complete."
The same page carries the sentence that matters most for this example. "Note also that this behavior does not apply to navigation that is a result of clicking an element or submitting a form." Your GoToUrl waits. Your login click does not get the same guarantee.
Three strategies exist, set through ChromeOptions:
options.PageLoadStrategy = PageLoadStrategy.Eager;Normal waits for complete and is the default. Eager returns at interactive, once the DOM is parsed but before images finish. None does not block at all. Eager is a real speedup on asset-heavy pages, and it moves all the responsibility for knowing when content exists onto your waits.
Finding the fields
var userNameField = driver.FindElement(By.Id("username"));
var passwordField = driver.FindElement(By.Id("password"));
var loginButton = driver.FindElement(By.CssSelector("button[type='submit']"));By offers eight classic strategies plus relative locators; the locators reference covers them. Prefer By.Id when an id exists and looks stable. Prefer a CSS selector anchored on a semantic attribute over one anchored on layout. A selector built from generated class names survives exactly until the next front-end build.
FindElement throws NoSuchElementException the instant the element is missing. It does not wait. That property is what makes the next section necessary.
Filling in and submitting
userNameField.SendKeys("tomsmith");
passwordField.SendKeys("SuperSecretPassword!");
loginButton.Click();Those credentials are printed on the practice page itself. SendKeys types character by character into the focused element, and Click dispatches a real pointer sequence at the element's in-view centre point.
The wait that looks right and is not
The earlier version of this article waited like this:
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
var resultElement = wait.Until(d => d.FindElement(By.CssSelector(".subheader")));Open the practice app's source and the problem is immediate. Its views/login.erb contains <h4 class="subheader">This is where you can log into the secure area..., and views/secure.erb contains <h4 class="subheader">Welcome to the Secure Area.... Both pages carry .subheader. The condition is already satisfied before the click, so the wait is not waiting for anything. Whether you get the secure page or the login page depends on whether Chrome has swapped documents by the time the first poll fires.
The WebDriver specification is candid about that race. In the Element Click algorithm it says the remote end should "perform implementation-defined steps to allow any navigations triggered by the click to start," then adds: "It is not always clear how long this will cause the algorithm to wait, and it is acknowledged that some implementations may have unavoidable race conditions." That is the W3C WebDriver spec, Working Draft of 02 July 2026, describing its own click as racy.
In practice ChromeDriver usually wins this race and the old code appears to work. Set PageLoadStrategy.Eager, or point the same pattern at a single-page app where the click fires an XHR instead of a navigation, and it stops working. The failure is silent: you write the login page's own instructions to result.txt and everything reports success.
The app's server.rb makes the second trap explicit. A correct login sets flash[:success] = 'You logged into a secure area!' and redirects to /secure. A wrong password sets flash[:error] = 'Your password is invalid!' and renders the login page again, with no redirect. The banner element is <div data-alert id='flash' class='flash error'> in both cases. Wait for #flash alone and a rejected password gets recorded as a successful scrape.
Wait for the destination, then verify which outcome you got:
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
// A condition only the destination can satisfy.
wait.Until(d => d.Url.EndsWith("/secure", StringComparison.Ordinal));
// The banner renders for failures too, so read its class, not just its text.
var banner = wait.Until(d => d.FindElement(By.Id("flash")));
var bannerClass = banner.GetDomAttribute("class") ?? string.Empty;
if (!bannerClass.Contains("success"))
{
throw new InvalidOperationException("Login failed: " + banner.Text.Trim());
}A locator that matches on both the old page and the new one is not a wait. It is a coin flip with good manners.
Two rules about waits, and one number
WebDriverWait polls every 500 ms by default; that value is DefaultSleepTimeout in the constructor. The same constructor calls IgnoreExceptionTypes(typeof(NotFoundException)), and that single line explains most confusing wait behaviour.
NoSuchElementException derives from NotFoundException, so it is swallowed and the loop keeps going. That is what makes wait.Until(d => d.FindElement(...)) work. StaleElementReferenceException derives from WebDriverException, not from NotFoundException. It is not ignored. Hold an IWebElement across a navigation, touch it inside a Until, and the wait aborts on the first poll instead of retrying. Add the type yourself when you need it:
wait.IgnoreExceptionTypes(typeof(StaleElementReferenceException));ElementNotInteractableException derives from InvalidElementStateException and is likewise not ignored, which is why waiting for a button that exists but is still covered by an overlay fails immediately rather than after ten seconds.
The other rule comes straight from the waiting strategies page: "Do not mix implicit and explicit waits. Doing so can cause unpredictable wait times. For example, setting an implicit wait of 10 seconds and an explicit wait of 15 seconds could cause a timeout to occur after 20 seconds." The implicit wait defaults to 0. Leave it there and use explicit waits for everything.
What .Text actually returns
var result = resultElement.Text;
File.WriteAllText("result.txt", result);Text is not textContent. The WebDriver specification defines Get Element Text as returning an element's text "as rendered," implemented by deferring to a function from the Selenium project, bot.dom.getVisibleText, pinned in the spec to revision 775cfb33b193eb8832cd5488f298006f45254685. The standard then says something worth quoting in full: "the approach presented here is known to be flawed, but provides the best compatibility with existing users."
The practical consequence: an element hidden with display: none, or scrolled out of a collapsed accordion, returns an empty string from .Text while its markup holds the value you want. When that happens, read the property instead.
var hidden = element.GetDomProperty("textContent");
var declared = element.GetDomAttribute("data-price");The pairing matters more than it looks. Per the .NET changelog, GetAttribute "will return one of either a declared attribute or an IDL property of the element, without distinction between the two." GetDomAttribute returns only what is written in the markup; GetDomProperty returns only the live DOM property. Both arrived in Selenium 4.0. The bindings deprecated GetAttribute in 4.27.0 and reverted that deprecation in 4.28.0, so the old method is staying, ambiguity included.
The complete program
using System;
using System.IO;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Support.UI;
var options = new ChromeOptions();
// options.AddArgument("--headless=new"); // turn on once the run is stable
using IWebDriver driver = new ChromeDriver(options);
driver.Navigate().GoToUrl("https://the-internet.herokuapp.com/login");
driver.FindElement(By.Id("username")).SendKeys("tomsmith");
driver.FindElement(By.Id("password")).SendKeys("SuperSecretPassword!");
driver.FindElement(By.CssSelector("button[type='submit']")).Click();
var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));
wait.IgnoreExceptionTypes(typeof(StaleElementReferenceException));
// Wait for the destination, not for markup both pages share.
wait.Until(d => d.Url.EndsWith("/secure", StringComparison.Ordinal));
var banner = wait.Until(d => d.FindElement(By.Id("flash")));
if (!(banner.GetDomAttribute("class") ?? string.Empty).Contains("success"))
{
throw new InvalidOperationException("Login rejected: " + banner.Text.Trim());
}
var heading = driver.FindElement(By.CssSelector(".subheader")).Text.Trim();
File.WriteAllText("result.txt", heading);
Console.WriteLine(heading);Written against .NET 10 and Selenium.WebDriver 4.47.0. Top-level statements mean the implicit Main disposes the driver at the end, so no explicit driver.Quit() is needed. Calling Quit() and then letting the using dispose is harmless, since disposal on an already-closed session is a no-op, but two ways of saying the same thing in ten lines of code is one too many.
What breaks between one page and ten thousand
The script above is a demo. A crawl is a different animal, and the difference shows up in four places.
Process leaks. Every unhandled exception outside a using leaves Chrome and its driver resident. Wrap the driver lifetime, and add a periodic sweep on long-running agents. This is the single most common reason a nightly job that worked for a month starts failing on day thirty-one.
One driver per page is expensive; one driver forever is fragile. Reusing a single browser across ten thousand pages accumulates cookies, storage, and memory. Recycling the driver every few hundred pages costs a browser start each time and keeps the failure blast radius small. Pick a number, measure it against your own target, and write it down.
Restarts must be idempotent. A ten-thousand-page run will be interrupted. Keep a durable list of what is done so a restart resumes rather than repeats, and make the write step tolerate being run twice on the same record.
Concurrency belongs in Grid or in a queue, not in Parallel.ForEach. Each browser is a full process. Fan out past what the machine can hold and pages start timing out for reasons that look like anti-bot defences but are actually swap. Sites that fight back on top of that are where a managed extraction service or a proxy pool starts to look cheaper than another week of tuning.
The browser announces itself
WebDriver does not hide. The W3C specification defines a webdriver-active flag that "is set to true when the user agent is under remote control," surfaced to page scripts as navigator.webdriver. The spec explains its purpose without embarrassment: it exists "so that alternate code paths can be triggered during automation," and then notes, in the standard's own words, that "it is acknowledged that this is complementary to the Evil Bit [RFC3514]."
That is a standards body telling you the flag is a courtesy. Any page can read it in one line, and commercial defences read a great deal more than that: driver-injected properties, CDP traffic, timing signatures, TLS and HTTP/2 fingerprints. Our article on how to tell your site is being scraped covers the detection side in detail.
Two community NuGet packages port the "undetected" patch idea to C#. Selenium.UndetectedChromeDriver by fysh711426 is at 1.1.4, published 21 May 2026, with 173,050 downloads. Selenium.WebDriver.UndetectedChromeDriver by emre-gon is at 2.3.1 with 44,170. Neither ships from the Selenium project, and neither publishes a measured success rate against any named defence. Treat them as what they are: a patch set maintained by one person each, against targets that change on the vendor's schedule rather than theirs.
Where WebDriver is the wrong tool
When the data is in an API. Covered above, and still the answer most of the time.
When you need fine-grained network control. Selenium's DevTools bindings are versioned against specific Chrome builds. Release 4.46.0 lists support for CDP v148, v149 and v150 in its changelog, while Chrome stable on 10 August 2026 was 151. That gap is structural, not a bug, and it is the reason the project keeps moving functionality to WebDriver BiDi, which is a standard rather than a Chrome-specific protocol.
When request interception is the point. Playwright for .NET sits at 1.62.0 in its repository's version file and bundles its own browser builds plus a Node runtime. Route interception, network stubbing, and auto-waiting are first-class there. Selenium's advantages are the W3C protocol, Grid, and the fact that the same script drives Firefox and Safari without a rewrite.
When the page is static HTML. An HttpClient request plus a parser is one to two orders of magnitude cheaper per page. Our broader guide to web scraping in C# covers that route.
Before you point this at someone else's site
The target used here is a practice application. Its README describes it as "an example application that captures prominent and ugly functionality found on the web. Perfect for writing automated acceptance tests against." Permission is the point of the site, which is why it is safe to publish a script against it.
That app is also a good lesson in what "still works" means. Its README is stamped "The Internet 0.58.0 (10, February 2020)", the Gemfile pins Ruby 2.7.2 and selenium-webdriver ~> 3.4.0, and the layout still loads jQuery 1.11.3. The deployment answered every request we made on 10 August 2026. Frozen and alive are not opposites.
Real targets are a different question, and logging in changes it materially. Clicking through a login means accepting terms you can be held to, and US case law turns on exactly that gate. Van Buren v. United States, decided by the Supreme Court in 2021, framed the Computer Fraud and Abuse Act as a gates-up-or-down question about areas you are not authorised to enter. The hiQ Labs v. LinkedIn saga, still summarised across the web as a scraper victory, ended in December 2022 with hiQ accepting a $500,000 judgment and a permanent injunction against scraping LinkedIn. If a write-up tells you hiQ won, it stopped reading in 2019.
Personal data adds a second, independent layer that has nothing to do with access controls. Our GDPR and web scraping article covers the part that applies whether or not the data was public. None of this is legal advice, and the boundary is worth naming: the US case law here is still moving, and the summaries in circulation are usually one ruling behind.
Conclusion
WebDriver asks for very little ceremony. You press keys, you click, and the browser handles the requests, redirects, and cookies underneath. The setup that used to be the hard part is gone: one NuGet package, no driver download, no version matching for a laptop run.
What is left is the part that decides whether the script survives contact with a real site. Wait for something only the destination page can satisfy. Check which branch you landed on before you save the result. Know that .Text gives you rendered text and nothing else. Assume the site can see navigator.webdriver, because the standard put it there on purpose. Get those four right and the browser stops being the flaky part of your pipeline.
The official Selenium documentation goes further into frames, windows, JavaScript execution, and Grid.