Scraping Login-Protected Sites: Complete Guide with Code
Most web-scraping jobs are solved with a simple GET request to a public page. But the moment the data you need sits behind a login — in a member area, a gated section, or under a paid subscription — an ordinary request just returns the sign-in form or a 401/403 error. To reach the content, your scraper has to authenticate first, exactly the way a user's browser does.
This guide covers scraping behind a login end to end: the four authentication mechanisms you'll meet, and working code for web scraping login flows in Python, Node.js, PHP, and Go, including session cookies, CSRF tokens, and headless-browser logins.
Where This Comes Up Most
The most common reason to scrape a website that requires login is price and stock monitoring. A typical situation: you have a supplier whose current prices and inventory are only visible in your account on their B2B portal. There's no proper data API, no export in the format you need, and prices change often enough that copying them by hand is a waste of time.
It's worth stressing: this access is agreed with the supplier and does not violate the site's terms of use. You're automating retrieval of data you're already allowed to see under your own account — you're just doing it programmatically instead of by clicking. In that form, authenticated scraping is a legitimate working tool, not a way to bypass restrictions.
Before you write any code, always confirm that:
- access to the data is permitted by the resource owner (a contract, written consent, partner-program terms);
- automated collection isn't prohibited by the terms of service (ToS) or the
robots.txtfile; - the load on the site stays reasonable and doesn't interfere with its operation;
- third parties' personal data isn't collected or processed without a legal basis (mind GDPR/CCPA where they apply).
How Authentication Works: Four Main Mechanisms
To choose your approach in code, you need to understand how the site authenticates a user. In practice there are four main variants.
1. Login form and session cookies. The most common case. You send a username and password in a POST request to the auth endpoint, the server responds by setting a session cookie (for example sessionid or PHPSESSID), and that cookie is then attached to every subsequent request. The session lives as long as the cookie is valid. This is the foundation of session cookies scraping.
2. CSRF token. Many forms are protected by a token hidden in the HTML of the login page (in a hidden field or a meta tag) or issued in a separate cookie. Before submitting the form you first load the login page, extract the token, and send it together with the credentials. Without it the server rejects the request.
3. Tokens (Bearer / JWT). Modern sites and SPAs often authenticate through an API, returning a token in JSON. From then on the token is sent in the Authorization: Bearer <token> header. Cookies may not be used at all.
4. HTTP Basic Auth. The simplest variant: username and password are base64-encoded and sent in the Authorization header. Found in internal systems and some APIs.
A separate complication is sites that build their content with JavaScript. There a plain HTTP client isn't enough, and you need a "headless" browser (Playwright, Puppeteer, Selenium) that runs the scripts and hands you the finished DOM.
Login CAPTCHAs. If a site throws a CAPTCHA on the sign-in form, no HTTP client will get past it on its own. Options are a headless browser plus a CAPTCHA-solving service, or — cleaner when it's your own account — reusing a saved, already-authenticated session (see the Playwright example below) so you only clear the CAPTCHA once.
Python: requests with a Session
requests.Session() automatically stores cookies between requests — an ideal base for web scraping authentication. Here's an example that first fetches the CSRF token:
import requests
from bs4 import BeautifulSoup
LOGIN_URL = "https://supplier.example.com/login"
PRICES_URL = "https://supplier.example.com/account/prices"
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (compatible; PriceMonitor/1.0)"
})
# 1. Load the login page and grab the CSRF token
login_page = session.get(LOGIN_URL, timeout=30)
soup = BeautifulSoup(login_page.text, "html.parser")
csrf_token = soup.select_one('input[name="csrf_token"]')["value"]
# 2. Submit the login form
payload = {
"username": "your_login",
"password": "your_password",
"csrf_token": csrf_token,
}
resp = session.post(LOGIN_URL, data=payload, timeout=30)
resp.raise_for_status()
if "My Account" not in resp.text:
raise RuntimeError("Login failed — check your credentials")
# 3. Session established — request the protected prices page
prices_page = session.get(PRICES_URL, timeout=30)
soup = BeautifulSoup(prices_page.text, "html.parser")
for row in soup.select("table.prices tr"):
cells = row.select("td")
if len(cells) >= 2:
name = cells[0].get_text(strip=True)
price = cells[1].get_text(strip=True)
print(f"{name}: {price}")
Good practice: keep the username and password out of the code, in environment variables (os.environ) or a .env file, so you never accidentally commit them to a repository.
Python: Token Authentication (API)
If the site authenticates through a JSON API and returns a token, the code is simpler:
import requests
auth = requests.post(
"https://supplier.example.com/api/auth/login",
json={"login": "your_login", "password": "your_password"},
timeout=30,
)
auth.raise_for_status()
token = auth.json()["access_token"]
headers = {"Authorization": f"Bearer {token}"}
data = requests.get(
"https://supplier.example.com/api/prices",
headers=headers,
timeout=30,
).json()
for item in data["items"]:
print(item["sku"], item["price"])
Python: Playwright for JavaScript Sites
When the member area is an SPA and prices load via scripts, a headless browser helps. Playwright can log in like a real user and even save the session state to a file, so it doesn't have to sign in again on every run.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
context = browser.new_context()
page = context.new_page()
# Log in
page.goto("https://supplier.example.com/login")
page.fill("input[name='username']", "your_login")
page.fill("input[name='password']", "your_password")
page.click("button[type='submit']")
page.wait_for_url("**/account/**")
# Save the session to reuse later
context.storage_state(path="auth_state.json")
# Go to the prices and wait for the data to load
page.goto("https://supplier.example.com/account/prices")
page.wait_for_selector("table.prices")
rows = page.query_selector_all("table.prices tr")
for row in rows:
cells = row.query_selector_all("td")
if len(cells) >= 2:
print(cells[0].inner_text(), "—", cells[1].inner_text())
browser.close()
The saved auth_state.json is then loaded via browser.new_context(storage_state="auth_state.json") — and you can skip the login step until the session expires.
Node.js: axios with Cookie Persistence
In Node, to store cookies between requests you combine axios + tough-cookie + axios-cookiejar-support.
const axios = require("axios");
const { wrapper } = require("axios-cookiejar-support");
const { CookieJar } = require("tough-cookie");
const cheerio = require("cheerio");
const jar = new CookieJar();
const client = wrapper(axios.create({ jar, withCredentials: true }));
async function run() {
// 1. Get the CSRF token from the login page
const loginPage = await client.get("https://supplier.example.com/login");
const $ = cheerio.load(loginPage.data);
const csrf = $('input[name="csrf_token"]').val();
// 2. Log in
await client.post(
"https://supplier.example.com/login",
new URLSearchParams({
username: "your_login",
password: "your_password",
csrf_token: csrf,
}),
);
// 3. Request the prices
const pricesPage = await client.get(
"https://supplier.example.com/account/prices",
);
const $$ = cheerio.load(pricesPage.data);
$$("table.prices tr").each((_, el) => {
const cells = $$(el).find("td");
if (cells.length >= 2) {
const name = $$(cells[0]).text().trim();
const price = $$(cells[1]).text().trim();
console.log(`${name}: ${price}`);
}
});
}
run().catch(console.error);
Node.js: Puppeteer for Dynamic Pages
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://supplier.example.com/login");
await page.type("input[name='username']", "your_login");
await page.type("input[name='password']", "your_password");
await Promise.all([
page.click("button[type='submit']"),
page.waitForNavigation(),
]);
await page.goto("https://supplier.example.com/account/prices");
await page.waitForSelector("table.prices");
const prices = await page.evaluate(() =>
Array.from(document.querySelectorAll("table.prices tr"))
.map((row) => {
const td = row.querySelectorAll("td");
return td.length >= 2
? { name: td[0].innerText.trim(), price: td[1].innerText.trim() }
: null;
})
.filter(Boolean),
);
console.log(prices);
await browser.close();
})();
PHP: cURL with Session Cookies
In PHP, cookies are stored between requests in a file via the COOKIEJAR and COOKIEFILE options.
<?php
$cookieFile = __DIR__ . "/cookies.txt";
function curlInit(string $cookieFile) {
$ch = curl_init();
curl_setopt_array($ch, [
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_COOKIEJAR => $cookieFile,
CURLOPT_COOKIEFILE => $cookieFile,
CURLOPT_USERAGENT => "Mozilla/5.0 (compatible; PriceMonitor/1.0)",
]);
return $ch;
}
// 1. Load the login page and extract the CSRF token
$ch = curlInit($cookieFile);
curl_setopt($ch, CURLOPT_URL, "https://supplier.example.com/login");
$html = curl_exec($ch);
preg_match('/name="csrf_token"\s+value="([^"]+)"/', $html, $m);
$csrf = $m[1] ?? "";
// 2. Submit the login form
curl_setopt($ch, CURLOPT_URL, "https://supplier.example.com/login");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query([
"username" => "your_login",
"password" => "your_password",
"csrf_token" => $csrf,
]));
curl_exec($ch);
// 3. Request the prices
curl_setopt($ch, CURLOPT_URL, "https://supplier.example.com/account/prices");
curl_setopt($ch, CURLOPT_POST, false);
$pricesHtml = curl_exec($ch);
curl_close($ch);
// Parse the HTML
$dom = new DOMDocument();
@$dom->loadHTML($pricesHtml);
$xpath = new DOMXPath($dom);
foreach ($xpath->query("//table[@class='prices']//tr") as $row) {
$cells = $row->getElementsByTagName("td");
if ($cells->length >= 2) {
echo trim($cells->item(0)->textContent) . ": "
. trim($cells->item(1)->textContent) . PHP_EOL;
}
}
Go: net/http with a cookiejar
Go's standard library includes net/http/cookiejar, which manages cookies for you automatically.
package main
import (
"fmt"
"net/http"
"net/http/cookiejar"
"net/url"
"strings"
"github.com/PuerkitoBio/goquery"
)
func main() {
jar, _ := cookiejar.New(nil)
client := &http.Client{Jar: jar}
// 1. Get the CSRF token
resp, _ := client.Get("https://supplier.example.com/login")
doc, _ := goquery.NewDocumentFromReader(resp.Body)
resp.Body.Close()
csrf, _ := doc.Find(`input[name="csrf_token"]`).Attr("value")
// 2. Log in
form := url.Values{
"username": {"your_login"},
"password": {"your_password"},
"csrf_token": {csrf},
}
client.Post(
"https://supplier.example.com/login",
"application/x-www-form-urlencoded",
strings.NewReader(form.Encode()),
)
// 3. Parse the prices
pricesResp, _ := client.Get("https://supplier.example.com/account/prices")
pricesDoc, _ := goquery.NewDocumentFromReader(pricesResp.Body)
pricesResp.Body.Close()
pricesDoc.Find("table.prices tr").Each(func(_ int, s *goquery.Selection) {
cells := s.Find("td")
if cells.Length() >= 2 {
name := strings.TrimSpace(cells.Eq(0).Text())
price := strings.TrimSpace(cells.Eq(1).Text())
fmt.Printf("%s: %s\n", name, price)
}
})
}
Practical Recommendations
Reuse the session. Don't log in on every request — it's needless load and raises your block risk. Save the cookie or token and refresh it only when the session has expired.
Handle session expiry. Cookies and tokens have a lifetime. Build in a check: if a request comes back as a redirect to the login form or a 401, re-authenticate and retry the request.
Keep a reasonable pace. Put delays between requests (say, 1–3 seconds) and don't fire off dozens of parallel threads. It's polite to the supplier's server and lowers your chance of tripping anti-bot defenses. If you must scale, add rotating proxies and spread requests over time rather than hammering.
Store secrets safely. Usernames, passwords, and tokens belong in environment variables or a secrets vault — not in code, and never in a public repository.
Be resilient to markup changes. Sites change, selectors break. Log parsing errors and set up alerts so you notice quickly when the page structure shifts.
Send an honest User-Agent and, where possible, contact details. If the supplier has agreed to the access, an identifiable bot makes diagnosis easier on their side if something goes wrong.
Conclusion
Technically, web scraping login boils down to reproducing the steps a browser takes when signing in: fetch and submit the form (with the CSRF token if there is one), store the session cookie or token, and attach it to subsequent requests. For static pages an HTTP client with session support is enough (requests, axios, cURL, net/http); for dynamic ones you need a headless browser (Playwright, Puppeteer).
The most important part, though, isn't in the code — it's in the basis for collecting the data. Scraping a supplier's member area for price monitoring is a legitimate, workable tool precisely when the access is agreed with the resource owner and doesn't break the terms of use. The technology works equally well in both directions, so the responsibility for using it correctly stays with you.
If you'd rather not build and babysit authenticated scrapers yourself, scraping.pro runs this as a done-for-you data extraction service — including price and stock monitoring from supplier portals, with the logins, sessions, and change alerts handled for you.