Auto Scroll a Web Page to Scrape Infinite Scroll Content

Plenty of modern sites don't paginate anymore — they just keep loading more content as you scroll. Social feeds, business directories, job boards, product grids, and news sites all use infinite scroll, where new items appear only after you reach the bottom. A plain HTTP request grabs the initial page and stops, so a scraper that just downloads the HTML sees a fraction of the data. To scrape infinite scroll content you first have to make the page auto scroll so its JavaScript loads everything, then extract the fully rendered result.

This guide covers the practical ways to do that: a quick in-browser JavaScript snippet for one-off jobs, robust Selenium and Playwright automation for pipelines, how to handle "Load more" buttons, and the faster alternative of hitting the underlying API directly.

Why HTTP-only scrapers miss the data

Infinite scroll is driven by client-side JavaScript. As you approach the bottom of the page, a script fires a background request (usually fetch/XHR to a JSON endpoint), receives the next batch of items, and injects them into the DOM. None of that happens when you download the raw HTML with requests, curl, or httpx — those tools don't run JavaScript, so they only ever see the initial payload.

You also cannot simply replay the site's "load more" network request from an HTTP client in most cases. Those endpoints are frequently guarded by the browser's same-origin policy, CORS rules, CSRF tokens, and headers the site's own JavaScript adds. Spoofing all of that by hand is brittle. The reliable approach is to drive a real browser so the site's JavaScript runs exactly as it would for a human, then read the result.

The quick way: auto-scroll from the browser console

For a one-off job, you don't need any code project — you can make the current tab scroll itself. Open DevTools (F12, or Ctrl+Shift+I / Cmd+Option+I), go to the Console tab, paste this, and press Enter:

const scroll = setInterval(() => window.scrollBy(0, 1000), 2000);

setInterval runs the function on a timer; scrollBy(0, 1000) nudges the page down 1000 pixels each time. Every two seconds (2000 ms) the page scrolls a bit further, giving the site time to load the next batch. Tune both numbers for the site: shorter interval for fast sites, larger scroll step to move quicker. You can close DevTools and the timer keeps running.

To stop it, run:

clearInterval(scroll);

Reloading the page (F5) also stops it, but sends you back to the top. When the page has finished growing, you can copy the DOM or save the HTML for extraction. It's crude, but for a single directory or feed it beats writing a script.

You can wire the same idea into a page of your own with two buttons:

<button onclick="win = setInterval(() => window.scrollBy(0, 1000), 1500)">Start scroll</button>
<button onclick="clearInterval(win)">Stop scroll</button>

The robust way: Playwright

For anything repeatable, automate a headless browser. Playwright (Microsoft) is the current default for this kind of work — it's fast, cross-browser, ships clean async APIs for Python and Node, and has first-class auto-waiting that removes most of the flakiness older scripts suffered from. The pattern for infinite scroll is: scroll to the bottom, wait for the height to grow, repeat until it stops growing.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/listings")

    prev_height = 0
    while True:
        page.mouse.wheel(0, 20000)          # scroll far down
        page.wait_for_timeout(1500)         # let new items load
        height = page.evaluate("document.body.scrollHeight")
        if height == prev_height:           # nothing new loaded -> done
            break
        prev_height = height

    items = page.query_selector_all(".listing-card")
    print(f"Loaded {len(items)} items")
    for it in items:
        print(it.inner_text())
    browser.close()

The key is the stop condition: keep scrolling only while the page keeps getting taller. When scrollHeight stops changing, you've reached the end and can extract. This is far more reliable than scrolling a fixed number of times, which either stops too early or wastes time.

Selenium (updated API)

If your stack is already on Selenium, the same logic applies. Note that the modern Selenium 4 API replaced the old find_element_by_xpath(...) helpers with the explicit find_element(By.XPATH, ...) form, and Python 3 uses print(...) — copy-pasting pre-2020 snippets will not run as written.

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome()
driver.get("https://example.com/listings")

prev_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(1.5)
    height = driver.execute_script("return document.body.scrollHeight")
    if height == prev_height:
        break
    prev_height = height

cards = driver.find_elements(By.CSS_SELECTOR, ".listing-card")
print(f"Loaded {len(cards)} items")
driver.quit()

Handling a "Load more" button

Some sites don't auto-load on scroll — they show a Load more button you have to click to reveal the next batch. The recipe is the same idea with a click instead of (or in addition to) scrolling: scroll the button into view, click it, wait, repeat until it disappears. Use an explicit wait so you never click before the button exists.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com/jobs")

while True:
    try:
        btn = WebDriverWait(driver, 5).until(
            EC.element_to_be_clickable((By.XPATH, "//*[text()='Load more']"))
        )
        driver.execute_script("arguments[0].scrollIntoView({block:'center'});", btn)
        btn.click()
    except Exception:
        break  # button gone -> all items loaded

jobs = driver.find_elements(By.CSS_SELECTOR, ".job")
print(f"Loaded {len(jobs)} jobs")
driver.quit()

Two things trip people up here. First, if the button becomes stale (the site's JavaScript re-renders it and your reference goes dead), re-find it inside the loop each iteration rather than holding one reference — the code above already does this. Second, some sites cap how many times a button can be used per session; if it stops responding after N clicks, that's a deliberate limit, and the fix is a fresh session or a different query rather than more clicks.

The faster alternative: scrape the API directly

Driving a browser is reliable but slow and resource-heavy. Before committing to it, check whether you can skip the rendering entirely. Open DevTools → Network tab, filter to Fetch/XHR, and scroll the page: you will often see the exact JSON request that loads each batch, usually with a page or offset parameter you can increment yourself.

GET /api/listings?page=2&limit=50
GET /api/listings?page=3&limit=50

If you can call that endpoint directly (some are public, some need a token or specific headers the site sends), you get clean structured data with no browser, no scrolling, and a fraction of the runtime — often hundreds of times faster. This API-based approach is the single biggest speed win available for infinite-scroll sites, so it's always worth ten minutes in the Network tab before you write any scroll loop.

Practical tips

• Prefer a stop condition over a fixed count. Scroll until the height (or item count) stops changing, not a hard-coded number of times.
• Wait for content, not the clock. Where possible, wait for a new element to appear rather than a blind sleep; it's both faster and more reliable.
• Watch memory on huge pages. A feed that loads tens of thousands of items can balloon the DOM and exhaust RAM. Extract in batches, or use the API approach, for very large sets.
• Respect the site. Add delays, and for large jobs rotate identity with rotating proxies so you're not hammering one endpoint from one IP.

When it's more trouble than it's worth

Infinite scroll, "Load more," lazy images, and anti-bot checks stack up fast, and maintaining browser automation across sites that change their markup is real ongoing work. If you'd rather receive the finished dataset than babysit scroll loops, scraping.pro runs web scraping as a managed service — including JavaScript-heavy, infinite-scroll sources — and delivers the fully loaded results as clean structured data on whatever schedule you need.