Using External Input Data in Off-the-Shelf Web Scrapers

Here's a question that comes up constantly once you get past scraping a single page: "What if I need to scrape hundreds of URLs that are built from data I already have in a spreadsheet or database?"

A reader once put it perfectly: "I have a large list of Amazon ASINs. I'd like to scrape ten or so fields for each ASIN. Is there a web scraper that can read each ASIN from a file and build the target URL — like amazon.com/dp/{ASIN} — and scrape the data?"

That's the whole idea of external input data: instead of pointing a scraper at one page, you feed it a list — of URLs, ASINs, SKUs, search keywords, ZIP codes, whatever — and it runs the same extraction once per row. It's the single most useful feature for turning a one-off scrape into a repeatable batch web scraping job, and nearly every serious off-the-shelf web scraper supports it. This guide covers the pattern and how the current tools implement it.

The core pattern: a list in, structured rows out

Every input-driven scrape follows the same three-part shape:

1. A source of input values — a text file of URLs, a CSV of IDs, a column of keywords, or a live database query.
2. A URL template or search action — a rule that turns each input value into something to fetch. For the ASIN example, the template is literally https://www.amazon.com/dp/{value}. For keyword scraping, the value gets typed into a search box.
3. An extraction recipe — the fields to pull from each resulting page (title, price, rating, stock…).

The scraper loops over the input, applies the template, extracts the fields, and appends a row to your output for each input value. That's it. Whether the tool is a browser extension or a cloud platform, this loop is what "external input" buys you.

There are two common flavors:

• Input as URLs — you already have (or can build) the full page addresses. Fastest path.
• Input as parameters — keywords typed into a search box, filter values selected on a form, or IDs slotted into a template. The scraper constructs the navigation from each value.

How the current tools do it

The scraping-tool landscape turned over completely since this question was first asked — the desktop apps of the early 2010s (Visual Web Ripper, Web Content Extractor, Screen Scraper, WebSundew, Helium Scraper) have mostly faded, replaced by cloud platforms and browser extensions. Here's how today's mainstream tools handle input data.

Web Scraper (webscraper.io) — browser extension

The popular free Chrome/Firefox extension supports multiple start URLs directly. In the sitemap editor you can paste a whole list of URLs, or use its range/pagination syntax like https://example.com/products?page=[1-50] to generate them. For an ID list, generate the full URLs in a spreadsheet first (concatenate the template with each ASIN), then paste the column in. Good for small-to-medium batches without writing code.

Octoparse — desktop + cloud, no-code

Octoparse has a dedicated "loop from a list of URLs" action: paste or import a text/CSV column of URLs and it runs the same workflow for each. It also supports looping a list of text values into a search box, which is the keyword/parameter flavor — type each keyword, submit, scrape the results. Cloud runs let you schedule the whole batch.

ParseHub — desktop + cloud

ParseHub lets you start from a list of URLs and supports templates where a value is substituted into the address. You can upload a CSV of start URLs so a single project fans out across all of them.

Apify — cloud platform for developers and no-coders

Apify is the most explicitly input-driven of the bunch. Every "Actor" takes a JSON input, and general scrapers like the Web Scraper and Cheerio Scraper actors accept a startUrls array (or a linked file/dataset of URLs). You can also drive an actor from a Google Sheet or an uploaded CSV. Because input is a first-class JSON field, chaining it into automation and scheduling is straightforward:

{
  "startUrls": [
    { "url": "https://www.amazon.com/dp/B08N5WRWNW" },
    { "url": "https://www.amazon.com/dp/B07FZ8S74R" }
  ]
}

Generate that array from your ASIN list programmatically and you've got a clean batch job.

Data Miner / Instant Data Scraper — extensions

Data Miner recipes can take a list of URLs to run in sequence, which suits repeatable extractions across a known page set straight from the browser.

Bright Data & enterprise platforms

Bright Data's Web Scraper tools and dataset builders accept input lists (URLs or search terms) and are built for scale, with proxy rotation baked in. This is the tier you reach for when the batch is large enough that blocking becomes the real constraint.

Worked example: the ASIN list

Let's answer the original question concretely with a modern no-code tool. Say you have asins.csv:

asin
B08N5WRWNW
B07FZ8S74R
B09G3HRMVB

1. Turn IDs into URLs. In a spreadsheet, build a column: ="https://www.amazon.com/dp/"&A2. Now you have full product URLs.
2. Load the list. In your tool of choice, use its "loop from list of URLs" / "multiple start URLs" / startUrls input and paste or import that column.
3. Build the recipe once on a single product page — click the title, price, rating, availability, etc., to define the fields.
4. Run the batch. The scraper visits each URL, extracts the same fields, and outputs one row per ASIN to CSV/JSON/Excel.

If you'd rather keep the IDs and not pre-build URLs, the parameter flavor works too: many tools let you substitute {asin} into a template inside the workflow, or type each ASIN into the site's search box and scrape the first result.

When to graduate from off-the-shelf to code

No-code batch scraping is great until you hit one of these walls:

• The input lives in a live database, not a file. Then you want a script that queries the DB, builds the URLs, and drives the scrape. In Python that's a few lines wrapping requests or a scraping framework; tools that support a direct DB/SQL input are increasingly rare, so code is the clean answer.
• The batch is large and the site fights back. Thousands of requests trip rate limits and blocks. You'll need rotating proxies and, for guarded sites, CAPTCHA handling — cloud platforms bundle some of this, but at scale a purpose-built Scrapy pipeline gives you more control.
• The pages are JavaScript-rendered. If the data loads after the page does, you need a tool that runs a real browser, or you call the underlying API directly with your input values as parameters.
• You want it fully automated and scheduled. Feeding a fresh input list on a cron and delivering results to a database or sheet is where a small custom pipeline pays off — see running a long-running scraper detached so batches keep going unattended.

Practical tips for input-driven batches

• Deduplicate your input first — repeated ASINs or URLs waste requests and muddy the output.
• Keep a key column in the output. Carry the input value (the ASIN, the keyword) through to each result row so you can join the scraped data back to your source list.
• Handle misses gracefully. Some inputs will 404 or be out of stock; log them rather than letting one bad row kill the run.
• Throttle. A big list run flat-out looks exactly like an attack. Pace requests to stay under the radar and unblocked.
• Normalize the output. Fields come back inconsistent across pages; a normalization step makes the batch analysis-ready.

The takeaway

External input data is the feature that scales a scraper from "one page" to "ten thousand pages" without ten thousand clicks. The pattern is always the same — list in, template applied, fields out — and every current tool of note (Web Scraper, Octoparse, ParseHub, Apify, Bright Data) supports it in either the URL flavor or the search-parameter flavor. Start no-code for a file of URLs; move to a script when the input lives in a database, the volume triggers blocking, or you need it scheduled and hands-off.

And when the batch is big, recurring, and the target sites push back, running the whole thing reliably is a job in itself. That's precisely what scraping.pro handles as a data-as-a-service feed — you supply the list of inputs, we deliver the structured rows.