Using External Input Data in Off-the-Shelf Web Scrapers

Four thousand ASINs sit in column A of a spreadsheet. You want ten fields for each one, and clicking through them is not a plan. A reader put the question to us years ago and a version of it still lands most weeks: "I have a large list of Amazon ASINs. I'd like to scrape ten or so fields for each ASIN. Is there a web scraper that can read each ASIN from a file and build the target URL, like amazon.com/dp/{ASIN}, and scrape the data?"

Yes, and in every tool worth naming. The interesting part starts after the yes. How many rows will the tool accept in one go, what unit does it bill, and what does row 3,741 do to the rest of the run when it comes back a 404. Every price, ceiling and version number below was read from the vendor's own pricing page, help center or API documentation on 13 August 2026.

What the list costs before you pick a tool

Feeding a list is the easy half. The meter is the half that picks your tool for you, because no two products count the same thing. One counts URLs, one counts platform credits, one counts delivered records, one counts rows. The same job lands as four different invoices.

Take 10,000 Amazon product pages, one row of output each.

Two counting quirks decide more budgets than any feature comparison. ScraperAPI weights credits by target: a standard page is one credit, an Amazon page five, Google or Bing twenty-five, LinkedIn thirty, and anything behind Cloudflare, DataDome or PerimeterX adds ten per request. Browse AI counts extracted rows rather than pages, at ten rows per credit, which inverts the arithmetic on listing pages. Fifty products scraped from one search page is five credits there and one page-credit almost everywhere else.

The free tiers are smaller than the marketing suggests, with one exception. Apify's own Web Scraper page estimates that its $5 monthly credit covers 500 to 1,000 pages. Firecrawl gives 1,000 credits and two concurrent requests. Browse AI gives 50 credits a month. Octoparse is the outlier: 10 tasks and 50,000 exported rows a month, at no cost, provided the run happens on your own machine.

The core pattern: a list in, structured rows out

Every input-driven scrape has the same three parts.

1. A source of values. A text file of URLs, a CSV column of IDs, a keyword list, a database query.
2. A rule that turns a value into a navigation. Usually a URL template like https://www.amazon.com/dp/{value}. Sometimes a form: the value gets typed into a search box and submitted.
3. One extraction recipe. The fields you pull from whatever page the rule lands on.

The tool loops the input, applies the rule, runs the recipe, appends a row. That loop is the entire feature, and it is what separates a one-off scrape from a job you can re-run next month against a longer list.

Three shapes of input show up in practice. Full URLs are the fastest path, because nothing has to be constructed. Parameters are IDs, keywords or ZIP codes that the tool slots into a template or types into a form. A queue is the API version of the same idea, where the list arrives as a JSON array or an uploaded file and the platform hands out the work. Tools built for people prefer the first two. Tools built for pipelines prefer the third.

How each tool actually takes the list

Web Scraper (webscraper.io) - browser extension and cloud

The extension is alive and looked after: version 1.111.13, updated 15 July 2026, 800,000 users, 4.1 out of 5 from 1.1K ratings in the Chrome Web Store.

Its documentation describes two ways to get more than one start URL into a sitemap, and both are worth knowing exactly. You press + beside the URL field to add another field. Or you use range syntax in a single field: https://example.com/page/[1-3] expands to three pages, [001-100] keeps the zero padding, and [0-100:10] steps in tens.

Neither mechanism is a column of 4,000 arbitrary product URLs, and this is where our earlier version of this article was too generous. Ranges cover paginated listings beautifully. An unordered ID list is a different problem, and the extension's documented answer is one input field per URL.

Cloud pricing as of this check: Project $50 a month for 5,000 URL credits and 2 concurrent scrapers, Professional $100 for 20,000 credits and 3, Scale from $200 with unlimited URL credits and 60-day retention. Residential proxy is an add-on at $2.50 per GB. Data retention on the lower plans is 30 days, which matters if your batch runs on a schedule and nobody collects the output until the end of the month.

Octoparse - the largest documented ceiling

Octoparse is the one tool here that publishes a hard number for how long your list may be, and the number is generous. Its help center says to keep pasted lists under 10,000 URLs, then adds: "using the Batch URL input feature, you can input up to 1 million URLs." A second article confirms the million-URL ceiling for lists imported from a local file, from another task, or typed in, and recommends dropping the pagination steps entirely once you have the URLs, because a flat list splits across cloud nodes and a pagination loop does not.

For patterned URLs there is a batch generator: highlight the varying part, give it a repeat count, and the tool writes the list for you. That is the right tool for ?page=1..500 and the wrong tool for an ASIN list, which has no pattern to exploit.

The desktop app is at 10.1.1, released 28 July 2026, with 10.1.0 having added local browser scraping four weeks earlier. On the pricing page, Free gives 10 tasks, 50,000 rows of export a month and local runs only. Standard is $69 a month, or about $58 billed annually, for 100 tasks and 3 concurrent cloud runs. Professional is $249, or about $209 annually, for 250 tasks and 20 concurrent cloud runs. The pricing table also lists a 10,000-row cap on a single export at every tier, which turns a 200,000-row batch into twenty exports or an API pull.

Two things the round-ups get wrong here, including the earlier version of this page. Cloud runs and scheduling are not part of the free tier, so "schedule the whole batch" is a paid sentence. And the task-management API arrives with Professional, not with Standard, which is the plan most people assume covers it. Our Octoparse review goes through the workflow builder itself.

ParseHub - the cleanest template mechanic, and the quietest vendor

ParseHub's approach is the most direct answer to the ASIN question. In Settings you fill a Starting Value with JSON:

{"urls": ["https://www.amazon.com/dp/B08N5WRWNW", "https://www.amazon.com/dp/B07FZ8S74R"]}

Or you use Import from CSV/JSON with a header row and one URL per line. Then a Loop command walks the list and a Go To Template command visits each entry.

The ID version skips the spreadsheet entirely. ParseHub's help article on navigating a list of URLs builds the address inside the tool with string concatenation, its worked example being "http://www.barchart.com/quotes/stocks/"+stock over a list like ["XOM","GE","MSFT"]. For our case the expression is "https://www.amazon.com/dp/"+asin and the input file stays a column of raw IDs.

Signs of maintenance are thin. The help article on entering URL lists carries a last-updated date of 19 August 2024, and we could not find a post newer than 2023 on the company blog. The marketing and pricing pages render only through JavaScript, so an automated read of them returns an empty document. We could not verify a current ParseHub price first-hand, and the figures circulating in comparison posts are not something we can confirm. The product works; the company is quiet.

Apify - input as a first-class field

Apify is the most explicitly input-driven of the group, because every Actor takes a JSON input and the general-purpose scrapers expose startUrls as an array:

{
  "startUrls": [
    { "url": "https://www.amazon.com/dp/B08N5WRWNW" },
    { "url": "https://www.amazon.com/dp/B07FZ8S74R" }
  ]
}

The console does not make you build that by hand. The Web Scraper actor accepts start URLs typed one at a time, uploaded as a CSV, or linked from a Google Sheets document, and the page function can add more at run time with await context.enqueueRequest(). Linking a sheet is the underrated option: the list stays where your analysts edit it, and the next run picks up their changes without a re-upload.

Plans are Free, Starter at $29 a month, Scale at $199 and Business at $999, each figure being the credit you then spend on Store actors or on your own. The ceiling that bites on long lists is concurrency, and Apify documents it per plan: 25 concurrent Actor runs on Free, 32 on Starter, 128 on Scale, 256 on Business.

Data Miner, and one extension that does not belong on this list

Data Miner's paid tiers include a feature called Automate Scrape Crawls, described in its own words as: "Say you have 100 URLs representing the pages of a product catalog. You can use Data Miner to automatically open each URL and scrape information from it." Pricing runs Starter free at 500 pages a month, Solo $19.99 for 500, Small Business $49 for 1,000, Business $99 for 4,000, Business Plus $200 for 9,000. The extension is current: version 5.8.102, updated 18 January 2026, 300,000 users, 3.9 out of 5 from 703 ratings.

Instant Data Scraper is usually named in the same breath, and it should not be. It is a fine tool, with a million users and 4.9 out of 5 from 7.6K ratings on version 1.6.1, updated 16 July 2026. Its store listing describes automatic navigation to the next page and support for infinite scrolling, and nothing else. It works on the page in front of you. There is no field anywhere in it for a list you supply. Pairing it with Data Miner, as our earlier version did, put a tool in the batch-input category that has never had the feature.

Bright Data - list in, records out

Bright Data's Web Scraper API inverts the usual model: you do not build a recipe, you pick a ready-made scraper and hand it inputs. The trigger endpoint takes either a JSON array or a CSV uploaded as multipart form data in a field called data. Product-page scrapers take URLs:

[
  {"url":"https://www.airbnb.com/rooms/50122531"},
  {"url":"https://www.airbnb.com/rooms/50127677"}
]

Discovery scrapers take the parameter shape instead, as keywords, category URLs or locations, which is the closest thing here to typing a term into a site's own search box at scale. Pricing is per delivered record: 5,000 records a month free, $1.5 per 1,000 records pay as you go, or $499 a month for 384,000 records and $1.3 per 1,000 beyond that. Failed deliveries are not billed, which changes how you think about a dirty input list.

When the list is the whole interface: batch endpoints

Two API products treat the list as the primary object rather than a setting inside a project.

ScraperAPI takes a POST to https://async.scraperapi.com/batchjobs with an array of URL strings. The documented ceiling is 50,000 URLs per batch job, described as the maximum allowed and not exceedable, so a bigger list means splitting it yourself. Results are polled, not pushed: the response hands back a statusUrl per URL, which you check until each one is done. That is a design decision, and the bookkeeping it hands you is covered further down.

Firecrawl exposes /batch/scrape with a urls array, an optional maxConcurrency, webhooks for start, per-page and completion events, and ignoreInvalidURLs, which defaults to true and returns the bad entries in a separate field rather than failing the request. No maximum count is documented. Plans start free at 1,000 credits and two concurrent requests, then $16 a month billed yearly for 5,000 credits, $83 for 100,000 and 50 concurrent, $333 for 500,000, $599 for a million.

The desktop generation, which is not as dead as we said

The earlier version of this article buried the 2010s desktop tools in one line. That was too quick. Helium Scraper is still selling: the footer reads "Copyright © 2026 Helium Software", the download is a ten-day trial of Helium Scraper 3, and the licenses are one-time payments of $99, $199, $399 and $699, with the page stating plainly that there are no subscriptions or recurring charges. For a fixed, recurring batch that runs on one Windows machine, a one-time license is arithmetic worth doing against $69 a month forever. The wider survey of this category lives in our web scraping tools overview, and the browser-only options in free online web scrapers.

Worked example: the ASIN list

Start with asins.csv:

asin
B08N5WRWNW
B07FZ8S74R
B09G3HRMVB

The spreadsheet route, which works everywhere. Build a column with ="https://www.amazon.com/dp/"&A2, fill down, and you have full URLs. Paste them into Octoparse's URL list, upload them to Apify or ParseHub as a CSV, or point Bright Data's trigger endpoint at the array. Build the extraction recipe once on a single product page. Run.

The template route, which keeps your IDs as IDs. In ParseHub, the Starting Value is {"asins": ["B08N5WRWNW", "B07FZ8S74R"]} and the Go To Template expression is "https://www.amazon.com/dp/"+asin. Nothing is pre-built, so a fresh export from your database drops straight in. The same idea in Apify lives in the page function, where you construct the URL and enqueue it.

The search route, for when you have no IDs at all. Feed keywords instead, let the tool type each one into the site's search box, and scrape the result list. Octoparse and ParseHub both loop a list of text values into a form field; Bright Data's discovery scrapers take keywords directly.

One correction to the standard advice for this specific target. The usual closing line is "or just use the official API", and for Amazon that line is now out of date. Amazon's own documentation states that Product Advertising API 5.0 has been deprecated and is being replaced by the Creators API, with no shutdown date published. Anyone who built an ASIN pipeline on PA-API 5 is migrating whether they planned to or not, and a scraper driven by an ASIN list is suddenly the stable half of that stack rather than the improvised half.

What breaks when the list gets long

A hundred URLs hides every problem in this section. Ten thousand exposes all of them.

Concurrency is the clock, not the tool. Ten thousand pages at four seconds each is eleven hours of work. On Firecrawl's free tier, two at a time, that is about five and a half hours of wall clock. At the Standard tier's fifty, about thirteen minutes. This is arithmetic rather than a benchmark, and your per-page time will differ. The shape will not: the concurrency number in the pricing table is the number that decides whether a batch is an overnight job or a coffee break.

Retries spend credits. A blocked or failed request is still a request on most meters. Bright Data is the exception here, billing only delivered records. On a weighted meter the damage compounds: 10,000 Amazon URLs at five credits each is 50,000 credits, half of ScraperAPI's Hobby plan, and a retry pass over the 8% that failed is another 4,000.

Export ceilings turn up at the end. Octoparse's pricing table lists 10,000 rows per export on every tier. Discovering that after a 200,000-row cloud run is a bad evening. Web Scraper Cloud keeps data 30 days on the lower plans and 60 on Scale.

Polling is your problem. ScraperAPI's batch endpoint returns one status URL per submitted URL. At 50,000 URLs, that is 50,000 status URLs to track, which means you need a table to record them, a loop to check them, and a rule for the ones still running when your process dies. The API gives you the batch. It does not give you the bookkeeping.

One bad row should not kill the run. Firecrawl's ignoreInvalidURLs exists for exactly this, and it defaults to true. Tools without an equivalent will hand you a partial dataset with no record of what is missing, which is worse than a failure, because it looks like success.

The output has to join back. Carry the input value into every output row. Without an ASIN column in the results, a 4,000-row batch with 43 silent 404s is a dataset you cannot reconcile against the list you started with.

Deduplicate before you pay. A list assembled from three exports is typically 5 to 15% duplicates. On 10,000 Amazon pages through a weighted meter, that is thousands of credits for rows you already had.

Where the input list stops being enough

• The input lives in a live database. Nothing here reads SQL. Exporting a CSV every morning is a manual step waiting to be forgotten, and at that point a script that queries the database, builds the URLs and drives the scrape is fewer moving parts, not more. In Python it is a short wrapper around requests or a scraping framework.
• The list is not fixed. Batch input assumes you know the targets in advance. Crawls that discover targets as they go, or lists that depend on what page three returned, need a queue rather than an upload.
• The site fights back. Thousands of requests from one address trip rate limits, then blocks. You will need rotating proxies and, on guarded targets, CAPTCHA handling. Cloud platforms bundle some of this. At real volume a purpose-built Scrapy pipeline gives you the retry and throttle control you will end up wanting, and long-running recurring feeds against defended targets are the point at which a managed extraction service stops looking like an extravagance.
• The data arrives after the page does. JavaScript-rendered fields need a tool that runs a real browser, or a direct call to the underlying API with your input values as parameters. The second option is usually faster and always cheaper.
• It has to run unattended. Feeding a fresh list on a schedule and delivering rows into a database or a sheet is where a small custom pipeline earns its keep. See running a scraper detached for the mechanics.
• The tool is a browser extension. That is a dependency on a store policy as much as on a vendor. Chrome disabled Manifest V2 extensions for all users with Chrome 138 on 24 July 2025, removed the enterprise override policy in Chrome 139, and has scheduled removal of the remaining MV2 extensions from the Web Store for 31 August 2026. The extensions named here made the migration. The category moves under you regardless.

Practical tips for input-driven batches

• Test on twenty rows. Run the first twenty, open the output, check every field. Then run the rest. The cost of finding a broken selector at row 20 versus row 9,800 is the whole reason this list exists.
• Keep the input value in the output. The single most useful habit in batch scraping, and the one most often skipped.
• Deduplicate and normalize the input first. Trailing slashes, tracking parameters and mixed case all produce duplicate fetches that no tool will spot for you.
• Chunk long lists. Five batches of 2,000 give you five resume points and five chances to notice a target has changed its markup.
• Throttle. A ten-thousand-row list run flat out is indistinguishable from an attack, and gets treated as one.
• Log the misses separately. A file of failed inputs is a re-run. A missing row is a mystery.
• Normalize the output. Prices with currency symbols, ratings as text, stock as three different phrasings. A normalization step is what makes the batch analyzable.

What to reach for

For a file of URLs and a modest volume, the no-code tools are the right answer. The choice among them is mostly about the ceiling you need. Octoparse for the million-URL list, Apify when the list should live in a Google Sheet, ParseHub when you would rather keep raw IDs and build the URL inside the tool. For a pipeline that already has the list in code, a batch endpoint is fewer parts than a project. Move to a script when the input comes from a live database, when the volume attracts blocking, or when it has to run at 3am without anyone watching.

When the batch is large, recurring, and pointed at sites that push back, the run itself becomes the work rather than the scraper. That is the shape of a data-as-a-service feed: you supply the list of inputs, and the structured rows arrive on a schedule.