Web Scraping with Google Apps Script

Google Apps Script (GAS) is one of the most underrated ways to build a small scraper. It is JavaScript that runs for free on Google's servers, it is wired directly into Google Sheets, Docs and Drive, and it can run on a schedule without a computer left switched on. For a lot of everyday jobs - pull a table off a page every morning, watch a handful of prices, mirror an open data feed into a sheet - Google Apps Script web scraping is the quickest route from "a website has the data" to "the data is in my spreadsheet."

If you have already met the IMPORTXML and IMPORTHTML formulas in Google Sheets scraping, think of Apps Script as the next step up. Those formulas are brilliant until the page needs a header, a login, a POST request, or any logic more involved than "grab this XPath." That is exactly where an IMPORTXML alternative written in Apps Script takes over.

Why use Apps Script for scraping

The trade-offs are worth understanding before you commit:

In its favour

• Free and serverless. No hosting, no VM, no Python environment. You write code in the browser and Google runs it.
• Native Sheets integration. The whole point: data lands in cells with one call, ready to chart or share.
• Scheduling built in. Time-driven triggers run your scraper hourly or nightly on Google's infrastructure.
• JSON parsing out of the box. JSON.parse is native, so hitting a site's internal API is trivial.

Against it

• No real browser. UrlFetchApp downloads raw HTML only. It does not run JavaScript, so single-page apps that render client-side will look empty (you will need their underlying API or a headless browser elsewhere).
• Quotas. There is a daily UrlFetchApp call limit and a per-execution time cap (about 6 minutes on consumer accounts, 30 on Workspace).
• Shared Google IPs. Requests leave from Google's address ranges, which some anti-bot systems throttle or block.

For light-to-medium jobs, none of that gets in the way. Let us build one.

Setup

Open a Google Sheet, then Extensions -> Apps Script. You land in a cloud code editor. Everything below goes into that editor; press Run to execute a function (you will be asked to authorise the script the first time).

Fetching a page with UrlFetchApp

The core of any scraper is the HTTP request. In Apps Script that is UrlFetchApp.fetch:

function getHtml(url) {
  const options = {
    method: 'get',
    headers: {
      // Send a realistic User-Agent - the default GAS agent is often blocked
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ' +
                    'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36'
    },
    muteHttpExceptions: true,   // return the response instead of throwing on 4xx/5xx
    followRedirects: true
  };

  const response = UrlFetchApp.fetch(url, options);
  const code = response.getResponseCode();

  if (code === 200) {
    return response.getContentText();
  }
  Logger.log('Request failed: HTTP ' + code);
  return null;
}

For an API that expects a POST, add a payload and set the contentType. Because JSON is native, working with a JSON endpoint is a two-liner:

function getJson(url) {
  const text = UrlFetchApp.fetch(url, { muteHttpExceptions: true }).getContentText();
  return JSON.parse(text);   // straight to an object - no parser needed
}

Parsing the HTML

This is the part that has changed most since the early Apps Script days. The old trick of Xml.parse(html, true) to leniently parse messy HTML is gone - the legacy Xml service was retired years ago, and its replacement, XmlService.parse, only accepts well-formed XML and will throw on real-world HTML. You have three practical options today.

1. Extract with regular expressions or markers

For a single value or a simple, repeating pattern, a regex on the raw HTML is often enough. Keep patterns narrow and lazy - and never try to parse a whole nested document this way (see the notes on regex for scraping):

function extractTitles(html) {
  const titles = [];
  const re = /<h2[^>]*class="product-title"[^>]*>(.*?)<\/h2>/gis;
  let match;
  while ((match = re.exec(html)) !== null) {
    titles.push(match[1].replace(/<[^>]+>/g, '').trim());  // strip inner tags
  }
  return titles;
}

2. Use a real DOM parser via the Cheerio library

For anything structural - "the third cell of every row," "the link inside each card" - add the community Cheerio port as an Apps Script library (Project Settings -> Libraries, using the public Cheerio script ID you can find in the Apps Script community docs). It gives you jQuery-style selectors:

function parseWithCheerio(html) {
  const $ = Cheerio.load(html);
  const rows = [];
  $('table#prices tbody tr').each(function (i, tr) {
    rows.push([
      $(tr).find('td.name').text().trim(),
      $(tr).find('td.price').text().trim()
    ]);
  });
  return rows;
}

This is the closest thing to Beautiful Soup or CSS selectors inside Apps Script, and it is the route I reach for most.

3. Let a Sheet formula do the XPath

If the page is static and you only need one XPath, you do not need code at all - =IMPORTXML(url, "//div[@class='price']") in a cell is the simplest tool. Apps Script earns its keep when the formula route cannot authenticate, page through results, or clean the data.

A small internal-link crawler

The classic Apps Script demo is a mini crawler that walks a site collecting links. Here is a modern, tidy version using a Set for deduplication and respecting the execution-time limit:

function crawl(startUrl, maxPages) {
  const base = new URL(startUrl).origin;
  const queue = [startUrl];
  const seen = new Set(queue);
  const results = [];
  const start = Date.now();

  while (queue.length && results.length < maxPages) {
    if (Date.now() - start > 5 * 60 * 1000) break;  // stay under the 6-min cap

    const url = queue.shift();
    const html = getHtml(url);
    if (!html) continue;

    results.push([url, (html.match(/<title>(.*?)<\/title>/i) || [,''])[1]]);

    // queue new internal links
    const linkRe = /href="([^"#]+?)"/gi;
    let m;
    while ((m = linkRe.exec(html)) !== null) {
      let href = m[1];
      if (href.startsWith('/')) href = base + href;         // resolve relative
      if (href.startsWith(base) && !seen.has(href) && !/\.(pdf|jpg|png|zip)$/i.test(href)) {
        seen.add(href);
        queue.push(href);
      }
    }
    Utilities.sleep(500);   // be polite between requests
  }
  return results;
}

Writing the results to Google Sheets

The old approach dumped text into a Google Doc; today you almost always want structured rows in a Sheet. Build a 2D array and write it in a single setValues call - looping cell by cell is dramatically slower:

function scrapeToSheet() {
  const sheet = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
  const html = getHtml('https://example.com/catalog');
  const rows = parseWithCheerio(html);          // [[name, price], ...]

  sheet.clear();
  sheet.getRange(1, 1, 1, 2).setValues([['Product', 'Price']]);   // header
  if (rows.length) {
    sheet.getRange(2, 1, rows.length, 2).setValues(rows);         // all data at once
  }
}

That single setValues on the whole range is the one performance rule that matters most in Apps Script.

Running it on a schedule

To scrape automatically, add a time-driven trigger. In the editor click the clock icon (Triggers) -> Add Trigger, choose your function and a "Time-driven" interval - or do it in code:

function setupTrigger() {
  ScriptApp.newTrigger('scrapeToSheet')
    .timeBased()
    .everyHours(6)
    .create();
}

The script now runs on Google's servers every six hours whether or not your machine is on - something a desktop scraper cannot do without an always-on box.

Exporting and emailing the data

Once the sheet is populated, you often need to hand it off as a real file - an Excel workbook for a colleague, a CSV for another system, a PDF for a report, or a Word document from a Google Doc. The reliable, current way to do this is to hit Google's built-in export URL with your OAuth token, then save or email the resulting blob:

function exportAndEmail() {
  const ssId = SpreadsheetApp.getActiveSpreadsheet().getId();
  const url = 'https://docs.google.com/spreadsheets/d/' + ssId + '/export?format=xlsx';
  const token = ScriptApp.getOAuthToken();

  const blob = UrlFetchApp
    .fetch(url, { headers: { Authorization: 'Bearer ' + token } })
    .getBlob()
    .setName('scraped-data.xlsx');

  const file = DriveApp.createFile(blob);        // keep a copy in Drive

  MailApp.sendEmail({
    to: Session.getActiveUser().getEmail(),
    subject: 'Your scraped data',
    body: 'Attached, and stored in Drive: ' + file.getUrl(),
    attachments: [blob]
  });
}

Swap the format value to get other file types: csv, pdf, or tsv for a Sheet. The same pattern converts a Google Doc to .docx - use https://docs.google.com/document/d/<DOC_ID>/export?format=docx. This export-URL method has replaced the old, fiddly Advanced Drive Service code that used to be required for format conversion; it is fewer lines and does not depend on a specific Drive API version. (If you prefer the advanced service, Apps Script now maps it to Drive API v3, where you would call Drive.Files.export.)

Limits and gotchas

Keep these in mind before you point Apps Script at a big job:

• Execution time is capped (~6 minutes consumer, ~30 minutes Workspace). Long crawls must be split into chunks driven by successive triggers, saving progress to the sheet or PropertiesService between runs.
• UrlFetchApp daily quota limits how many fetches you get per day (thousands to tens of thousands depending on account type).
• No JavaScript rendering. If the data only appears after client-side JS runs, UrlFetchApp will not see it - inspect the network tab, find the background API call, and fetch that JSON instead.
• Blocks. Google's shared IPs and the lack of a browser fingerprint mean protected sites may return CAPTCHAs or 403s. Handling those needs rotating proxies and sometimes CAPTCHA solving - neither of which Apps Script provides.

When to move beyond Apps Script

Apps Script is perfect for personal automations and small business dashboards. Once you need to render JavaScript-heavy pages, run thousands of requests an hour, rotate proxies, or feed a production database, you have outgrown the sandbox and want a dedicated pipeline. If you would rather skip the plumbing entirely, scraping.pro can run the whole thing as a managed web scraping service and drop clean data straight into your sheet, database or dashboard.

Conclusion

Google Apps Script turns a spreadsheet into a small, free, cloud-hosted scraper: UrlFetchApp pulls the page or API, regex or the Cheerio library extracts the fields, setValues writes them to Sheets, and a time-driven trigger keeps it fresh - with export URLs to ship the result as Excel, CSV, PDF or Word. It will not render SPAs or defeat serious anti-bot defences, but for the huge middle ground of "get this data into my spreadsheet on a schedule," it is one of the fastest tools going, and a genuine step up from the IMPORTXML formula when the formula runs out of road. If you are new to the wider subject, start with what web scraping is and work back to this when you are ready to automate.