API Scraping: How to Extract Data from Any API

API scraping is the practice of pulling data straight from a service's programming interface instead of parsing its HTML. When a site or app exposes an API, calling it directly gives you clean, structured data with far less effort and far fewer breakages than scraping the rendered page. This guide explains what API scraping is, how to find and call web APIs, how to handle authentication, pagination and rate limits, and when an API is the better choice over HTML scraping.

What is an API, and what "API scraping" means

An API (Application Programming Interface) is an interface through which one program talks to another to retrieve data or trigger actions, without knowing its internals. In practice, "working with an API" most often means calling a web API over HTTP: the client sends a request to a specific address (an endpoint) and the server returns a response.

API scraping usually refers to the whole cycle of obtaining and parsing data from an external service:

1. Build and send a correct HTTP request.
2. Receive the server's response.
3. Check the response status and headers.
4. Parse the response body (most often JSON) into program objects.
5. Handle errors, retries and pagination.

See also: our companion piece on parsing JSON covers the narrow step of turning a JSON string into objects. This article covers the full workflow of working with an API, of which that parsing step is one part.

Unlike "website scraping" (parsing HTML), API work relies on structured responses in a machine-readable format, so it is more reliable, more stable and almost always preferable when a service offers an official API.

How to find a site's hidden API

Many sites you would assume require HTML scraping actually load their content from an internal JSON API in the background. Modern pages fetch data with JavaScript after the initial load, and that data comes over exactly the kind of endpoint you can call yourself. Finding it turns a fragile HTML scrape into clean API scraping.

Here is the workflow:

1. Open the page in a browser and launch DevTools (F12 or right-click, then Inspect).
2. Go to the Network tab and filter by Fetch/XHR.
3. Interact with the page: scroll, paginate, search, open a product. Watch the requests appear.
4. Look for responses that return JSON. Click one and inspect its Response and Headers.
5. Right-click the request and choose Copy as cURL to reproduce it exactly, then translate it into your language of choice.

A few things to check once you have found the endpoint: the query parameters that control filtering and paging, any required headers (an Authorization token, an API key, a Referer, or a custom header like X-Requested-With), and whether the token is short-lived. Undocumented internal endpoints can change without notice, so treat them as less stable than a public, versioned API, and always confirm that using them is within the site's terms.

Anatomy of an HTTP request

Every web-API request has a few parts.

Method (HTTP verb)

The method describes the intent of the request:

• GET - retrieve data (does not change state).
• POST - create a new resource or submit data.
• PUT / PATCH - update a resource (fully / partially).
• DELETE - remove a resource.

URL and query parameters

The endpoint address can carry query parameters for filtering, sorting and pagination:

https://api.example.com/users?role=admin&page=2&limit=50

Headers

Headers pass request metadata. The most important ones for API work:

• Authorization - authentication data (a token or key).
• Content-Type - the format of the body you send (for example application/json).
• Accept - the format you want the response in.
• User-Agent - the client identifier.

Request body

For POST/PUT/PATCH, the body carries the data, usually as a JSON string that you first serialize from program objects.

Anatomy of a response and how to parse it

A server response consists of a status code, headers and a body.

Status codes

Before parsing the body, check the status code:

• 2xx - success (200 OK, 201 Created, 204 No Content).
• 3xx - redirect.
• 4xx - client-side error (400 bad request, 401 unauthorized, 403 forbidden, 404 not found, 429 too many requests).
• 5xx - server-side error.

It only makes sense to parse the body as successful data on a 2xx code. Note that error bodies often contain useful JSON describing the problem too.

Response headers

Response headers carry important information: Content-Type (the body format), pagination parameters, rate limits (X-RateLimit-Remaining), and cache control.

Response body

The body is the data itself. In most APIs this is JSON, which you need to parse. Less often you will get other text formats: XML (see parsing XML) and CSV (see parsing CSV), along with binary formats.

Authentication and authorization

Most APIs require the client to prove it is allowed access. Common methods:

• API key - a simple string passed in a header or query parameter.
• Bearer token / OAuth 2.0 - a token in the Authorization: Bearer <token> header; the most common approach in modern APIs.
• Basic Auth - a username and password in encoded form.
• HMAC / request signature - the request is signed with a secret, used in payment and cloud APIs.

Important: keys and tokens are secrets. Never hard-code them or commit them to a repository; use environment variables or a secrets manager.

Common challenges in API scraping

API scraping rarely comes down to a single request. Here are the typical challenges to prepare for.

Pagination

An API almost never returns large collections all at once - the data is split into pages. The main models:

• Offset/limit - ?page=2&limit=50 or ?offset=100&limit=50.
• Cursor-based - the response includes a pointer to the next page (next_cursor) that you pass into the next request.
• Keyset - the next page is requested by the value of the last item (for example an id or a date).

To collect everything you need a loop that walks the pages until they run out.

Rate limiting

Services cap the number of requests per period. Exceed it and you get a 429 code. The correct response is to watch the rate-limit headers and apply exponential backoff with retries.

Network unreliability and retries

Network requests can fail on timeout or from transient 5xx errors. For resilience, use:

• sensible timeouts;
• retries for idempotent requests with increasing delay;
• the circuit breaker pattern for systematic failures.

If a site sits behind bot protection, you may also need rotating proxies and, occasionally, CAPTCHA solving to reach the endpoint at all.

Data variability and validation

An API response can arrive incomplete, with null fields, or with a changed structure after a version bump. Never blindly trust the response shape - check for fields and validate the data.

Versioning

APIs evolve and their versions change (/v1/, /v2/). Pin a specific version and watch for deprecation announcements.

Nesting and mixed formats

Useful data is often buried deep in a nested structure (data.items[0].attributes.name). Sometimes you get XML or CSV instead of JSON, which needs a different parser.

Implementation examples across languages

Below are minimal examples of the full cycle: request the API, check the status, parse the JSON response. For consistency, they all use the placeholder endpoint https://api.example.com/users/42.

Python

The popular requests library handles both the request and JSON parsing. For async workloads, httpx offers the same API with await.

import requests

url = "https://api.example.com/users/42"
headers = {"Authorization": "Bearer YOUR_TOKEN"}

response = requests.get(url, headers=headers, timeout=10)

# Check the status
response.raise_for_status()  # raises on 4xx/5xx

# Parse the JSON response into a dict
user = response.json()
print(user["name"])

Collecting all pages (offset/limit):

def fetch_all_users():
    users, page = [], 1
    while True:
        resp = requests.get(
            "https://api.example.com/users",
            params={"page": page, "limit": 50},
            timeout=10,
        )
        resp.raise_for_status()
        batch = resp.json()["data"]
        if not batch:
            break
        users.extend(batch)
        page += 1
    return users

JavaScript (Node.js / browser)

The built-in fetch (global in Node 18+ and every modern browser) returns a promise; JSON parsing is the .json() method.

const url = "https://api.example.com/users/42";

const response = await fetch(url, {
  headers: { Authorization: "Bearer YOUR_TOKEN" },
});

// Check the status
if (!response.ok) {
  throw new Error(`API error: ${response.status}`);
}

// Parse JSON
const user = await response.json();
console.log(user.name);

Sending data (POST):

const response = await fetch("https://api.example.com/users", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    Authorization: "Bearer YOUR_TOKEN",
  },
  body: JSON.stringify({ name: "Ada Lovelace", role: "admin" }), // serialization
});

const created = await response.json();

Java

Modern Java ships a built-in HttpClient, and JSON parsing uses a library (Jackson here).

import java.net.URI;
import java.net.http.*;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.JsonNode;

public class ApiExample {
    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newHttpClient();

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create("https://api.example.com/users/42"))
                .header("Authorization", "Bearer YOUR_TOKEN")
                .GET()
                .build();

        HttpResponse<String> response =
                client.send(request, HttpResponse.BodyHandlers.ofString());

        if (response.statusCode() != 200) {
            throw new RuntimeException("API error: " + response.statusCode());
        }

        ObjectMapper mapper = new ObjectMapper();
        JsonNode user = mapper.readTree(response.body());
        System.out.println(user.get("name").asText());
    }
}

Go

The standard library provides both the HTTP client (net/http) and the parser (encoding/json).

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
)

type User struct {
    Name string `json:"name"`
}

func main() {
    req, _ := http.NewRequest("GET", "https://api.example.com/users/42", nil)
    req.Header.Set("Authorization", "Bearer YOUR_TOKEN")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        panic(fmt.Sprintf("API error: %d", resp.StatusCode))
    }

    var user User
    json.NewDecoder(resp.Body).Decode(&user) // streaming parse of the body
    fmt.Println(user.Name)
}

C

.NET uses HttpClient and the built-in System.Text.Json.

using System;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

record User(string Name);

class Program
{
    static async Task Main()
    {
        using var client = new HttpClient();
        client.DefaultRequestHeaders.Add("Authorization", "Bearer YOUR_TOKEN");

        var response = await client.GetAsync("https://api.example.com/users/42");
        response.EnsureSuccessStatusCode();

        // Parse JSON straight into a typed object
        var user = await response.Content.ReadFromJsonAsync<User>();
        Console.WriteLine(user?.Name);
    }
}

PHP

cURL makes the request and json_decode parses the response.

<?php
$ch = curl_init("https://api.example.com/users/42");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, ["Authorization: Bearer YOUR_TOKEN"]);

$body = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);

if ($status !== 200) {
    throw new Exception("API error: $status");
}

// Parse JSON into an associative array
$user = json_decode($body, true);
echo $user["name"];

Rust

A popular pairing: the async reqwest client and serde for parsing.

use serde::Deserialize;

#[derive(Deserialize)]
struct User {
    name: String,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();

    let response = client
        .get("https://api.example.com/users/42")
        .header("Authorization", "Bearer YOUR_TOKEN")
        .send()
        .await?;

    if !response.status().is_success() {
        return Err(format!("API error: {}", response.status()).into());
    }

    // Parse JSON into a struct
    let user: User = response.json().await?;
    println!("{}", user.name);
    Ok(())
}

Tool comparison

Best practices

When building reliable API work, stick to a few principles:

• Always check the status code before parsing the body.
• Wrap parsing in error handling - external data is unreliable.
• Never store secrets in code - use environment variables.
• Respect rate limits - apply delays and backoff on 429.
• Set timeouts on every request so you never hang.
• Log requests and errors - it makes integrations far easier to debug.
• Cache rarely changing data to reduce load and stay under limits.
• Pin the API version and watch for deprecation notices.
• Validate the response shape before using its fields.

FAQ

Is API scraping legal? Calling a public, documented API within its terms of service is generally fine. Using an undocumented internal endpoint sits in a grayer area - review the site's terms, avoid authentication you were not granted, and do not overload the service. When in doubt, get legal advice for your specific case.

Is API scraping the same as web scraping? They overlap. Web scraping traditionally means parsing HTML; API scraping means calling a structured endpoint. Many real projects mix the two - you scrape HTML for pages that render server-side and hit APIs for the parts loaded dynamically.

When should I prefer an API over HTML scraping? Whenever an official or discoverable API exists. It returns structured data, breaks less often when the site's design changes, and is usually faster and lighter to run.

How do I handle rate limits? Read the X-RateLimit-* headers, throttle your request rate, and back off exponentially when you receive a 429. For large jobs, spread requests over time or across rotating proxies.

Conclusion

API scraping is the full cycle of interacting with an external service: building the request, authenticating, checking the status and headers, parsing the response body, and handling edge cases - pagination, limits, network failures. Technically the parsing step almost always comes down to parsing JSON, but a robust integration means accounting for everything around it. The tooling exists in every common language: sometimes the HTTP client and parser are built in, sometimes you reach for a popular library, but the principle is the same everywhere - turn a remote service's response into reliable data you can safely work with.

If you would rather not build and maintain all of this yourself, scraping.pro runs API scraping as a managed data extraction service, including endpoint discovery, auth handling and scheduled delivery. For a deeper look at the parsing step, see our guides on parsing JSON, parsing XML and parsing CSV.