Web Scraping Stock Data: APIs, Sources, and Storage

This is a companion piece to our guide on scraping currency exchange rates. A lot of it carries over almost unchanged — especially the data-type discipline (never a float for prices) and the caching practices. But stock quotes have their own quirks: tickers and exchanges, trading sessions and market holidays, corporate actions (splits and dividends), and — the single biggest difference from central-bank FX rates — data licensing. Real-time market data is regulated by the exchanges and by bodies like the SEC and FINRA. Unlike a freely published reference rate, you often can't pull it for free, and you almost never have the right to redistribute it.

1. What a "quote" actually is, and which data types exist

"Quote" means different things depending on the task, and pinning that down is the first thing you need to do:

• Last / current price — the price of the most recent trade. This is what the "share price now" widget shows.
• OHLCV bar (candle) — Open, High, Low, Close and Volume over an interval (minute, hour, day). The basis of charts and backtests.
• Bid/Ask (order book) — the best buy and sell prices. Needed for trading; usually the most expensive and heavily licensed data.
• EOD (end-of-day) — the day's final closing prices. Cheap or free, and fine for analytics and portfolio accounting.
• Adjusted close — the closing price corrected for splits and dividends so the history stays continuous.

And the key axis is freshness: real-time → 15–20 minute delay → end-of-day. The fresher and more granular the data, the stricter the license and the higher the price.

2. Exchange-native data and official sources

The gold standard is data straight from the exchange — but for equities that is almost always licensed and paid. In the US, real-time consolidated quotes flow through the SIP (Securities Information Processor) and the exchanges' own feeds, and are resold by authorized vendors under a market-data agreement. The same pattern holds for the LSE in the UK, the ASX in Australia and the TSX in Canada. What you can usually get for free is delayed (about 15 minutes) or end-of-day data.

A developer-friendly source for historical data is Nasdaq Data Link (formerly Quandl), a REST service owned by Nasdaq that returns end-of-day prices and fundamentals as JSON, XML or CSV.

A quirk of the Nasdaq Data Link format: data comes back "column-oriented" — a separate array of column names (column_names) and an array of rows (data). To pull a field you have to map the column name to its index rather than relying on a fixed order (see the PHP example below). Several exchange and vendor APIs use this same shape, so it's worth learning once.

Handy bridge to the FX guide: many of the aggregators below (Finnhub, Twelve Data, Alpha Vantage) return not only equities but also FX pairs and crypto. One integration can cover both stocks and live exchange rates — the market rate, not an official central-bank reference rate.

3. Global market-data APIs

There is no fully open "just hit the endpoint" source here the way there is for a central-bank FX feed: they all use a key and enforce limits, and real-time is almost always paid.

A current detail worth flagging: IEX Cloud shut down on August 31, 2024. If some older guide still recommends it, that's out of date — the nearest replacements are Alpha Vantage and Financial Modeling Prep.

Free tiers are great for a prototype but hit their limits fast (Alpha Vantage's 25 requests a day is literally a couple dozen tickers). So on a free tier you always build the logic around a cache and local storage: pull the history once, then only append fresh points.

4. Why stock quotes are harder than FX rates

A few differences that break a naive scraper:

• A ticker isn't unique. The same symbol can trade on several exchanges. So the instrument key is the pair "exchange + ticker" — and more reliably, an international identifier like ISIN or FIGI.
• Trading sessions and holidays. Every exchange has its own schedule, time zone, and pre- and post-market sessions. The "last price" over a weekend is Friday's price.
• Corporate actions. A 1:10 split "drops" the price tenfold — but that's not a market crash, it's a technical recalculation. This is the direct analog of the denomination trap when scraping FX rates: ignore it and you get a false anomaly. Dividends and splits are folded into the adjusted close.
• Instrument currency. A security's price is quoted in the exchange's currency (USD, GBP, EUR, JPY…), and to hold a portfolio in one base currency you have to convert — which is exactly where the rates from the companion article come in.
• Volume. It's an integer, but a very large one — millions and billions of shares. It needs a 64-bit integer type.

5. Five solutions in different languages

Each example hits a different source to show off different formats. Throughout, the accent is on the two rules from the FX guide: prices are stored in a decimal type (never float), and volume in an integer type.

5.1. PHP — Nasdaq Data Link (columnar JSON)

<?php
declare(strict_types=1);

/**
 * Latest close from a Nasdaq Data Link time-series dataset.
 * Data comes back "column-oriented": column_names + rows.
 */
function nasdaqLastClose(string $database, string $code): ?string
{
    $key = getenv('NASDAQ_DATA_LINK_KEY');
    $url = sprintf(
        'https://data.nasdaq.com/api/v3/datasets/%s/%s.json?rows=1&api_key=%s',
        rawurlencode($database),
        rawurlencode($code),
        rawurlencode($key)
    );

    $raw = file_get_contents($url);
    if ($raw === false) {
        throw new RuntimeException('Failed to fetch Nasdaq Data Link');
    }
    $json = json_decode($raw, true, 512, JSON_THROW_ON_ERROR);

    // Columnar payload: names in one array, values in another
    $columns = $json['dataset']['column_names'] ?? [];
    $rows    = $json['dataset']['data'] ?? [];
    if (!$rows) {
        return null; // no data or bad code
    }

    $idx   = array_flip($columns);              // column name -> index
    $close = $rows[0][$idx['Close']] ?? null;   // latest close

    return $close === null ? null : (string) $close;
}

echo 'Close: ' . (nasdaqLastClose('WIKI', 'AAPL') ?? 'no data') . " USD\n";

The instructive part is array_flip($columns) — mapping a column name to its index. That's the correct way to read a columnar API: field order isn't guaranteed, and "magic" indexes like [4] break the moment the response changes. For bit-exact prices, prefer requesting CSV or keeping the raw text — the JSON decoder has already turned the number into a float.

5.2. Python — Finnhub (current quote, Decimal)

import os
import requests
from decimal import Decimal

FINNHUB_TOKEN = os.environ["FINNHUB_TOKEN"]  # free key, ~60 req/min


def finnhub_quote(symbol: str) -> dict[str, Decimal]:
    """Current quote for a symbol (US market on the free tier)."""
    resp = requests.get(
        "https://finnhub.io/api/v1/quote",
        params={"symbol": symbol, "token": FINNHUB_TOKEN},
        timeout=10,
    )
    resp.raise_for_status()
    d = resp.json()
    # Decimal(str(...)) pins exactly the value that arrived in the JSON
    return {
        "current":    Decimal(str(d["c"])),   # current price
        "open":       Decimal(str(d["o"])),
        "high":       Decimal(str(d["h"])),
        "low":        Decimal(str(d["l"])),
        "prev_close": Decimal(str(d["pc"])),
    }


if __name__ == "__main__":
    q = finnhub_quote("AAPL")
    print(f"AAPL: {q['current']} USD (open {q['open']}, high {q['high']})")

5.3. JavaScript / Node.js — Twelve Data (OHLC, volume)

// Free: ~800 requests/day. Key required.
const API_KEY = process.env.TWELVE_DATA_KEY;

async function twelveQuote(symbol) {
  const url = new URL("https://api.twelvedata.com/quote");
  url.searchParams.set("symbol", symbol);
  url.searchParams.set("apikey", API_KEY);

  const res = await fetch(url);
  const d = await res.json();
  if (d.status === "error") throw new Error(d.message);

  return {
    symbol: d.symbol,
    open: d.open,        // strings — don't coerce to Number needlessly
    high: d.high,
    low: d.low,
    close: d.close,
    volume: d.volume,    // volume is a big integer, keep it a string/BigInt
    exchange: d.exchange,
  };
}

twelveQuote("MSFT").then((q) =>
  console.log(`${q.symbol} (${q.exchange}): close ${q.close}, vol ${q.volume}`)
);

JavaScript has no decimal type, and Number is a double. So prices are left as strings; for arithmetic on prices reach for decimal.js / big.js, and for volume use the native BigInt.

5.4. Go — Alpha Vantage (daily OHLCV candles)

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "sort"
    "time"
)

// Alpha Vantage: free 25 req/day, 5/min.
type avDaily struct {
    Series map[string]struct {
        Open   string `json:"1. open"`
        High   string `json:"2. high"`
        Low    string `json:"3. low"`
        Close  string `json:"4. close"`
        Volume string `json:"5. volume"`
    } `json:"Time Series (Daily)"`
}

func dailyBars(symbol string) (avDaily, error) {
    key := os.Getenv("ALPHAVANTAGE_KEY")
    url := fmt.Sprintf(
        "https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=%s&apikey=%s",
        symbol, key,
    )
    client := &http.Client{Timeout: 15 * time.Second}
    resp, err := client.Get(url)
    if err != nil {
        return avDaily{}, err
    }
    defer resp.Body.Close()

    body, _ := io.ReadAll(resp.Body)
    var out avDaily
    if err := json.Unmarshal(body, &out); err != nil {
        return avDaily{}, err
    }
    return out, nil
}

func main() {
    bars, err := dailyBars("IBM")
    if err != nil {
        panic(err)
    }
    // find the most recent candle by date
    dates := make([]string, 0, len(bars.Series))
    for d := range bars.Series {
        dates = append(dates, d)
    }
    sort.Strings(dates)
    last := dates[len(dates)-1]
    b := bars.Series[last]
    // prices kept as strings; for math use shopspring/decimal
    fmt.Printf("IBM %s: O=%s H=%s L=%s C=%s V=%s\n",
        last, b.Open, b.High, b.Low, b.Close, b.Volume)
}

Conveniently, Alpha Vantage returns prices as strings — decimal precision survives out of the box, as long as you don't turn them into float64.

5.5. C# / .NET — Yahoo Finance (unofficial endpoint, decimal)

using System.Text.Json;

// ⚠️ Unofficial Yahoo Finance endpoint — the same one the yfinance library uses.
// Fine for prototypes and learning, but with no guarantees and NOT for production.
public static class YahooChart
{
    private static readonly HttpClient Http = new();

    public static async Task<(string Date, decimal Close)> LastDailyCloseAsync(string symbol)
    {
        var url = $"https://query1.finance.yahoo.com/v8/financial-data-services/chart/{symbol}?interval=1d&range=5d";

        var req = new HttpRequestMessage(HttpMethod.Get, url);
        req.Headers.UserAgent.ParseAdd("Mozilla/5.0"); // Yahoo requires a User-Agent

        var resp = await Http.SendAsync(req);
        resp.EnsureSuccessStatusCode();
        using var doc = JsonDocument.Parse(await resp.Content.ReadAsStreamAsync());

        var result = doc.RootElement.GetProperty("chart").GetProperty("result")[0];
        var timestamps = result.GetProperty("timestamp");
        var closes = result.GetProperty("indicators")
            .GetProperty("quote")[0].GetProperty("close");

        var i = timestamps.GetArrayLength() - 1;
        var unix = timestamps[i].GetInt64();
        var close = closes[i].GetDecimal();    // decimal — correct for a price
        var date = DateTimeOffset.FromUnixTimeSeconds(unix).ToString("yyyy-MM-dd");

        return (date, close);
    }
}

class Program
{
    static async Task Main()
    {
        var (date, close) = await YahooChart.LastDailyCloseAsync("AAPL");
        Console.WriteLine($"AAPL close {date}: {close} USD");
    }
}

This example is an honest look at the "scraping" route through Yahoo: no key needed, data close to real-time — but the endpoint is unofficial and can change without warning, so you can't build production on it. If you go this route at scale, plan for rotating proxies and rate limiting, because a single IP hammering the endpoint gets throttled fast.

6. What data type to store quotes in

The base rule is exactly the one from the currency-rate guide: prices and monetary values go in a decimal type only, never float/double. Binary floating point accumulates rounding error, and over a long backtest that produces discrepancies you can never reconcile.

An example table for OHLCV candles:

CREATE TABLE quotes (
    id           BIGSERIAL PRIMARY KEY,
    source       VARCHAR(16)   NOT NULL,   -- 'NASDAQ', 'FINNHUB', 'AV', 'YAHOO'
    exchange     VARCHAR(16)   NOT NULL,   -- 'NASDAQ', 'NYSE', 'LSE'...
    ticker       VARCHAR(20)   NOT NULL,   -- AAPL, MSFT...
    isin         CHAR(12),                 -- reliable instrument identifier
    ccy          CHAR(3)       NOT NULL,   -- price currency: USD, GBP...
    ts           TIMESTAMPTZ   NOT NULL,   -- bar / quote timestamp
    interval     VARCHAR(8)    NOT NULL DEFAULT '1d', -- 1m | 1h | 1d
    open         NUMERIC(18,6) NOT NULL,
    high         NUMERIC(18,6) NOT NULL,
    low          NUMERIC(18,6) NOT NULL,
    close        NUMERIC(18,6) NOT NULL,
    adj_close    NUMERIC(18,6),            -- adjusted for splits/dividends
    volume       BIGINT        NOT NULL DEFAULT 0,
    fetched_at   TIMESTAMPTZ   NOT NULL DEFAULT now(),
    UNIQUE (source, exchange, ticker, interval, ts)
);

Separately, it's worth storing corporate actions (splits with their ratio and dividends with their date), because when they occur you have to recompute the adj_close history retroactively.

7. Practical tips

• Distinguish real-time, delayed and EOD. For most tasks (portfolio, analytics, a dashboard) delayed or end-of-day data is enough — it's cheaper and easier to license.
• Respect the licenses. This is the main difference from central-bank FX rates. Real-time market data is regulated (exchanges, FINRA, SEC), and even re-displaying it to your users can require an agreement. That's why, for example, Alpha Vantage's US real-time tier is paid. Check the source's terms before you publish anything.
• Build everything around a cache. With a 25-requests-a-day limit there's no other way: pull the history once, then do incremental updates and serve the rest from your own database.
• Treat splits like a denomination change. A sudden N-fold jump in price is almost always a corporate action, not a market move. A simple "change vs. previous day exceeds X%" check catches both splits and parsing errors.
• Identify the instrument by exchange + ticker (better still, by ISIN/FIGI). One ticker lives on several venues.
• Remember sessions and holidays. An empty response on a market holiday is normal; take the last available date from the response, not the date you requested.
• Don't build production on scraping Yahoo. Great for a prototype; for a service that has to run for years, use an API with guarantees. When scraping is the only option, expect to deal with CAPTCHAs and IP blocks.

8. Where this gets used

• Portfolio trackers — current asset value, P&L, conversion to a base currency.
• Algo trading and bots — signals and execution (this is where you actually need real-time and the order book).
• Dashboards and BI — market visualization, sector cuts.
• Screeners and backtests — filtering securities and testing strategies on history (OHLCV + adjusted).
• Robo-advisors and fintech — recommendations and automated management.
• Accounting and valuation — marking positions to market on a given date.
• Alerts — notifications when a price hits a set level.

Wrapping up

Scraping stock quotes is technically close to scraping FX rates — the same decimal types, the same caching, the same caution around "technical" jumps (only instead of a currency's denomination, here it's splits). But three things get added: the ticker is tied to an exchange (and better, to an ISIN/FIGI), the history needs adjusting for corporate actions, and — the most important thing in practice — the data is licensed: nearly everywhere only delayed and end-of-day are free, while real-time costs money and comes with strings attached. For most analytics and portfolio use cases a free aggregator plus a local cache is all you need, and it dovetails neatly with scraping exchange rates for multi-currency portfolios.

If you'd rather not babysit rate limits, corporate-action adjustments and licensing across a dozen sources, scraping.pro can build and run the pipeline for you as a data-as-a-service feed or a one-off custom data extraction job — clean OHLCV and fundamentals delivered on a schedule. Combine type discipline, a cache and respect for licenses, and you get quotes you can actually rely on in your calculations.