How to Web Scrape a Table in Python: BeautifulSoup & Pandas

Tables are one of the most common formats for structured data on the web: exchange rates, sports stats, price lists, rankings. This article shows how to web scrape a table in Python and turn it into a clean dataset — from a one-line pandas.read_html to hand-parsing tricky tables with merged cells using BeautifulSoup.

It's a practical offshoot of the broader guide on web scraping with Python, which covers the basics of fetching pages and working with the core libraries.

What we'll cover: the anatomy of an HTML table, the fast route with pandas.read_html, the flexible route with BeautifulSoup, extracting headers, handling colspan/rowspan, encoding and number-format pitfalls, cleaning and exporting the data, JavaScript-rendered tables, and a pros-and-cons summary.

1. The Anatomy of an HTML Table

Before you parse, you need to understand the markup:

<table>
  <thead>
    <tr><th>City</th><th>Population</th></tr>
  </thead>
  <tbody>
    <tr><td>Tokyo</td><td>13 100 000</td></tr>
    <tr><td>Sydney</td><td>5 600 000</td></tr>
  </tbody>
</table>

• <table> — the table container;
• <thead> / <tbody> — head and body (not always present);
• <tr> — a table row;
• <th> — a header cell, <td> — a data cell.

2. The Fast Route: pandas.read_html

If the table is "well-formed" (a normal <table> with no tricks), pandas parses it in one line. Under the hood it uses lxml or BeautifulSoup.

import pandas as pd

# read_html returns a LIST of all tables on the page
tables = pd.read_html("https://example.com/stats")
df = tables[0]          # the first table
print(df.head())
df.to_csv("data.csv", index=False)

Useful parameters:

tables = pd.read_html(
    url,
    match="Population",  # only take tables containing this word
    header=0,            # which row is the header
    thousands=" ",       # thousands separator (for "13 100 000")
    decimal=",",         # decimal separator (many EU locales)
)

Tip: if the site blocks pandas' requests, download the HTML with requests using the right headers and pass the text: pd.read_html(response.text).

read_html is perfect for simple tables. But it stumbles on non-standard markup, merged cells, and tables built from divs instead of a real table. For those you need manual parsing.

3. The Flexible Route: BeautifulSoup by Hand

BeautifulSoup gives you full control, which is why "beautifulsoup scrape table" is such a common search. The basic loop over rows and cells:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com/stats", timeout=10)
resp.encoding = resp.apparent_encoding
soup = BeautifulSoup(resp.content, "lxml")

table = soup.find("table")
rows = []
for tr in table.find_all("tr"):
    cells = [td.get_text(strip=True) for td in tr.find_all(["td", "th"])]
    if cells:                      # skip empty rows
        rows.append(cells)

for row in rows:
    print(row)

find_all(["td", "th"]) captures both regular and header cells. get_text(strip=True) trims stray whitespace and line breaks.

Selecting a specific table

If there are several tables, anchor onto a class, id, or nearby element:

table = soup.find("table", class_="prices")
table = soup.select_one("#main-table")
table = soup.find("h2", string="Prices").find_next("table")

4. Extracting Headers

To get a meaningful dict/DataFrame, separate the headers from the data:

table = soup.find("table")

# headers — from thead or the first row
headers = [th.get_text(strip=True) for th in table.select("thead th")]
if not headers:
    first_row = table.find("tr")
    headers = [c.get_text(strip=True) for c in first_row.find_all(["th", "td"])]

# data
data = []
for tr in table.select("tbody tr"):
    cells = [td.get_text(strip=True) for td in tr.find_all("td")]
    if len(cells) == len(headers):
        data.append(dict(zip(headers, cells)))

import pandas as pd
df = pd.DataFrame(data)

dict(zip(headers, cells)) turns a row into a "header → value" dictionary — from there it's easy to assemble a DataFrame.

5. Tricky Tables: colspan and rowspan

Merged cells break naive parsing: the number of <td>s per row stops matching. You have to "expand" the merges.

colspan (horizontal merge)

def expand_row(tr):
    cells = []
    for td in tr.find_all(["td", "th"]):
        text = td.get_text(strip=True)
        span = int(td.get("colspan", 1))
        cells.extend([text] * span)   # duplicate across the merge width
    return cells

rowspan (vertical merge)

rowspan is harder — the value "flows" into the rows below. You need to keep a carry-over buffer:

def parse_table_with_rowspan(table):
    result = []
    rowspans = {}          # {column_index: (value, rows_left)}

    for tr in table.find_all("tr"):
        row = []
        col = 0
        cells = tr.find_all(["td", "th"])
        cell_iter = iter(cells)

        while col < len(rowspans) or cells:
            # first, fill cells "flowing down" from above
            if col in rowspans and rowspans[col][1] > 0:
                value, left = rowspans[col]
                row.append(value)
                rowspans[col] = (value, left - 1)
                col += 1
                continue
            try:
                td = next(cell_iter)
            except StopIteration:
                break
            text = td.get_text(strip=True)
            rs = int(td.get("rowspan", 1))
            if rs > 1:
                rowspans[col] = (text, rs - 1)
            row.append(text)
            col += 1
        if row:
            result.append(row)
    return result

This is a simplified skeleton — real tables can be fussier. But the principle is clear: keep a dict of active rowspans and substitute the values into subsequent rows. Often it's easier to try pandas.read_html first (it can expand many merges) and drop to manual parsing only if pandas can't cope.

6. Encoding and Number Formats in Tables

If the cells come back garbled, the problem is in the response encoding, not the table. Pass the parser the raw bytes (resp.content) or set the encoding (resp.encoding = resp.apparent_encoding). Most modern pages are UTF-8, but you'll still meet legacy encodings (Windows-1252, ISO-8859-1, and others) on older sites.

A separate gotcha is numbers in international formats: "13 100 000" with a space as the thousands separator (common on European and many international sites). Clean it before converting to a number:

value = "13 100 000".replace("\xa0", "").replace(" ", "")
number = int(value)   # 13100000

\xa0 is a non-breaking space — a frequent "invisible" guest in web tables that looks like a normal space but isn't. Strip it explicitly, or int()/float() will choke.

7. Cleaning and Saving the Data

After extraction the data is almost always "dirty": whitespace, currency symbols, units.

import re

def clean_price(text):
    # "$1,299" -> 1299
    digits = re.sub(r"[^\d]", "", text)
    return int(digits) if digits else None

df["price"] = df["price"].apply(clean_price)

Saving to various formats via pandas:

df.to_csv("data.csv", index=False, encoding="utf-8-sig")   # -sig for Excel
df.to_excel("data.xlsx", index=False)
df.to_json("data.json", orient="records", force_ascii=False)

utf-8-sig adds a BOM so Excel displays accented and non-ASCII characters correctly. force_ascii=False keeps non-ASCII characters as-is rather than as \uXXXX. For more on working with JSON, see parsing JSON in Python.

8. Dynamic Tables (JavaScript)

If the table is loaded by a script (pagination without a reload, AJAX), it won't be in the original HTML. Two paths:

1. Find the data source. Open the Network tab in your browser — often the table is populated from a JSON API. Parsing the API is easier and more reliable than parsing HTML; see parsing JSON.
2. Render with a browser. Playwright or Selenium will wait for the render, and then you parse the finished HTML:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(url)
    page.wait_for_selector("table")
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "lxml")
# ... then treat it like a normal table

9. Pros and Cons of Each Approach

Practical recommendation: start with pandas.read_html. If it can't cope, use BeautifulSoup. If the table is JavaScript-driven, look for the JSON API, and only render with a browser as a last resort. And if identical tables are spread across hundreds of pages (a paginated catalog, an archive of quotes), fetch them in parallel — it multiplies your collection speed; see asynchronous scraping in Python.

Need the same tables pulled from hundreds or thousands of pages, cleaned, and delivered on a schedule? That's exactly what scraping.pro does — a done-for-you data extraction service or ongoing data as a service, with the parsing, cleanup, and export to CSV/Excel/database handled for you.