Document Data Extraction: Formats, Libraries, Approaches

Five Python libraries read the same 200-page PDF for this article. The fastest finished in 0.64 seconds. The most convenient one, the one every tutorial reaches for first, took 62.5 seconds for the same characters. Neither was broken. They answer different questions, and picking the wrong one costs a factor of ninety-seven, because on a three-page test file both feel instant.

That gap, repeated across ten formats, is what document data extraction actually is. The textbook definition — pull structured data out of files and turn it into rows — hides the decision that matters: whether the structure you want is in the file, or only ever existed in the eye of whoever looked at it. A CSV carries its columns. A scanned invoice carries pixels, and the column is something you assert about them.

The job is also never one format: a price list in Excel, a CRM export in CSV, a contract in PDF, a partner feed in JSON or XML. Below, each gets its tool, that tool's version on 10 August 2026, and the specific way it fails. If the pipeline itself is the thing you would rather not own, we also build and run document extraction under contract.

Three steps, and the third one is the job

Every parse decomposes the same way. Reading gets the bytes from a folder, a URL, or a stream. Parsing turns those bytes into objects: a tree, a list of rows, a workbook. Extraction and normalization selects the fields, coerces dates and currencies into types, and writes the result.

For JSON and XML the middle step is free: there is a formal grammar, and the parser must build a tree or fail with a syntax error you can read. For PDF, logs and arbitrary text, the middle step barely exists and the third one swallows the budget, because you are reconstructing a structure the format never stored, using regular expressions, coordinates and guesswork.

So the first question is not which library. Does the file carry its own structure, or did somebody's eyes supply it?

Formats that carry their own structure

JSON

JSON is the response format of most APIs and maps onto language primitives with no impedance: object to dictionary, array to list. Parsing is one standard-library call; the work starts afterward, walking whatever nesting the vendor invented. Streaming and the pitfalls that bite in production are in the guide to parsing JSON.

import json

with open("data.json", encoding="utf-8") as f:
    data = json.load(f)

print(data["items"][0]["name"])

Two clauses of RFC 8259 explain most JSON incidents. Object names "SHOULD be unique", and when they are not, "the behavior of software that receives such an object is unpredictable". Python keeps the last duplicate silently, so a feed that repeats a key is legal JSON that hands two consumers two different answers.

The second clause is the number line. The RFC guarantees interoperability only for integers between -(2^53)+1 and (2^53)-1, because implementations backed by IEEE 754 doubles hold no more. A nineteen-digit order ID survives Python's arbitrary-precision integers and quietly loses its last digit when the same file reaches JavaScript. Nothing raises; the row joins to the wrong parent. Read IDs as strings, in the schema.

XML

XML is a tree of tagged nodes, stricter and far more verbose than JSON, and a great many industry standards sit on it: marketplace exports, banking messages, industrial exchange. You either stream it incrementally or build a tree and query it with XPath. Libraries and examples are in the guide to parsing XML with lxml.

lxml is at 6.1.1 (May 2026). Here is a claim worth retiring, one this article used to repeat: that every XML parser is an unpatched XXE hole and defusedxml is mandatory. lxml's own FAQ states that "since version 5.x, lxml disables the expansion of external entities (XXE) by default", and that you must set resolve_entities=True to get the old behaviour back. The defusedxml README marks its own lxml module "DEPRECATED", and its table now reads "Maybe" rather than "Vulnerable" for the standard-library parsers, because expat 2.4.0 ships sensible defaults.

Entity expansion inside the document has not gone away. Billion laughs still turns a few hundred bytes into roughly 3 GB of memory through nine levels of ten-fold expansion. Parse untrusted XML with resolve_entities=False and cap the input size before the parser sees it.

RSS and Atom

RSS is XML with a fixed vocabulary for feeds and podcasts, and a dedicated library returns normalized entries with dates and links already coerced. The mechanics are in the guide to parsing RSS feeds. The two specifications behave very differently under load. RSS 2.0 has been frozen since 2.0.11, published by the RSS Advisory Board on 30 March 2009, and it is permissive to the point of uselessness for a parser author: "All elements of an item are optional, however at least one of title or description must be present." Dates follow RFC 822, so two-digit years are legal. Atom, RFC 4287 from December 2005, requires exactly one id, title and updated per entry. That is why deduplication against Atom works and against RSS is a heuristic.

feedparser 6.0.14 (July 2026) exists because real feeds are broken. It sets a bozo flag when the document is not well-formed, keeps parsing, and stashes the exception in bozo_exception. Treat that flag as data: a feed that starts failing well-formedness usually belongs to a publisher who just changed CMS, which is when your field mapping breaks too.

Sitemaps

A sitemap lists a site's URLs for crawlers, and reading it first gives you an inventory instead of blind link discovery. Plain files, index files, lastmod and gzipped variants are covered in the guide to parsing sitemaps and XML. The numbers in it are hard limits: sitemaps.org caps one file at 50,000 URLs and 50MB uncompressed — precisely 52,428,800 bytes — and an index at 50,000 sitemaps. Gzip is allowed, and the limit applies after decompression, which is what people miss when a 6MB archive is rejected.

Google's documentation then deletes two fields from your model: "Google ignores <priority> and <changefreq> values." lastmod survives only "if it's consistently and verifiably accurate", so a generator that stamps today's date on every URL has taught the crawler to ignore the one field you wanted.

Tables that only look like tables

CSV

CSV is rows and columns separated by a comma, or a semicolon, or a tab, or a pipe. That is the whole problem. Quoting, embedded line breaks, encodings and byte-order marks are where the damage lives, and a hand-rolled split(",") is wrong on the second file. The failure modes are in the guide to parsing CSV files.

import csv

with open("prices.csv", encoding="utf-8-sig", newline="") as f:
    for row in csv.DictReader(f):
        print(row["sku"], row["price"])

Two arguments there are load-bearing, and the earlier version of this article had neither.

encoding="utf-8-sig". A CSV exported from Excel begins with a UTF-8 byte-order mark. Decode it as plain utf-8 and your first column name is not sku but \ufeffsku, so row["sku"] raises KeyError on the header row of every Excel export in existence. Verified here: the same bytes decoded as utf-8 give the header ['\ufeffsku', 'price'], and as utf-8-sig give ['sku', 'price'].

newline="". Python's csv documentation states the consequence of omitting it: "newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use \r\n line endings on write an extra \r will be added."

Two more. csv.field_size_limit() defaults to 131072 bytes on CPython 3.11, so one oversized field aborts the read with a _csv.Error rather than truncating. And pandas reads in chunks by default, which the docs call "possibly mixed type inference": a column of digits that turns alphanumeric at row 900,000 comes back as object, and an integer join silently returns zero matches.

One clause belongs in every exporter rather than every importer. CSV injection, in OWASP's phrasing, "occurs when websites embed untrusted input inside CSV files": a cell beginning with =, +, -, @, a tab, a carriage return or their full-width twins is a formula when a spreadsheet opens it. OWASP recommends prefixing such a cell with a tab inside the quoted field, noting that the tab then survives into whatever reads the file next.

When the file is large and its shape unknown, DuckDB is the pragmatic first look: its sniffer inspects 20,480 lines and store_rejects = true diverts malformed lines into a rejects table instead of dying on line four million.

Excel

Excel is not text. XLSX is a ZIP archive of XML parts; the older XLS is a binary compound file. Sheets, formulas, number formats and cached values live inside, and none of it survives being read line by line. Walking sheets, merged cells and formula results is covered in the guide to parsing Excel files. Microsoft's published limits are the boundary of the format: 1,048,576 rows by 16,384 columns per worksheet, 32,767 characters per cell. Output that might exceed a million rows needs a different container, and a quarterly export is a bad place to discover that.

The formula trap. openpyxl reads either the formula or the value "stored the last time Excel read the sheet". A file generated by a server and never opened in Excel has no cached value. Demonstrated here with openpyxl 3.1.5: write =A1*A2 into a cell, save, reopen with data_only=True, and the cell reads None; with data_only=False it reads '=A1*A2'. openpyxl does not evaluate formulas and never has. A generated workbook of computed totals hands you a column of nulls, and those nulls look exactly like empty cells.

The engine trap. Tutorials still say xlrd reads xlsx. It has not since xlrd 2.0.0, published 11 December 2020; pandas now documents plainly that "xlrd supports old-style Excel files (.xls)." The current map is openpyxl for .xlsx, xlrd for .xls, pyxlsb for .xlsb, odf for OpenDocument, and calamine for all of them.

That last engine is worth a migration. Measured here on a generated 50,000-row, five-column workbook of 1,836,017 bytes, best of three runs: python-calamine 0.8.2 read it in 0.45 s and openpyxl 3.1.5 in read-only mode took 3.46 s; through pandas.read_excel the same two engines took 1.08 s and 4.22 s. Sevenfold, on a file small enough that nobody would think to benchmark it.

A correction to this article's earlier advice. The previous version suggested converting price lists to CSV first and processing everything uniformly. Do not. The conversion is where types die: dates become locale-dependent strings, decimal separators flip between . and , with the exporting machine's regional settings, leading zeros vanish from postal codes, and long identifiers get rewritten in scientific notation. Read the workbook as a workbook.

The cost of that coercion is documented. In Genome Biology in August 2016, Ziemann, Eren and El-Osta of the Baker IDI Heart and Diabetes Institute screened 35,175 supplementary Excel files from 18 journals and found gene-name errors in 704 articles, about 19.6% of papers carrying supplementary gene lists, because Excel had converted gene symbols to dates and floats. In 2020 the HGNC gave up and renamed the affected human genes so spreadsheets would stop mangling them.

A JavaScript footnote that costs money. npm install xlsx gets you 0.18.5, published 24 March 2022 and untouched since, because SheetJS moved distribution off the public registry. Its own installation page says so: "the registry is out of date." The other common Node choice, exceljs, last shipped in October 2023.

PDF: a format that does not know what a word is

PDF describes pages for printing and viewing, which makes it the hardest thing here to extract from reliably. Text may be selectable characters or an image needing OCR, and tables are not stored as tables but as line segments and text runs at coordinates. Which library gets text, which gets tables, and when recognition is unavoidable is covered in the guide to scraping data from PDF.

pypdf's documentation states the core fact better than any summary: "The text within a PDF document is absolutely positioned, meaning that every single character could be positioned on the page", and "PDF files don't contain a semantic layer." There is no word boundary in the file. Spaces are inferred from gaps and kerning adjustments, which is why extracted text sometimes reads t o t a l and sometimes subtotalVAT.

The current specification is ISO 32000-2:2020, the dated revision of PDF 2.0 first published in July 2017, and it has been free from the PDF Association since 5 April 2023.

Five extractors, one file

Measured on a single machine on 10 August 2026: Python 3.11.15, a 200-page PDF of 396,783 bytes generated with reportlab, roughly 48 lines per page, best of three runs, whole-document text extraction only. Absolute numbers will differ on your hardware; the ratios are stable.

All five returned between 726,733 and 736,933 characters, so this is not a quality trade in the ordinary sense. pdfplumber is slow because it keeps every character, line and rectangle as an addressable object with coordinates — exactly what you need to reconstruct a table, and pure overhead when you want a wall of text. Three hundred milliseconds a page across ten thousand pages is close to an hour; four milliseconds is under a minute. Triage with the cheap extractor, then spend the expensive one on the pages that matter.

Licensing is a real constraint here. PyMuPDF is dual licensed: AGPL, or a commercial licence from Artifex, and its documentation says outright that if you cannot meet the AGPL's requirements you should contact Artifex. Shipping it inside a closed-source product is a decision for someone with signing authority. pypdf, pdfminer.six, pdfplumber and pypdfium2 carry no such condition.

For tables the current versions are pdfplumber 0.11.10 (14 June 2026), with four detection strategies, and Camelot 2.0.0 (June 2026), whose documentation states the boundary flatly: "Camelot only works with text-based PDFs and not scanned documents."

One quiet change in your favour. Tagged PDF — the structure tree recording reading order, table cells and heading levels — used to be rare. Directive (EU) 2019/882, the European Accessibility Act, applies to services provided to consumers after 28 June 2025, and its text exempts office file formats published before that date. EU-facing organisations now have a compliance reason to emit tagged PDFs, and a tagged document turns reading order from a guess into a lookup. Check for a structure tree before committing to coordinate-level work.

HTML, text files and logs

HTML is structurally close to XML and never behaves like it: unclosed tags, injected markup, content that exists only after JavaScript runs. It dominates when collecting data from websites rather than from files — parsers, CSS selectors and link normalization are in the guide to parsing HTML. The same lxml install handles both, but the tolerant HTML path and the strict XML path have different security postures.

Plain .txt files and server logs have no shared schema; whoever wrote the file defined the format. That usually means line-oriented reading plus regular expressions. Logs are common and awkward enough to warrant their own article on log parsing; for one-off text files the same discipline applies as in the PDF case. A pattern that works on today's sample has not met next month's release notes. Anchor on stable tokens, log the lines you failed to match, and count them. An extractor with no unmatched-line counter cannot tell you when it broke.

Scans, and what OCR actually gives you

A large share of real documents — invoices, receipts, IDs, signed contracts — arrive as photographs of paper wrapped in a PDF container, with no text layer. Recognition comes first, and it is its own discipline.

• Tesseract is the open-source baseline, at 5.5.3 as of 24 July 2026 per the project's own release notes, driven from Python by pytesseract 0.3.13. That wrapper has not shipped since August 2024, which is what a stable subprocess wrapper looks like, not an abandoned one.
• OCRmyPDF wraps Tesseract to add a searchable text layer to an existing PDF without altering its appearance, at 17.10.0 (5 August 2026), MPL-2.0. For an archive of scans this is usually the right first pass: run it once, then treat everything downstream as a normal text PDF.
• Preprocessing with OpenCV — deskew, denoise, threshold — moves accuracy more than swapping engines does. Tesseract's quality guide asks for at least 300 dpi and warns that skew wrecks line segmentation.

import pytesseract
from PIL import Image

text = pytesseract.image_to_string(Image.open("invoice_scan.png"), lang="eng")

Two things the roundups get wrong about Tesseract. The first is the recurring "99% accurate" figure, which appears in vendor comparisons and in no Tesseract document anywhere; the project publishes no accuracy claim, and real rates move with dpi, font, language and preprocessing across a range far wider than one percentage describes. The second is tables: Tesseract's documentation says it "has a problem to recognize text/data from tables" without custom segmentation. If your scans are mostly tables, OCR alone is not the answer.

OCR returns characters. Turning them into invoice number, date and total is still the third step, harder here because the coordinates you would anchor on are approximate.

The model tier

For high layout variation — thousands of invoice templates, résumés, forms where no two vendors agree — rules stop scaling. The last two years produced a third category between library and cloud API: open-weight models you run yourself.

• Docling, started at IBM Research Zurich and now hosted by the LF AI & Data Foundation, MIT licensed, at 2.119.0 on 10 August 2026. It covers PDF, Office formats, HTML, EPUB and images, and returns a structured document model rather than a flat string.
• Marker 2.0.0 (20 July 2026), Apache-2.0 code with weights under a modified OpenRAIL-M limited by its README to "research, personal use, and startups under $5M funding/revenue".
• MinerU 3.4.4 (10 July 2026), under a custom licence derived from Apache 2.0 but not identical to it. Read it before deploying.
• olmOCR from the Allen Institute for AI, Apache-2.0, whose README claims "less than $200 USD per million pages converted" from a 7B vision model that needs a GPU.
• unstructured 0.25.2 is an ingestion layer rather than a recognition model; Apache Tika 3.3.2 is the JVM equivalent, claiming "over a thousand different file types".

The benchmark numbers exist only in the projects' own tables. Marker's README reports olmocr-bench accuracy of 76.0% for its balanced mode, 72.7% for the MinerU pipeline and 50.3% for Docling, at 2.9, 0.54 and 2.1 pages per second respectively on a single B200 GPU, with Gemini Flash 3.5 at 76.4%. olmOCR's board puts olmOCR at 82.4±1.1 on olmOCR-Bench v0.4.0, with Chandra at 83.1±0.9. OmniDocBench, from OpenDataLab and accepted at CVPR 2025, evaluates on 1,651 annotated PDF pages across ten document types and is the closest thing to a neutral referee.

Each of those tables is published by a party with an entry in it, and the rankings reshuffle every few months. What stays stable is the shape: on born-digital PDFs the pipeline tools stay within a point or two of the models at a fraction of the compute, and the models pull away on scans, handwriting and unruly layouts.

The hosted equivalents skip the GPU and charge by the page instead. Validate either way. A model asked for a total will return a plausible total.

What a page costs

Read from each vendor's own pricing page on 10 August 2026, first-tier rates, per 1,000 pages.

Notes that change the arithmetic. Textract's plain OCR drops to $0.60 and its forms tier to $40.00 above the first million pages, and Layout is free when used with Tables. Google's Enterprise Document OCR falls to $0.60 above five million pages a month, and its Layout Parser is a flat $10.00 with no volume discount. LlamaParse is priced in credits at $1.25 per 1,000: one a page for Fast, ten for Agentic, forty-five for Agentic Plus. Azure AI Document Intelligence is the gap here. Every figure on its public pricing page renders as `$-` and Microsoft directs you to the calculator or a sales contact, so beyond a free tier of 500 pages a month its prices are not knowable from outside.

The spread is the point. Ten thousand pages a month costs $15 through Textract's text detection and $500 through its forms extraction. Same documents, same month, a bill thirty-three times larger, decided by which API name someone typed. Most invoice pipelines need key-value extraction on the two pages carrying the header and plain OCR on the rest; the ones that do not split the work pay the premium on every appendix page.

Per-token LLM pricing resists this table. Gemini lists 2.5 Flash-Lite at $0.10 per million input tokens and $0.40 per million output, with heavier models an order of magnitude above. What no pricing page publishes is how many tokens a page image becomes, which is the number you actually need. Measure it on your own documents before forecasting.

What breaks at ten thousand documents a night

Hard page limits arrive first. Textract's synchronous API accepts exactly one page and 10 MB; the asynchronous one takes 3,000 pages and 500 MB, and its set quotas also rule out XFA-based and password-protected PDFs. Google Document AI allows 40 MB and 15 pages for most extraction processors online, 30 with imageless_mode; batch takes 1 GB and 5,000 files. Azure's documented limits are 4 MB and two pages free, 500 MB and 2,000 pages on standard, with 15 analyse transactions a second before you file a support request. Any of these turns a working prototype into a queue and a splitter.

Billing units compound this: Azure counts each Excel worksheet and each slide as a page, so a spreadsheet with twelve tabs is twelve pages.

Memory is the second wall. openpyxl's read-only and write-only modes exist for this, promising "(near) constant memory consumption". The default modes load the whole workbook, so a worker that survives every test file and dies on one 80 MB export was written against the default constructor.

Partial success needs a design, not a retry loop. On a 3,000-page async job, page 1,412 failing should not discard the other 2,999. Store per-page results, retry a page at a time, and key idempotency on a document hash so a redelivered file does not double-count.

Schema drift is the failure you will not see. A vendor adds a column, renames total to total_incl_vat, or starts padding SKUs with zeros. Nothing raises. Rows land, counts look normal, one field is null for a month. Assert at field level — null rates, value ranges, row counts against the previous run — and fail the batch when an assertion moves more than a few percent. Encoding behaves the same way: a feed that is UTF-8 for eleven months and cp1251 the day someone re-exports produces mojibake that validates.

Cost scales before quality does. A model-based extractor at $0.05 a page is fine for a hundred documents a day and is $18,250 a year at a thousand. The hybrid most production systems converge on is what the invoice forces, not a preference: deterministic parsing for the structured majority, a model for the residue, validation over both.

Choosing an approach, and where each one stops

Every row has a cliff edge. Standard libraries stop where the file stops being what it claims: a CSV whose delimiter changes halfway through is two files concatenated, and no reader will say so. Format libraries stop where structure was never recorded. OCR stops where the pixels stop, and produces characters, never fields.

LLM extraction stops at verification. A model fills every field in your schema whether or not the value is on the page, and the failure is a confident, correctly typed, plausible number. If you cannot check the output against a total, a checksum or a database, you do not have extraction, you have generation. Deterministic parsing fails loudly; model extraction fails quietly. So use the cheapest deterministic tool the format allows and escalate only where structure genuinely breaks down. Escalating early is the expensive mistake, and it is the one the tooling encourages.

Which language and stack

Python is the default, for inventory rather than syntax: pandas 3.0.5, lxml 6.1.1, openpyxl 3.1.5, python-calamine 0.8.2, pdfplumber 0.11.10 and pypdf 6.15.0 cover every format above in a few lines each. Note that pandas 3.0.0, released 21 January 2026, made Copy-on-Write mandatory and gave string columns a dedicated str dtype. Code that checked dtype == object or relied on chained assignment breaks on upgrade, and extraction code often does both.

JavaScript and Node make sense when parsing lives inside a web project or when you need a browser to render first. Check publication dates before committing: csv-parse 7.0.2, papaparse 5.5.4, fast-xml-parser 5.10.1 and pdfjs-dist 6.2.108 are current, and the spreadsheet options are the stale ones above. PHP stays reasonable when the data lands in a site on that stack. Go and Rust get picked for throughput; calamine illustrates why. Java owns the enterprise ingestion tier through Tika and Tabula, the engine underneath tabula-py.

The format dictates the library; the version dictates whether the advice you are reading still holds. Identify the exact format, then go to the matching guide: JSON, XML, RSS, sitemaps, CSV, Excel, PDF, HTML, text files, or logs.

Everything above was checked against primary sources on 10 August 2026: vendor pricing pages, package registries, project documentation and the relevant RFCs. Benchmark rankings age fastest, prices next, version numbers after that. Where a pile of mixed-format documents has to become one clean schema on a schedule and nobody wants to own the pipeline, that is the work data as a service and custom extraction absorb.