Big Data and Data Mining: Basics and Differences

In January 2025 a DuckDB engineer ran the full TPC-H benchmark against a 300 GB dataset on a Raspberry Pi 5. The board had 16 GB of RAM and an NVMe SSD, and the whole rig cost $300. All 22 queries completed, with a geometric mean runtime of 55.2 seconds and about 26 minutes end to end. Keep that number nearby while reading anything that tells you where big data begins.

Big data and data mining get used as synonyms, and they are not. Big data is the material and the plumbing: large, fast-moving, varied datasets plus the systems that store and move them. Data mining is what you do with it, the analysis that pulls patterns, relationships and predictions out of that material. Neither produces anything alone. What has shifted since both phrases were coined is the boundary between ordinary data and big data, and most explainers still draw it where it sat in 2012.

Everything below was read from vendor pricing pages, Apache release announcements and project repositories on 10 August 2026. Prices carry the region they came from, because they differ by region and they move.

Where the single-machine line actually sits

An earlier version of this article offered the standard test: if the dataset fits in memory on one server and opens in a spreadsheet, it is not big data. The spreadsheet half still works. The one-server half has aged badly, and it is worth seeing by how much.

Jordan Tigani, who led BigQuery engineering at Google, published the numbers in February 2023 in an essay called Big Data is Dead. Among BigQuery customers spending over $1,000 a year, 90% of queries processed less than 100 MB. Median storage among heavy users sat well below 100 GB. In May 2025 Hannes Mühleisen, one of DuckDB's creators, added the other side of the ledger in The Lost Decade of Small Data?. The median scan in Amazon Redshift and Snowflake reads roughly 100 MB, and the 99.9th percentile reads under 300 GB. He then ran every TPC-H query at scale factor 1000, a full terabyte, on a 2012 MacBook Pro with four cores and 16 GB of RAM. Runtimes came in between 23 seconds and 34 minutes.

Apache Spark's own home page now describes it as an engine for "single-node machines or clusters". The cluster stopped being the assumption.

So what actually forces distribution? Four things, and dataset size is the weakest of them.

Concurrency. One analyst querying a terabyte is a laptop problem. Four hundred dashboards refreshing against the same terabyte every fifteen minutes is not, because each of them wants memory, CPU and I/O at the same moment. Almost every warehouse migration that gets described afterwards as a volume problem was a concurrency problem.

Ingest rate with a deadline attached. Data arriving faster than one machine can durably write it, with a latency budget on the acknowledgement, is the honest case for a distributed system. Payment authorisation, ad bidding and telemetry from a vehicle fleet sit here. Nightly batch does not, whatever its size.

Retention and reprocessing. If you must be able to recompute three years of history after a bug in a transformation, the cost is not storage but the compute needed to replay it inside a maintenance window.

Failure tolerance as a requirement. A job that must finish even though a machine died halfway through needs a framework that restarts the lost work. That was MapReduce's original selling point and it remains the best reason to accept a cluster.

Most big data projects are ordinary data projects carrying a distributed bill.

Big data basics: the Vs and what they are for

The three-V framing is older than the phrase big data. It is normally traced to a 2001 META Group research note by Doug Laney on managing data volume, velocity and variety. Gartner absorbed META Group in 2005 and no longer serves that document, so treat the attribution as industry consensus rather than something you can open and read. Two more Vs were bolted on later and vendor decks have kept adding them. Five is the working set; the rest is marketing.

• Volume: the size. The V everyone quotes and the one that constrains least.
• Velocity: the rate at which records arrive and the deadline attached to each one. Velocity is the V that forces architecture, because a stream that never stops cannot be served by a job that runs at 3 a.m. The design question is not where to put the records but what you are allowed to do before acknowledging them. Kafka and Flink exist because of this V, and so does most of the operational pain in the field.
• Variety: structured tables, semi-structured JSON, logs, and unstructured text, images and audio in one pipeline. In practice variety means schema drift. The field that was an integer last quarter is a string this quarter, nothing announced it, and the job that fails loudly is the lucky case.
• Veracity: whether the records are true. Large datasets are duplicated, stale, partially collected and quietly wrong. This is where mining projects die, and it appears in no architecture diagram ever drawn.
• Value: whether any of it is worth the storage bill. The price table further down is what that bill looks like.

Four questions do more work than the list. How much data arrives per second, and what is the deadline on it? How many people and services query it at once? How far back must you be able to recompute? What does one full scan cost in dollars? Answer those and the architecture mostly falls out of them.

What data mining actually is

Data mining is the discovery of useful, non-obvious patterns in data using statistics, machine learning and database techniques. It is the analysis step, the part that produces "customers who buy X churn unless they adopt Y", or "these transactions look fraudulent", or "these documents belong to the same topic". The techniques are old, well understood, and largely untouched by the last decade of hype.

• Classification assigns records to known categories: spam or not, will churn or will not.
• Clustering groups similar items with no labels supplied in advance. Customer segments and related-product sets come from here.
• Regression predicts a continuous number: price, demand, lifetime value.
• Anomaly detection flags the records that do not fit, usually fraud, intrusions or sensor faults. The algorithm is never the hard part. The base rate is. At one fraudulent transaction in ten thousand, a model with 99% accuracy loses to one that always answers no, and reporting accuracy on a problem like this is a way of not answering the question.
• Association rule mining finds things that co-occur, and it carries the most-repeated anecdote in the whole field.

That anecdote is mostly fiction, and it is worth correcting because it appears in textbooks. Daniel Power traced it in a 2002 DSSResources newsletter. In 1992 Thomas Blischok, then running a retail consulting group at Teradata, analysed about 1.2 million market baskets across roughly 25 Osco Drug stores and found beer and diapers bought together between 5 and 7 p.m. Osco managers did not act on it. Nobody moved the shelves. One participant quoted in that account puts it plainly: "Our job was to sell the value of systems. Sometimes in selling, fact blurred with folklore." A generation of course material turned an unexploited correlation into a case study in profit.

For the techniques themselves in depth, see our guide to data mining. The boundary between mining and modern machine learning is thin and frequently argued about, and we take it apart in data mining vs. machine learning.

Big data vs. data mining: the difference in one table

Big data is the mine and data mining is the digging. You can hold an enormous pile and extract nothing from it, and you can mine a small dataset beautifully.

Where the machinery came from

MapReduce

The pattern that made cluster processing routine was MapReduce, described by Jeffrey Dean and Sanjay Ghemawat in a paper at OSDI'04. The paper reported that upwards of a thousand MapReduce jobs ran on Google's clusters every day. Three stages, running in parallel across many machines:

1. Map: each worker reads a slice of the input and emits key-value pairs.
2. Shuffle: the framework groups all values by key.
3. Reduce: each key's group is combined into a final result.

Counting words across a document set is the canonical example. Map emits (word, 1) for every word, Reduce sums per word. The point was never the word count. The programmer wrote two functions and the framework handled distribution, parallelism and recovery when a machine died mid-job. That last part is what made racks of cheap hardware viable.

Hadoop, and who actually built it

MapReduce needed a filesystem underneath it. Hadoop paired an open MapReduce engine with the Hadoop Distributed File System, a reimplementation of the ideas in Google's file-system paper. HDFS splits files into large blocks, replicates each block across nodes on different racks, and keeps serving while machines fail. Fault tolerance plus data locality, moving computation to the data instead of the data to the computation, is what let commodity machines process web-scale datasets at a price anyone could pay.

One correction to the version of this article that stood here before, and to a great many other write-ups: Hadoop was not built at Yahoo. Doug Cutting and Mike Cafarella wrote it inside the Nutch search-engine project, and it was split out into its own Apache subproject in January 2006, with release 0.1.0 following on 1 April 2006. Cutting worked at Yahoo at the time and Yahoo put serious engineering behind it, but the code came out of an open-source crawler. The name came from Cutting's son's toy elephant. By February 2008 Yahoo was running its Search Webmap on a Hadoop cluster of more than 10,000 cores.

The generational jump shows up cleanly in one benchmark. Sort Benchmark records a 2013 Daytona GraySort result from Yahoo of 102.5 TB in 4,328 seconds on 2,100 nodes. In 2014 a Databricks team running Spark sorted 100 TB in 1,406 seconds on 207 EC2 nodes, three times the throughput on a tenth of the machines. The category record has not moved since 2016, when Tencent Sort managed 100 TB in 98.8 seconds. A benchmark nobody has bothered to beat in a decade tells you where the industry's attention went.

We walk through the storage side of this architecture in distributed web scraping and MapReduce.

What is actually alive in August 2026

Raw MapReduce is legacy. The Hadoop project is not, and the two get confused constantly. Apache Hadoop shipped 3.5.0 on 2 April 2026, the first stable release of the 3.5 line, with 485 fixes and improvements since 3.4. HDFS and YARN are maintained and widely deployed. What died is hand-writing map and reduce functions in Java.

The rest of the ecosystem, read from project release pages on 10 August 2026:

• Apache Spark is the general-purpose engine, at 4.2.0 since 14 July 2026, with the 3.5 line still patched (3.5.9, 16 July 2026). Spark 4.0 made ANSI SQL mode the default, added a VARIANT type for semi-structured data and shipped a 1.5 MB Python client for Spark Connect. It also removed Mesos support and deprecated SparkR, which corrects something the earlier version of this page said: Spark's first-class languages are Python, SQL, Scala and Java, and R is on the way out.
• Apache Kafka reached 4.0 on 18 March 2025 and became the first major release to run entirely without ZooKeeper. A runbook that still describes a ZooKeeper ensemble describes a system you cannot upgrade in place.
• Apache Flink is at 2.3.0, released 25 June 2026, and remains the default for stateful stream processing.
• Apache Cassandra shipped 5.0.9 on 7 August 2026, three days before this sentence was written, along with patches to the 4.1 and 4.0 lines. Apache HBase reached 3.0.0 on 5 August 2026. MongoDB is at 8.3.
• scikit-learn is at 1.9.0 and stays the single-node default for mining work that fits on one machine, which, per the section above, is most of it.

The dead list is longer and more instructive. The Apache Attic, where retired projects go to be archived, holds Sqoop, Crunch, Tajo, Falcon, Apex, Metron, Hama, Lens, Eagle, Giraph, Chukwa and Twill. Sqoop, the standard way to move data between Hadoop and relational databases for most of the 2010s, was retired in June 2021. Apache Pig went eight years between releases: 0.17.0 in June 2017, then 0.18.0 on 15 September 2025. Apache Mahout, the Hadoop-era machine learning library in every big data syllabus written before 2018, now presents itself as the home of Qumat, a Python library for quantum computing. Inside Spark, the RDD-based spark.mllib API has been in maintenance mode since Spark 2.0 and GraphX is deprecated.

The live argument is over table formats, not engines. Apache Iceberg is at 1.11.0 and is readable by Spark, Flink, Trino, Snowflake, BigQuery, Redshift, Athena, ClickHouse and DuckDB. Databricks, which created Delta Lake, acquired Tabular in June 2024, hired Iceberg's original authors and committed publicly to converging the two formats. Both are Parquet files with a metadata layer on top. If you are choosing storage today the format question matters less than it did two years ago, and the catalog question matters more.

The prices that decide the architecture

Architecture arguments are usually budget arguments wearing a lab coat. These figures were read from each vendor's own pricing page on 10 August 2026, for US East regions.

One of those lines is a correction. BigQuery on-demand querying is quoted at $5 per terabyte across an enormous number of articles, tutorials and cost calculators. It has not been $5 for a while. The published rate is $6.25 per TiB. Snowflake's consumption table carries an effective date of 7 August 2026, three days before this article, which tells you how much confidence a two-year-old price in a blog post deserves.

Now the arithmetic that decides designs. A dashboard query scanning a 2 TiB table costs $12.50 per run on BigQuery on-demand. Refresh it hourly and that is roughly $9,000 a month for one dashboard. Partition and cluster the same table so the query touches 20 GiB and the same dashboard costs about $88 a month. The numbers on screen are identical either way. The whole difference lives in the physical layout, which is why "put it in the warehouse" is not an architecture.

Storage is the cheap part, and it is the part people optimise first anyway. A terabyte parked in S3 Standard runs about $24 a month. Scanning a tebibyte once in BigQuery costs $6.25. Run one full scan a day for a year and you have spent $2,281 reading data that cost $283 to store.

The word-count job, 2004 and 2026

The Java MapReduce word count was the hello-world of this entire field. Here it is in PySpark, written against Spark 4.2.0:

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode, split, lower, count

spark = SparkSession.builder.appName("word-count").getOrCreate()

df = spark.read.text("s3a://my-bucket/documents/*.txt")

counts = (
    df.select(explode(split(lower(df.value), r"\W+")).alias("word"))
      .filter("word <> ''")
      .groupBy("word")
      .agg(count("*").alias("n"))
      .orderBy("n", ascending=False)
)

counts.write.parquet("s3a://my-bucket/word-counts/")

Spark spreads that across the cluster and survives node failures while doing it. If the corpus is a few hundred gigabytes and the job is a one-off, the honest answer is that you do not need a cluster at all:

SELECT word, count(*) AS n
FROM (
  SELECT unnest(regexp_split_to_array(lower(content), '\W+')) AS word
  FROM read_text('documents/*.txt')
)
WHERE word <> ''
GROUP BY word
ORDER BY n DESC;

DuckDB runs that on a laptop. Choosing between the two blocks is the real engineering decision, and it is settled by the four questions above rather than by the word big.

What breaks when you run it on real data

Small-data intuitions fail in specific, repeatable ways. These are the ones that cost the most.

Skew. A distributed job finishes when its slowest task finishes. Group by customer and one customer holding 4% of the rows keeps the whole job open while everything else idles. The symptom is a stage where 1,999 tasks finish in seconds and one runs for an hour. The fixes are salting the key or broadcasting the small side of a join, and neither helps until somebody looks at the task duration histogram instead of the total.

Small files. Ten million 4 KB objects and one 40 GB object hold the same bytes and cost wildly different amounts to work with. Listing and opening dominates, and on S3 every one of those opens is a GET at $0.0004 per thousand plus its own latency. Compaction into files of a few hundred megabytes is unglamorous and is usually the largest single speedup available.

Shuffle is the bill. Map and filter are nearly free, because they run where the data already sits. Anything that regroups rows across the network moves them between machines and spills to disk when memory runs out: joins, group-bys, sorts, window functions. Sizing a cluster by CPU count when the job is shuffle-bound is a common and expensive mistake.

Sampling is smart, not lazy. You rarely need every row to find a pattern, and a well-drawn sample trains a model in minutes instead of hours. The failure mode is sampling before deduplication, which bakes the duplicate rate into the sample and hides it from every check afterwards.

Leakage through time. This is the mining-specific failure and it survives code review every time. Join a feature computed on Tuesday to a label from Monday and the model learns the future. Offline accuracy looks superb, production accuracy collapses, and nothing in the test suite fires, because the code is correct and the semantics are not. Every feature needs an as-of timestamp and every training join has to respect it.

Preparation, and the number everyone quotes about it. "Data scientists spend 80% of their time on data preparation" appears in roughly every article on this subject. It traces to a 2016 CrowdFlower survey of about eighty data scientists asked to estimate their own time: 60% cleaning and organising, 19% collecting. That is a self-report from a sample of eighty, recycled for a decade as if it were a measurement. Preparation does dominate the work. Nobody has established by how much, and quoting the number as though somebody has is the same habit that produced the beer and the diapers. The mechanics are covered in data normalization and data enrichment. Garbage at petabyte scale is still garbage, and it costs more to store.

Common use cases

• Retail and e-commerce. The densest user of both halves. Market-basket analysis, recommendation, demand forecasting and competitor price monitoring, where the volumes are modest and the collection problem is the hard part.
• Finance. Fraud detection, credit scoring and risk models over complete transaction histories, under regulatory pressure to explain every individual decision.
• Telecom and industrial IoT. Churn prediction and predictive maintenance from sensor streams, and the clearest genuine velocity problem on this list.
• Marketing. Segmentation and targeting across web, app and CRM data.
• Search and web. Trend detection, topic clustering, log analysis at scale.

Where the raw material comes from

A mining project is bounded by its inputs, and the valuable datasets are rarely handed over neatly. Internal systems cover one part: CRM records, transactions, logs. The rest comes off the open web as prices, listings, reviews, company and contact data, collected through web scraping and then cleaned, deduplicated and joined into something a model can read.

That collection layer is what teams underestimate, because it breaks weekly rather than annually. Sites change markup, add challenges, rate-limit and go down, and the parser that worked in March is not the parser you need in August. Some teams run it themselves. Others hand the layer to a managed extraction service and take delivery against a fixed schema. Where the requirement is a continuing feed rather than a one-off pull, the same work is normally bought as data as a service.

Two boundaries are worth knowing before pointing a collector at anything. Public availability is not a legal basis under the GDPR. The European Data Protection Board's Opinion 28/2024, adopted on 18 December 2024, sets out what a legitimate-interest argument for personal data in AI models would have to establish, and "it was visible on a page" is not it. Separately, the EU Data Act, Regulation 2023/2854, has applied since 12 September 2025 and reworks the switching rules between cloud data-processing providers, which matters if your warehouse choice is meant to be reversible. None of this is legal advice and all of it moves.

Where that leaves big data and data mining

Big data is the scale and the infrastructure. Data mining is the analysis that extracts value from it. MapReduce and Hadoop proved that fault-tolerant, web-scale processing was possible on cheap machines, and that proof is now twenty years old and largely of historical interest. The working stack in 2026 is Spark, object storage with Iceberg or Delta on top, a cloud warehouse, a streaming layer where velocity demands one, and scikit-learn or DuckDB on a single node for a surprising share of the actual analysis.

The technology stopped being the hard part. Storing and querying a few terabytes is a solved problem with a published price list attached. What has not become easier is holding data that is complete, current and true, knowing where each field came from, and asking a question whose answer would change a decision. Those three are the whole job now.