Apache Spark

This lecture introduces Spark as a fast unified engine for large-scale data processing (in-memory computing, deployment modes, components). It compares Spark with Hadoop MapReduce on speed and design. After this page you can compute disk passes for iterative jobs and explain each Spark component.

Part 5Hortonworks · Spark crash course: GraphX

Understanding Apache SparkIBM StreamSets · Understanding Apache Spark

Why Hadoop was not enough Search on YouTube

Industries used Hadoop MapReduce to analyse massive data sets. Its programming model was simple, scalable, flexible, fault-tolerant and cost-effective. But MapReduce writes intermediate results to disk between every stage. The wait between queries and between job runs stayed high.

Spark started at the UC Berkeley AMPLab in 2009 and was donated to the ASF in 2013 to speed up this model. Spark is not a modified Hadoop. It has its own cluster manager and does not depend on Hadoop. Hadoop is one of several ways to run Spark: for storage (HDFS), for resource management (YARN), or neither. By 2026 most production Spark runs on Kubernetes or a managed cloud platform. Examples are Databricks, EMR, Dataproc and Synapse, with object storage (S3, ADLS, GCS) instead of HDFS.

A cook who washes every pan between courses works like MapReduce. A cook who keeps pans on the stove works like Spark. Iterative recipes suffer most from the washing.

Disk passes: the exam formula Search on YouTube

Each MapReduce iteration reads its input from disk and writes its output back to disk. That is two full disk passes per iteration. Cached Spark reads the data once and reuses it from RAM.

\text{MapReduce disk passes} \approx 2k \qquad\qquad \text{cached Spark disk passes} \approx 1

$k$ = number of iterations over the same data. Each MapReduce iteration costs one read pass plus one write pass. Cached Spark pays one initial read pass and then iterates in memory.

Worked example. A job iterates 10 times over a 50 GB data set. Compare total disk traffic.

MapReduce.

\begin{aligned} \text{passes} &= 2 \times 10 = \mathbf{20}\\ \text{traffic} &= 20 \times 50\,\text{GB} = 1000\,\text{GB} = \mathbf{1\,TB} \end{aligned}

Spark with the data cached.

\begin{aligned} \text{passes} &= \mathbf{1}\\ \text{traffic} &= 1 \times 50\,\text{GB} = \mathbf{50\,GB} \end{aligned}

Ratio. $1000/50 = \mathbf{20}$ . Spark moves twenty times less data through disk here. For $k = 5$ the same steps give $2 \times 5 = \mathbf{10}$ passes vs $\mathbf{1}$ pass.

What Spark is Search on YouTube

Spark is written in Scala. It offers rich APIs in Java, Scala, Python, R and SQL. Developers and data scientists across industry use it. It is an open-source framework for distributed computing.

The slides quote the standard definition:

Apache Spark is a unified computing engine and a set of libraries for parallel data processing on computer clusters. — Chambers and Zaharia, Spark: The Definitive Guide

Spark extends the MapReduce model to more computation types, including interactive queries and stream processing. Its headline feature is in-memory cluster computing, which raises processing speed sharply. RDDs minimise disk reads and writes and recover from any worker node failure through lineage.

Spark covers batch applications, iterative algorithms, interactive queries and streaming in one system. This removes the burden of maintaining separate tools. This "one engine, many workloads" idea is why Spark is called a unified analytics engine. It also underpins the lakehouse architecture (Delta Lake, Apache Iceberg, Apache Hudi), which unifies data warehousing and data lakes.

Speed claims and the fine print Search on YouTube

The slides cite Spark as up to $100\times$ faster than Hadoop MapReduce in memory, and $10\times$ faster on disk. Hadoop is the $1\times$ baseline on both bars. The axis is log-scaled so both bars stay readable.

\text{Spark time} \approx \frac{\text{Hadoop time}}{100}\ \text{(in memory)} \qquad\qquad \text{Spark time} \approx \frac{\text{Hadoop time}}{10}\ \text{(on disk)}

Worked example. A Hadoop job takes 200 minutes. Estimate the Spark time under each claim.

\begin{aligned} \text{in memory:} &\quad 200/100 = \mathbf{2\,min}\\ \text{on disk:} &\quad 200/10 = \mathbf{20\,min} \end{aligned}

Check: $2 \times 100 = 200$ and $20 \times 10 = 200$ . Both estimates divide back to the baseline.

Do not quote 100× and 10× as facts for every job. These figures come from early (2014-era) benchmarks and marketing materials. Real speedups vary heavily by workload. Treat them as the reason Spark got attention.

Evolution and history Search on YouTube

Spark started as a research project at the UC Berkeley AMPLab in 2009. It was open-sourced under a BSD licence in 2010 with the paper "Spark: Cluster Computing with Working Sets". It became an Apache top-level project in 2014 with Spark 1.0.

Databricks used Spark to set a world record sorting a 100 TB data set in 2014 (Daytona GraySort). Spark 2.0 arrived in 2016 with unified DataFrame/Dataset APIs and Structured Streaming. Spark 3.0 (2020) added Adaptive Query Execution and native Kubernetes support. Spark 3.5 brought Spark Connect to general availability in 2023. Spark 4.0 (2025) made ANSI SQL mode the default, required Java 17 or later, removed Mesos and deprecated SparkR. Spark 4.2 is the current stable release in 2026, and 3.5.x (LTS) reaches end of life.

Books and papers on the slides Search on YouTube

Still useful for 1.x fundamentals (RDDs, core concepts): Learning Spark (1st ed.), Karau, Konwinski, Wendell and Zaharia, O'Reilly, 2015. For current Spark the slides recommend Learning Spark, 2nd Edition (Damji, Wenig, Das and Lee, 2020, Spark 3.0). They also recommend Spark: The Definitive Guide (Chambers and Zaharia, 2018). Check the official docs at spark.apache.org/docs/latest for the version in the lab.

Two foundational papers explain why RDDs and lineage-based fault tolerance work. The first is "Spark: Cluster Computing with Working Sets" (Zaharia, Chowdhury, Franklin, Shenker, Stoica, USENIX HotCloud 2010). The second is "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing" (Zaharia et al., USENIX NSDI 2012). They predate DataFrames and Structured Streaming.

Deployment modes Search on YouTube

Three deployment modes matter in 2026. Standalone uses Spark's own built-in cluster manager on top of HDFS or cloud object storage. It is simple to set up and suits dedicated Spark clusters. Hadoop YARN runs Spark as a YARN application. It shares a Hadoop cluster's resources with other frameworks. It is still common on-premises. Kubernetes, the modern default for new deployments, submits jobs directly to a Kubernetes cluster. The driver and executors run as pods. It needs no Hadoop at all and works naturally with S3, ADLS and GCS. Managed platforms (Databricks, EMR on EKS, Dataproc, Synapse) run Spark this way.

Mesos was dropped in Spark 4.0. Spark-in-MapReduce (SIMR), a way to launch Spark without admin access on a Hadoop v1 cluster, is obsolete with Hadoop v1 itself.

Deployment modes. The manager row differs; only Kubernetes needs no Hadoop.

Execution modes Search on YouTube

Execution mode is where the job physically runs. Local mode runs Spark in one JVM on one machine (master("local[*]"), laptop or desktop, Windows or Linux, in an IDE). It suits testing and development. Standalone cluster mode runs Spark on a cluster under Spark's own cluster manager. The slide groups local and Standalone together. Cluster mode with YARN / HDFS runs Spark on an on-premises cluster managed by YARN. It suits large-scale jobs on existing Hadoop infrastructure. Cloud / Kubernetes mode runs Spark as pods on a Kubernetes cluster. The cluster may sit on-prem or in the cloud. A managed platform also works: Databricks, AWS EMR, Google Dataproc or Azure Synapse. Most new large-scale deployments run this way.

Components of Spark Search on YouTube

Spark Core sits at the centre. Four libraries plus an R frontend (SparkR) build on top of it. They are Spark SQL, Structured Streaming (with legacy DStreams), Spark MLlib and Spark GraphX.

Core Serves SQL, Streams, ML, Graphs, R

CORE is the base. SQL queries it. STREAMING reads it live. MLLIB learns from it. GRAPHX links it. R (SparkR) is deprecated.

Component stack. Core schedules and recovers; the rest specialise. SparkR (dashed) is deprecated.

Spark Core Search on YouTube

Spark Core is the central point of Spark. It provides the execution platform for all Spark applications. It handles task scheduling, memory management, fault recovery and interaction with storage systems. It gives a generalised platform so that SQL, Streaming, MLlib and GraphX all build on top of it.

Spark SQL Search on YouTube

Spark SQL runs SQL and HiveQL-style queries on top of Spark. It processes structured and semi-structured data. The Catalyst optimizer plans and rewrites queries. Adaptive Query Execution (AQE, introduced in Spark 3.0 and on by default since Spark 3.2) re-optimizes the plan at runtime with real statistics. The slides claim this runtime step is a large part of why unmodified queries run up to $100\times$ faster in practice.

Streaming in Spark Search on YouTube

Structured Streaming is current and recommended. It builds on the DataFrame/Dataset API. You write the same code as a batch job, and Spark runs it incrementally on a continuous stream. It supports event-time processing and watermarking for late data. Spark Streaming / DStreams is the legacy API. It converts live streams into micro-batches (RDDs) on Spark Core. It still exists in Spark 4.x but is in maintenance mode with no new features. For any new project, use Structured Streaming. Streaming details continue this story.

Spark MLlib Search on YouTube

MLlib is Spark's distributed machine learning library on the in-memory architecture. spark.ml is current and recommended: a DataFrame-based API with Pipelines, Transformers and Estimators, similar in spirit to scikit-learn. New code uses it. spark.mllib is the legacy RDD-based API in maintenance mode. The classic claim that MLlib ran about $9\times$ faster than Hadoop-based Apache Mahout comes from this early RDD implementation. MLlib details follow in the next lecture.

Spark GraphX Search on YouTube

GraphX is Spark's distributed graph-processing framework. It expresses graph computation through the Pregel abstraction, with a runtime optimized for it. GraphX is stable but has seen little active development. GraphFrames, a DataFrame-based community package, is the more actively used alternative for new graph work. Graph details follow later.

SparkR Search on YouTube

SparkR is an R package and a light front end to Spark from R. It lets data scientists analyse large data sets and run jobs interactively from the R shell. Its goal was R usability with Spark scalability. Status update: SparkR was formally deprecated in Spark 4.0 (2025). It still runs but is not the recommended path. Most R users stay on Spark 3.x with SparkR/sparklyr, or move to PySpark or Spark SQL.

Uses of Spark Search on YouTube

Data integration. Data from different systems is often inconsistent, which makes combination hard. Spark runs Extract-Transform-Load (ETL) pipelines faster and more cheaply than traditional tools.

Stream processing. Real-time data (log files, sensor data, clickstreams) is hard to handle well. Spark processes streams continuously through Structured Streaming. It detects anomalies or fraud as they happen.

Machine learning. ML methods grow more feasible and more accurate as data volume grows. Spark caches data in memory and re-runs queries fast. This suits the iterative nature of training.

Interactive analytics. Spark answers queries rapidly. Analysts explore data interactively instead of running only pre-defined batch reports.

Hadoop vs Spark Search on YouTube

	Hadoop MapReduce	Apache Spark
Processing model	Disk-based, batch (MapReduce)	In-memory, batch plus streaming plus interactive
Speed	Baseline	Up to 10–100× faster on iterative workloads
Ease of use	Low-level MapReduce API (verbose Java)	High-level DataFrame/SQL APIs in Scala, Java, Python, R
Real-time processing	Not designed for it	Structured Streaming supports near-real-time pipelines
Fault tolerance	Data replication (HDFS)	Lineage-based recovery (RDDs) plus checkpointing

The 10–100× figure is the early (2014-era) marketing benchmark. It is not the GraySort result below. GraySort gives about 3.1× in time. MapReduce details and HDFS details are separate lectures.

Disk-based vs memory-based frameworks Search on YouTube

Disk-based frameworks recover easily: data on disk survives node restarts with no extra bookkeeping. They suit ETL-like workloads that scan data once. Example: Hadoop MapReduce.

Memory-based frameworks are sensitive to RAM: if data does not fit, it spills to disk and speed degrades. They suit iterative workloads that reuse the same data many times (ML training loops, graph algorithms). Examples: Spark, Apache Flink.

Disk vs memory frameworks. Disk stages pay write plus read per iteration; cached stages reuse RAM.

Why Spark over Hadoop Search on YouTube

Three reasons appear on the slides: performance, development effort and API range.

The GraySort benchmark Search on YouTube

The Daytona GraySort benchmark sorted a 100 TB data set in 2014. Hadoop took 72 minutes on 2,100 on-prem nodes with an HDFS-based shuffle. Spark took 23 minutes on 206 AWS nodes with an in-memory shuffle.

Worked example. Compute the time and node savings from the GraySort figures.

Time ratio.

\begin{aligned} \text{ratio} &= 72/23 = \mathbf{3.13}\\ \text{check:} &\quad 23 \times 3 = 69,\ 23 \times 3.13 = 71.99 \approx 72 \end{aligned}

Node ratio.

\begin{aligned} \text{ratio} &= 2100/206 = \mathbf{10.19}\\ \text{check:} &\quad 206 \times 10 = 2060,\ 206 \times 10.19 = 2099.14 \approx 2100 \end{aligned}

Spark sorted the same 100 TB in roughly one third of the time on roughly one tenth of the nodes.

Less code: word count in two lines Search on YouTube

Classic word count needs dozens of lines of Java MapReduce boilerplate. The slide gives the same job in two lines of Spark Scala:

import org.apache.spark.sql.SparkSession
// spark is the SparkSession
import spark.implicits._
val textFile = spark.read.textFile("path/to/file")
val counts = textFile.flatMap(_.split(" "))
                     .groupBy("value").count()

flatMap splits each line into words. groupBy collects equal words. count counts each group.

Worked example. Trace the two lines on three lines of input: "hello world", "hello spark", "hello world".

Split. flatMap gives six words: hello, world, hello, spark, hello, world.

Group. groupBy collects them: hello appears 3 times, world 2 times, spark 1 time.

Count. count outputs three rows:

value	count
hello	3
world	2
spark	1

Check: $3 + 2 + 1 = \mathbf{6}$ words, matching the six words from the split step.

Rich API range Search on YouTube

Language APIs cover Scala, Java, Python, R and SQL. The low-level API is the RDD. Structured APIs are DataFrame, Dataset and SQL. Spark ships hundreds of built-in, high-level operators for interactive querying. Structured Streaming, machine learning (spark.ml) and the Spark Packages ecosystem complete the range.

Unification timeline. Specialized tools covered one job each; Spark covers all of them.

Standalone setup and line-count code Search on YouTube

A local setup needs JDK 17 or 21. Spark 4.x requires Java 17 or later (Java 8 and 11 no longer work). Spark 3.5.x (LTS) still supports Java 8, 11 and 17. Check the install first:

java -version

Install IntelliJ (IntelliJ IDEA Community Edition, or Eclipse) and Apache Spark for the platform. Add the Scala plugin from the welcome screen. Spark 4.x builds for Scala 2.13, so the project Scala SDK must match the downloaded Spark build. Then confirm a valid JDK is selected for new projects.

Counting lines: legacy RDD vs modern DataFrame Search on YouTube

Legacy RDD API (as in the original deck):

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
  .master("local[*]").appName("LineCount").getOrCreate()
val sc = spark.sparkContext
val lines = sc.textFile("data/sample.txt")
val numLines = lines.count()
println(s"The file has $numLines lines.")
spark.stop()

Modern DataFrame / SparkSession API (recommended starting point today):

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
  .master("local[*]").appName("LineCount").getOrCreate()
val lines = spark.read.textFile("data/sample.txt")
val numLines = lines.count()
println(s"The file has $numLines lines.")
spark.stop()

master("local[*]") runs Spark locally on all cores. textFile loads the file. count counts its lines. stop releases the session. Both versions print e.g. this result (the number depends on the file):

The file has 100000 lines.

Quick self-test Search on YouTube

1. A k-means job iterates 4 times over the same data. How many full disk passes does MapReduce pay, and how many does cached Spark pay?

MapReduce pays $2 \times 4 = \mathbf{8}$ passes (one read plus one write per iteration). Cached Spark pays $\mathbf{1}$ pass (the initial read; the rest reuses RAM).

2. GraySort 2014: Hadoop sorted 100 TB in 72 min on 2,100 nodes; Spark did it in 23 min on 206 nodes. Compute both ratios.

Time: $72/23 = \mathbf{3.13}$ (check: $23 \times 3.13 = 71.99$ ). Nodes: $2100/206 = \mathbf{10.19}$ (check: $206 \times 10.19 = 2099.14$ ). Same data, one third of the time, one tenth of the nodes.

3. A Hadoop batch job takes 90 minutes. Estimate its Spark time in memory and on disk under the slide claims. Give the in-memory answer in seconds too.

In memory: $90/100 = \mathbf{0.9}$ min $= 0.9 \times 60 = \mathbf{54}$ s. On disk: $90/10 = \mathbf{9}$ min. These are rough estimates from 2014-era claims, not guarantees.

4. Trace word count on the input lines "a b", "a c", "a". List each output row and verify the total.

Split gives a, b, a, c, a (5 words). Group: a appears 3 times, b once, c once. Output rows: (a, 3), (b, 1), (c, 1). Check: $3 + 1 + 1 = \mathbf{5}$ , matching the 5 split words.

5. Differentiate disk-based and memory-based frameworks. Name one example of each and state which workload suits each.

Disk-based frameworks write intermediate data to disk; recovery is easy and they suit single-scan ETL work (Hadoop MapReduce). Memory-based frameworks keep data in RAM; they suit iterative reuse such as ML loops (Spark, Flink). If data exceeds RAM it spills and speed degrades.

6. Name the three Spark deployment modes that matter in 2026. Which one needs no Hadoop at all?

Standalone (own manager), Hadoop YARN (shared cluster) and Kubernetes (driver and executors as pods). Kubernetes needs no Hadoop; it pairs with object storage. Mesos was removed in Spark 4.0 and SIMR is obsolete.

7. Differentiate spark.ml from spark.mllib, and Structured Streaming from DStreams.

spark.ml is the current DataFrame-based API with Pipelines, Transformers and Estimators; spark.mllib is the legacy RDD-based API in maintenance mode. Structured Streaming is the current DataFrame-based streaming API with event time and watermarking; DStreams is the legacy micro-batch RDD API in maintenance mode. New projects use spark.ml and Structured Streaming.

8. What do the Catalyst optimizer and Adaptive Query Execution do in Spark SQL?

Catalyst plans and rewrites queries before they run. AQE (introduced in Spark 3.0, on by default since Spark 3.2) re-optimizes the plan at runtime with real statistics. Together they explain much of the speedup on unmodified queries.

9. Why are Mesos and SIMR no longer answers, and what is the status of SparkR?

Mesos support was fully removed in Spark 4.0. SIMR launched Spark on Hadoop v1 clusters, which are no longer used. SparkR was deprecated in Spark 4.0; it still runs but new work moves to PySpark or Spark SQL.

10. Contrast lineage-based recovery with HDFS replication for fault tolerance.

HDFS replication stores several replicas of every block, so recovery reads a surviving replica. Spark lineage records the steps that built each RDD partition, so recovery recomputes only the lost partitions. Lineage avoids paying storage for replicas on every write.

MapReduce writes intermediate results to disk between every stage, so iterative jobs wait on disk. Spark keeps intermediate data in memory.
Disk-pass rule: MapReduce costs about $2k$ passes for $k$ iterations. Cached Spark costs about $1$ pass. Example: 10 iterations on 50 GB means 1 TB vs 50 GB.
Spark is a unified engine for parallel data processing on clusters (Chambers and Zaharia). It is written in Scala with Java, Scala, Python, R and SQL APIs.
Speed claims are $100\times$ in memory and $10\times$ on disk over Hadoop, from 2014-era benchmarks. Real speedups vary by workload.
History: 2009 Berkeley AMPLab and 2010 open source. 2013 donated to the ASF, 2014 Apache top level with Spark 1.0. 2014 GraySort record (100 TB: 72 min on 2,100 nodes vs 23 min on 206 nodes). 2016 Spark 2.0. 2020 Spark 3.0 with AQE. 2025 Spark 4.0. 2026 Spark 4.2 stable.
Deployment modes: Standalone (own manager), YARN (shared Hadoop cluster), Kubernetes (pods, no Hadoop, cloud default). Mesos removed in 4.0; SIMR obsolete.
Execution runs local (test/dev), on a YARN cluster (on-prem scale) or on Kubernetes / managed platforms (how most new deployments run).
Components: Core handles scheduling, memory and recovery. SQL adds Catalyst plus AQE. Structured Streaming is current and DStreams is legacy. spark.ml is current and spark.mllib is legacy. GraphX uses Pregel (GraphFrames is the alternative). SparkR is deprecated in 4.0.
Word count takes dozens of Java MapReduce lines vs two Scala lines (flatMap, groupBy, count).
Line count: spark.read.textFile plus count prints The file has 100000 lines. Spark 4.x needs Java 17 or later and Scala 2.13.

Why Hadoop was not enoughSearch on YouTubeBookmarkSaved

Disk passes: the exam formulaSearch on YouTubeBookmarkSaved

What Spark isSearch on YouTubeBookmarkSaved

Speed claims and the fine printSearch on YouTubeBookmarkSaved

Evolution and historySearch on YouTubeBookmarkSaved

Books and papers on the slidesSearch on YouTubeBookmarkSaved

Deployment modesSearch on YouTubeBookmarkSaved

Execution modesSearch on YouTubeBookmarkSaved

Components of SparkSearch on YouTubeBookmarkSaved

Spark CoreSearch on YouTubeBookmarkSaved

Spark SQLSearch on YouTubeBookmarkSaved

Streaming in SparkSearch on YouTubeBookmarkSaved

Spark MLlibSearch on YouTubeBookmarkSaved

Spark GraphXSearch on YouTubeBookmarkSaved

SparkRSearch on YouTubeBookmarkSaved

Uses of SparkSearch on YouTubeBookmarkSaved

Hadoop vs SparkSearch on YouTubeBookmarkSaved

Disk-based vs memory-based frameworksSearch on YouTubeBookmarkSaved

Why Spark over HadoopSearch on YouTubeBookmarkSaved

The GraySort benchmarkSearch on YouTubeBookmarkSaved

Less code: word count in two linesSearch on YouTubeBookmarkSaved

Rich API rangeSearch on YouTubeBookmarkSaved

Standalone setup and line-count codeSearch on YouTubeBookmarkSaved

Counting lines: legacy RDD vs modern DataFrameSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page