Streaming pipelines and Kafka

This lecture builds a real-time pipeline in five stages (ingest, buffer, process, store, serve). It covers Spark Streaming in both APIs (DStreams and Structured Streaming). After this page you can trace output modes, place Kafka partitions, and size consumer groups.

Stream Aggregations, Time Windows, WatermarksVanrish Technology · Windows and watermarks

Kafka Topics, Partitions & Consumer GroupsMaxcoTec Learning · Topics, partitions, consumer groups

Kafka Consumer Groups ExplainedConfluent, an IBM Company · Kafka consumer groups

Apache Kafka Architecture ExplainedSelenium Express · Kafka architecture

Topics, partitions, and offsets in KafkaKnowledge Amplifier · Topics, partitions and offsets

The five pipeline stages Search on YouTube

A real-time pipeline has five stages: sources, ingest and buffer, process, store, and serve and act. Sources are apps, sensors, databases and logs. Ingest and buffer use Kafka, Kafka Connect or Kinesis. Processing uses Spark Structured Streaming or Flink. Storage uses Delta Lake, Cassandra, HBase or Elasticsearch. Serving uses dashboards, APIs and alerts.

Five stages. The log between producers and consumers gives decoupling, fan-out and replay.

Three properties make the log design work. Decoupling: producers and consumers never talk directly. Each side scales and fails on its own. Fan-out: one topic feeds many consumers. The fraud model, the dashboard and the data lake read the same events. Replay: retained events can be reprocessed. Replays fix bugs and backfill new features.

S-I-P-S-S: Sources In, Process, Store, Serve

SOURCES are apps, sensors and logs. INGEST uses Kafka. PROCESS uses Spark or Flink. STORE uses Delta Lake or Cassandra. SERVE sends dashboards, APIs and alerts.

Fraud detection, step by step Search on YouTube

The slides trace one card swipe through the full pipeline. The swipe becomes an event {card, amount, merchant, city, ts}. The producer keys it by card number and appends it to the topic transactions. Keying by card keeps all swipes of one card in one Kafka partition, in order. The two-cities rule then sees them in sequence.

Produce the swipe event to the topic transactions, keyed by card number.
Read the topic with Spark Structured Streaming. Parse the JSON. Set a watermark on ts.
Enrich each event. Join it with a static table of home city and average spend per customer.
Apply the rules. Flag the event if $\text{amount} \gt 5\times\text{avg}$ , or if two cities appear within 10 min.
Send flagged events to the topic alerts for the SMS service. Send all events to Delta Lake.
Show alerts per minute on a dashboard. Retrain the model from Delta Lake later.

The latency budget drives the design. The decision must arrive before the payment is authorised. Every stage must stay fast.

Worked example. Customer c42 has average spend 8000. Two swipes arrive: Kochi at 10:00 for 50000, Delhi at 10:07 for 1200.

Amount rule.

\begin{aligned} \text{ratio} &= 50000/8000 = \mathbf{6.25}\\ 6.25 &\gt 5 \;\Rightarrow\; \text{flag the Kochi swipe} \end{aligned}

Two-cities rule.

\begin{aligned} \text{gap} &= 10{:}07 - 10{:}00 = \mathbf{7\,min}\\ 7 &\lt 10 \;\Rightarrow\; \text{flag the Delhi swipe} \end{aligned}

Both rules fire. The pipeline sends two events to alerts and both events to Delta Lake.

Pipeline design considerations Search on YouTube

Concern	Question to ask	Typical answer
Latency	How old may a result be?	Pick micro-batch vs record-at-a-time; tune the trigger interval
Throughput	Peak events per second?	Enough Kafka partitions and executors; test at burst load
Ordering	Must events of one entity stay in order?	Partition by entity key
Correctness	Can we lose or double-count events?	Checkpoints plus idempotent or transactional sinks
Late data	How late can events arrive?	Event time with a watermark; keep a late-data path
Schema evolution	What if producers add a field?	Avro or Protobuf with a schema registry
Observability	How do we know it is healthy?	Monitor consumer lag, batch duration and error rate

Spark Streaming Search on YouTube

Spark offers two streaming APIs. DStreams is the legacy API (package spark.streaming). New applications should use Structured Streaming. DStreams remain useful to understand micro-batching.

DStreams: a stream as RDD batches Search on YouTube

A DStream is a Discretized Stream: the continuous stream cut into a series of RDDs. The batch interval is set once on the StreamingContext (for example 5 seconds). Every 5 s a new RDD forms and Spark runs the same RDD operations on it.

DStream micro-batching. Live input becomes RDD@t1, RDD@t2, RDD@t3; each batch runs the same job.

DStream operations Search on YouTube

Operation	Type	What it does
`map`, `flatMap`, `filter`	Stateless	Applied to every element of every batch
`reduceByKey`	Stateless	Aggregates by key within one batch
`window(len, slide)`	Windowed	Union of the batches in a sliding window
`reduceByKeyAndWindow`	Windowed	Aggregates by key over a sliding window
`updateStateByKey`	Stateful	Keeps per-key state across batches (running totals)
`pprint`, `saveAsTextFiles`	Output	Triggers execution and emits results
`foreachRDD`	Output	Runs arbitrary code on each batch RDD (write to a database)

DStream word count over a socket Search on YouTube

from pyspark import SparkContext
from pyspark.streaming import StreamingContext

sc = SparkContext("local[2]", "NetworkWordCount")
ssc = StreamingContext(sc, 5)

lines = ssc.socketTextStream("localhost", 9999)
counts = (lines.flatMap(lambda line: line.split(" "))
               .map(lambda w: (w, 1))
               .reduceByKey(lambda a, b: a + b))
counts.pprint()

ssc.start()
ssc.awaitTermination()

Run it with two terminals. Terminal 1 runs nc -lk 9999 and you type sentences. Terminal 2 runs spark-submit wc.py. Every 5 s the counts for that batch print. local[2] gives one thread to the receiver and one to processing. Nothing runs until ssc.start().

Worked example. Batch interval 5 s. Batch 1 receives hello world. Batch 2 receives hello spark. reduceByKey aggregates within one batch only.

Batch 1. Split gives hello, world. Counts: (hello, 1), (world, 1).

Batch 2. Split gives hello, spark. Counts: (hello, 1), (spark, 1).

Batch 2 does not add to batch 1. The running total hello = 2 never appears. For running totals use updateStateByKey or Structured Streaming.

Structured Streaming: the unbounded table Search on YouTube

Structured Streaming treats the stream as a table that grows forever. Each new event is a new row. You write an ordinary DataFrame query. Spark runs it incrementally on every trigger and updates the result table. It keeps the counts as state, so it never rescans old rows.

Worked example. Query: split lines into words, then groupBy(word).count(). Three triggers arrive.

Trigger	New line	Result table
t = 1	cat dog	cat 1 · dog 1
t = 2	dog dog	cat 1 · dog 3
t = 3	owl cat	cat 2 · dog 3 · owl 1

Check. Total words seen: $2 + 2 + 2 = \mathbf{6}$ . Table totals: $2 + 3 + 1 = \mathbf{6}$ . The state matches the input.

Structured Streaming word count from Kafka Search on YouTube

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode, split, col

spark = SparkSession.builder.appName("KafkaWC").getOrCreate()
lines = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "sentences")
    .load()
    .selectExpr("CAST(value AS STRING) AS line"))
words = lines.select(
    explode(split(col("line"), " ")).alias("word"))
counts = words.groupBy("word").count()
query = (counts.writeStream.outputMode("complete")
    .format("console")
    .option("checkpointLocation", "/tmp/ckpt/wc")
    .start())
query.awaitTermination()

readStream replaces read: the same DataFrame API reads a stream. Kafka rows carry key, value, topic, partition, offset and timestamp. Value is bytes, so cast it. complete mode prints the whole result table each trigger. The checkpoint stores offsets and state for recovery. Run it with the Kafka connector package (spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.13:[spark version]).

Output modes and triggers Search on YouTube

\text{complete: write all} \qquad\qquad \text{update: write changed} \qquad\qquad \text{append: write new final rows}

Complete writes the entire result table each trigger. Update writes only rows that changed. Append writes only new, final rows. For aggregates, append needs a watermark: only a closed window is final.

Mode	Writes each trigger	At t = 3 in the word-count trace
Complete	The entire result table	cat 2 · dog 3 · owl 1
Update	Only rows that changed	cat 2 · owl 1 (cat rose 1 to 2; owl is new; dog did not change)
Append	Only new, final rows	Empty here: running counts without a watermark never close

Trigger	Behaviour
default	Next micro-batch starts as soon as the previous ends
`processingTime='10 seconds'`	One micro-batch every 10 s
`availableNow=True`	Process everything available, then stop
`continuous='1 second'`	Experimental low-latency mode

Rule of thumb from the slides: complete for small aggregate tables, update for large ones, append for raw events or finalised windows.

Worked example. Same trace as above. At t = 2 the table is cat 1 · dog 3. At t = 3 the input owl cat arrives.

Complete prints all three rows: cat 2 · dog 3 · owl 1.

Update prints two rows: cat 2 (changed) and owl 1 (new). Dog stays 3, so it is skipped.

Append prints nothing. No row is final yet, because no watermark closes the aggregation.

Event-time windows with a watermark Search on YouTube

from pyspark.sql.functions import from_json, col, window

# raw is the Kafka readStream DataFrame; schema is the JSON schema.
clicks = (raw
  .select(from_json(col("value").cast("string"),
                    schema).alias("c"))
  .select("c.*"))     # page, user, event_time

per_page = (clicks
  .withWatermark("event_time", "10 minutes")
  .groupBy(window("event_time", "5 minutes",
                   "1 minute"), "page")
  .count())

per_page.writeStream.outputMode("update").format("console").option("truncate", False).start()

Parse the Kafka JSON into columns, including event_time. window(5 min, 1 min) builds a sliding window: 5 minutes long, sliding every minute. withWatermark(10 min) accepts events up to 10 minutes late, then finalises and drops state. Without a watermark, window state would grow forever.

\text{watermark} = \text{max event time seen} - 10\,\text{min}

An event older than the watermark is dropped (or sent to a late-data path). An event newer than the watermark is accepted, even if it arrives after events with larger timestamps.

The slides show the two clocks with a game goal. A user scores at 10:00:05, but the phone uploads at 10:03:40. By processing time the goal lands in the 10:03 window: wrong. By event time it lands in 10:00, where it belongs. With a 10-minute watermark it is accepted. An event 2 hours late would be dropped.

Worked example. Watermark delay is 10 min. The largest event time seen so far is 10:30. Two late events arrive.

\begin{aligned} \text{watermark} &= 10{:}30 - 10\,\text{min} = \mathbf{10{:}20}\\ \text{event A at }10{:}25: &\quad 10{:}25 \gt 10{:}20 \;\Rightarrow\; \mathbf{accept}\\ \text{event B at }10{:}16: &\quad 10{:}16 \lt 10{:}20 \;\Rightarrow\; \mathbf{drop} \end{aligned}

Event A is 5 minutes late: accepted. Event B is 14 minutes late: dropped or sent to the late-data path.

Window counts on a timeline (recap)Search on YouTube

Full window theory sits in stream concepts. The recap here uses the slide figure: eight events at $t = 1, 2.5, 4, 6, 7.5, 11, 12, 14$ s.

Worked example. Count the same eight events under three window types.

Tumbling, size 5 s, no overlap. Windows $[0,5)$ , $[5,10)$ , $[10,15)$ hold $\{1, 2.5, 4\}$ , $\{6, 7.5\}$ , $\{11, 12, 14\}$ . Counts: 3 · 2 · 3.

Sliding, size 6 s, slide 3 s. Windows $[0,6)$ , $[3,9)$ , $[6,12)$ , $[9,15)$ hold $\{1, 2.5, 4\}$ , $\{4, 6, 7.5\}$ , $\{6, 7.5, 11\}$ , $\{11, 12, 14\}$ . Counts: 3 · 3 · 3 · 3.

Session, gap 3 s. Gaps are 1.5, 1.5, 2, 1.5, then 3.5, then 1, 2. Only $11 - 7.5 = 3.5 \gt 3$ splits. Sessions: $\{1, 2.5, 4, 6, 7.5\}$ and $\{11, 12, 14\}$ . Counts: 5 · 3.

Check: $3 + 2 + 3 = \mathbf{8}$ , $3 + 3 + 3 + 3 = \mathbf{12}$ (sliding double-counts by design), $5 + 3 = \mathbf{8}$ .

Practice numerical. Events at $t = 1, 3, 4, 8, 9$ s. Give 5 s tumbling counts and 3 s-gap session counts.

Tumbling. $[0,5)$ holds $\{1, 3, 4\}$ = 3. $[5,10)$ holds $\{8, 9\}$ = 2. Counts: 3 · 2.

Sessions. Gaps are 2, 1, 4, 1. Only $8 - 4 = 4 \gt 3$ splits. Sessions $\{1, 3, 4\}$ and $\{8, 9\}$ . Counts: 3 · 2.

Apache Kafka Search on YouTube

Producers append events to a log. The log keeps them durably for a configured time. Any number of consumers read at their own pace by tracking an offset.

Property	Meaning
Durable log	Events sit on disk and are replicated; reading never deletes them
Publish-subscribe	Producers never see consumers; add a consumer without touching producers
Horizontal scale	Topics split into partitions across brokers; add brokers and partitions for throughput
Replay	Reset an offset and reprocess history; fix bugs, backfill tables, test models

Core vocabulary Search on YouTube

Term	Meaning
Broker	A Kafka server that stores partitions and serves clients; a cluster has several
Topic	A named event stream, for example `clicks`; like a table name
Kafka partition	An ordered, append-only log; a topic splits into partitions
Offset	Position of an event inside its partition: 0, 1, 2, …
Producer	Writes events to a topic; the key decides the partition
Consumer	Reads events from partitions and tracks its offset
Consumer group	Consumers that share the work: each partition goes to one member
KRaft controller	Kafka built-in metadata quorum; replaces ZooKeeper

Partitions and keys Search on YouTube

A topic clicks with 3 partitions holds three independent logs. Partition 0 has offsets 0 to 8, partition 1 has 0 to 5, partition 2 has 0 to 7. Each partition keeps growing at its own pace.

\text{partition} = \text{hash}(\text{key}) \bmod p

$p$ = partition count. All events with one key land in one partition, in offset order. Ordering holds within a partition, never across partitions. Old events vanish by age or size (retention.ms), never by being read.

Three partitions of one topic. Offsets order events inside each lane only.

Worked example. Topic with $p = 3$ . Three keys have hash values 7, 8 and 9.

\begin{aligned} 7 \bmod 3 &= \mathbf{1} \quad (3 \times 2 = 6,\ 7 - 6 = 1)\\ 8 \bmod 3 &= \mathbf{2} \quad (3 \times 2 = 6,\ 8 - 6 = 2)\\ 9 \bmod 3 &= \mathbf{0} \quad (3 \times 3 = 9,\ 9 - 9 = 0) \end{aligned}

The three keys spread over all three partitions. All events of the key with hash 7 stay in partition 1, in order.

Consumer groups Search on YouTube

Within a group each partition goes to exactly one consumer, so work splits. Across groups every group gets every event. On the slides the topic orders has 4 partitions. Group A (billing) has two consumers: consumer 1 reads P0 and P1, consumer 2 reads P2 and P3. Group B (analytics) has one consumer that reads all four. A 5th consumer joining group A would sit idle: 4 partitions feed at most 4 members.

Share within, broadcast across. Group A splits P0–P3; group B sees every event.

Worked example. A topic has 6 partitions. One group starts 8 consumers. A second group starts 2 consumers.

First group. 6 partitions feed at most 6 members. Consumers 7 and 8 sit idle.

\text{idle} = 8 - 6 = \mathbf{2}

Second group. It gets every event independently. Its 2 members split the 6 partitions 3 and 3.

Groups never starve each other. Idle consumers only happen inside an over-full group.

Replication and acknowledgements Search on YouTube

Each partition has a replication factor (for example 3): one leader and two followers on different brokers. Producers and consumers talk to the leader. Followers copy its log. Followers that are caught up form the ISR (in-sync replicas). If the leader broker fails, a follower from the ISR becomes the new leader. With min.insync.replicas = 2 and acks=all, a write succeeds only if at least 2 replicas hold it.

`acks`	Producer waits for	Risk
0	Nothing	Events can be lost silently
1	Leader only	Lost if the leader dies before followers copy
all	All in-sync replicas	Safest; slightly higher latency

N-L-E: Nothing, Leader, Everyone

acks=0 waits for Nothing (loss possible). acks=1 waits for the Leader (lost if the leader dies early). acks=all waits for Everyone in sync (safest, slowest).

Exactly-once writes need two switches: enable.idempotence=true removes duplicates from producer retries, and transactions make writes across partitions atomic.

Worked example. Replication factor 3, min.insync.replicas = 2, acks=all. Brokers fail one by one.

One broker down. 2 replicas stay in sync. $2 \ge 2$ : writes succeed.

Two brokers down. 1 replica stays. $1 \lt 2$ : writes fail until a broker returns.

The setting tolerates exactly 1 dead broker. Reads of old committed events still work from the survivor.

Kafka from the command line Search on YouTube

docker run -d --name kafka -p 9092:9092 apache/kafka:latest
docker exec -it kafka bash   # then: cd /opt/kafka/bin

./kafka-topics.sh --bootstrap-server localhost:9092 --create \
  --topic clicks --partitions 3 --replication-factor 1

./kafka-console-producer.sh --bootstrap-server localhost:9092 --topic clicks
./kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic clicks --group demo --from-beginning

./kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic clicks
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group demo

Start single-node Kafka in KRaft mode with Docker. Create the topic with 3 partitions. Produce in one terminal and consume in another. The last two commands inspect the topic and the lag of group demo.

Producer and consumer in Python Search on YouTube

from kafka import KafkaProducer
import json, random, time

p = KafkaProducer(bootstrap_servers="localhost:9092",
    value_serializer=lambda v: json.dumps(v).encode())
for i in range(100):
    r = {"sensor": f"s{i % 3}",
         "temp": round(random.uniform(20, 45), 1),
         "ts": time.time()}
    p.send("temperature", key=r["sensor"].encode(), value=r)
p.flush()

from kafka import KafkaConsumer
import json

c = KafkaConsumer("temperature",
    bootstrap_servers="localhost:9092",
    group_id="alert-service",
    auto_offset_reset="earliest",
    value_deserializer=lambda b: json.loads(b))
for msg in c:
    r = msg.value
    if r["temp"] > 40:
        print("ALERT", r["sensor"], r["temp"],
              "partition", msg.partition, "offset", msg.offset)

The producer keys each reading by sensor id, so one sensor lands in one partition. The consumer joins alert-service and starts from the earliest offset. It prints an alert for any reading with $\text{temp} \gt 40$ . This consumer is the standing query from the pipeline example.

Retention versus log compaction Search on YouTube

Delete policy (default)	Compact policy
Keep events for a time or size limit (`retention.ms = 604800000 (7 days in milliseconds)`)	Keep at least the latest value for each key; drop older values
Older log segments vanish whole	A null-valued event (tombstone) deletes its key
Use for clickstreams, logs, sensor readings	Use for current state, e.g. latest address per customer

Worked example. A topic holds latest city per user. Offsets 0 to 4 before compaction:

Offset	0	1	2	3	4
Before	u1: Kochi	u2: Pune	u1: Kottayam	u3: Delhi	u2: Chennai
After	dropped	dropped	u1: Kottayam	u3: Delhi	u2: Chennai

Compact. Offset 2 supersedes offset 0 (same key u1). Offset 4 supersedes offset 1 (same key u2). Offsets 0 and 1 vanish. The latest value per key survives: 3 events left from 5.

The streaming ecosystem Search on YouTube

Layer	Tools
Ingest	Kafka Connect, Debezium (CDC), Apache NiFi, Apache Flume
Transport	Apache Kafka, Apache Pulsar, Amazon Kinesis, Google Pub/Sub
Process	Spark Structured Streaming, Apache Flink, Kafka Streams, ksqlDB, Apache Beam
Store	Delta Lake, Iceberg, Cassandra, HBase, Elasticsearch, Druid, Pinot
Serve	Grafana, Kibana, Superset, REST APIs, alerts

Transport moves events and stores them durably but computes little. Processing engines read from transport, compute (filter, aggregate, join, window) and write results. A typical stack is Kafka plus Spark, or Kafka plus Flink. Hadoop-era tools from this course still fit: HDFS archives the history, Hive and Spark SQL run batch queries over what the streams land.

Storm was the first widely used open-source stream processor. Flink suits true streaming with strong event-time and state support. Structured Streaming wins when the team already uses Spark for batch: the same DataFrame API covers both. Kafka Streams is a library, not a cluster: embed it in an ordinary Java application.

Kafka Connect, Streams, ksqlDB, Schema Registry Search on YouTube

Kafka Connect moves data with source and sink connectors configured in JSON. Example: stream every new row of a MySQL orders table into a topic, and every topic event into Elasticsearch, with no custom code. Kafka Streams is a Java library for stateful processing inside any app: filter, join, window and aggregate topics. State stays local and is backed up to compacted topics.

ksqlDB is SQL over streams:

CREATE TABLE hot AS SELECT sensor, AVG(temp) AS avg_temp FROM readings WINDOW TUMBLING (SIZE 1 MINUTE) GROUP BY sensor EMIT CHANGES;

This keeps a table of per-sensor one-minute average temperature. Schema Registry stores Avro, Protobuf or JSON schemas per topic and checks compatibility. A producer that adds a field does not break consumers.

End to end: IoT temperature monitoring Search on YouTube

Sensors run a Python producer keyed by sensor id into the topic temperature (3 partitions). Spark Structured Streaming parses the JSON, applies a 10-minute watermark, and averages per sensor over 1-minute windows. Averages above 40°C go to the topic alerts for the notifier. All events land in Delta Lake or Cassandra. Grafana draws the dashboard.

Every concept in this unit appears here: stream model, event time, windows, watermark, micro-batch, partitions by key, consumer groups, checkpoints and fan-out.

Labs on the slides Search on YouTube

Five labs use WSL2 Ubuntu plus Docker. Lab 1 runs single-node Kafka with console producer and consumer in two groups (topics, partitions, offsets, groups, lag). Lab 2 builds the kafka-python sensor producer and alert consumer (keys, partitioning, standing queries). Lab 3 runs the DStream socket word count, then the Structured Streaming word count (micro-batch, per-batch vs running state). Lab 4 streams from Kafka with event-time windows and a watermark (windows, late data, output modes). Lab 5 builds the IoT pipeline end to end with a checkpoint and restart test (fault tolerance, exactly-once, fan-out).

Quick self-test Search on YouTube

1. Events arrive at t = 1, 3, 4, 8, 9 s. Give the counts for 5 s tumbling windows and for sessions with a 3 s gap.

Tumbling: $[0,5)$ holds $\{1, 3, 4\}$ = 3; $[5,10)$ holds $\{8, 9\}$ = 2. Counts: 3 · 2. Sessions: gaps are 2, 1, 4, 1. Only $8 - 4 = 4 \gt 3$ splits. Sessions $\{1, 3, 4\}$ and $\{8, 9\}$ . Counts: 3 · 2.

2. A topic has 4 partitions. Keys hash to 5, 10, 11 and 14. Which partition does each key get?

$5 \bmod 4 = \mathbf{1}$ ( $4 \times 1 = 4$ , $5 - 4 = 1$ ). $10 \bmod 4 = \mathbf{2}$ . $11 \bmod 4 = \mathbf{3}$ . $14 \bmod 4 = \mathbf{2}$ ( $4 \times 3 = 12$ , $14 - 12 = 2$ ). Keys hashing to 10 and 14 share partition 2 but keep their own offsets.

3. Differentiate micro-batch from record-at-a-time. Name one engine for each and give the latency of each.

Micro-batch collects events for a short interval, then runs a small batch job. Latency is about the batch interval (for example 1 s). Engine: Spark Streaming and Structured Streaming. Record-at-a-time processes each event on arrival. Latency is milliseconds. Engines: Flink, Storm, Kafka Streams.

4. In Spark, why is append mode not allowed for a running count without a watermark?

Append may write only new, final rows. A running count without a watermark never closes, so no row is ever final. Add a watermark (then closed windows flush) or use update or complete mode.

5. A topic has 6 partitions and a group starts 8 consumers. What happens, and why?

6 partitions feed at most 6 members, so 2 consumers sit idle ( $8 - 6 = \mathbf{2}$ ). Within a group each partition goes to exactly one consumer. Add partitions (or cut consumers) to use them.

6. A consumer commits offsets before processing. Which guarantee is this, and what can go wrong?

This is at-most-once. A crash after the commit but before processing loses the event (counted 0 times). It suits metrics where a gap is tolerable. For no loss, commit after processing (at-least-once) with idempotent writes.

7. Replication factor 3, min.insync.replicas = 2, acks = all. How many broker failures can writes survive? What changes with acks = 1?

Writes need 2 of 3 replicas, so they survive exactly 1 dead broker ( $3 - 2 = 1$ ). With acks=1 the producer waits only for the leader, so writes stay fast but die with the leader if followers lag: lower latency, possible loss.

8. Compact this log: offsets 0–3 hold a:X, b:Y, a:Z, c:W. Which offsets survive, and what does a tombstone do?

Offset 2 (a:Z) supersedes offset 0 (a:X). Survivors: 1, 2, 3 (b:Y, a:Z, c:W). 4 events shrink to 3. A null-valued event (tombstone) deletes its key on a later pass.

9. Watermark delay is 10 min and the largest event time seen is 12:00. An event stamped 11:47 arrives. Is it accepted? What about one stamped 11:55?

Watermark $= 12{:}00 - 10$ min $= \mathbf{11{:}50}$ . 11:47 $\lt$ 11:50: dropped (13 minutes late). 11:55 $\gt$ 11:50: accepted (5 minutes late).

10. Contrast Lambda and Kappa. When would you still choose Lambda?

Lambda runs two paths: a batch layer (accurate, hours old) plus a speed layer (recent, approximate). Kappa runs one streaming path over a replayable log. Choose Lambda when heavy historical work (for example training ML models over years of data) runs more cheaply as batch jobs.

11. Why can MapReduce not process an unbounded stream directly?

It needs a bounded input that is fully stored before map tasks start, and it writes results only when the job ends. It cannot process an open-ended stream event by event. See standing versus ad-hoc queries and the Bloom filter.

A real-time pipeline runs sources → ingest and buffer → process → store → serve and act (S-I-P-S-S). The log gives decoupling, fan-out and replay.
Fraud pipeline: key swipes by card, watermark on event time, join static profiles, flag $\text{amount} \gt 5\times\text{avg}$ or two cities within 10 min, alert plus archive.
DStream = a series of RDDs, one per batch interval (legacy spark.streaming). reduceByKey aggregates within one batch; updateStateByKey keeps state across batches.
Structured Streaming = an unbounded table; DataFrame queries run incrementally per trigger. Word-count trace: cat 1 · dog 1, then cat 1 · dog 3, then cat 2 · dog 3 · owl 1.
Output modes: complete writes all, update writes changed rows, append writes final rows only (aggregates need a watermark). Rule: complete for small tables, update for large ones, append for raw events or closed windows.
Watermark $= \text{max event time seen} - \text{delay}$ . Older events drop. Window counts on the slide figure: tumbling 3 · 2 · 3, sliding 3 · 3 · 3 · 3, sessions 5 · 3.
Kafka: $\text{partition} = \text{hash}(\text{key}) \bmod p$ . Order holds within one partition only. Retention deletes by age or size, never by reads.
Consumer groups share within (one partition per member; extras idle) and broadcast across (every group gets all events).
Replication: leader plus followers, ISR, min.insync.replicas = 2 with acks=all survives 1 dead broker. acks: 0 = nothing, 1 = leader, all = everyone in sync. Idempotence plus transactions give exactly-once writes.
Log compaction keeps the latest value per key; tombstones delete keys. Delete policy suits event history; compact policy suits current state.
Ecosystem: Connect moves data, Streams/ksqlDB compute on topics, Schema Registry guards schemas. Typical stack: Kafka plus Spark or Flink; HDFS, Hive and Spark SQL still serve the archive.

Streaming pipelines and Kafka

The five pipeline stagesSearch on YouTubeBookmarkSaved

Fraud detection, step by stepSearch on YouTubeBookmarkSaved

Pipeline design considerationsSearch on YouTubeBookmarkSaved

Spark StreamingSearch on YouTubeBookmarkSaved

DStreams: a stream as RDD batchesSearch on YouTubeBookmarkSaved

DStream operationsSearch on YouTubeBookmarkSaved

DStream word count over a socketSearch on YouTubeBookmarkSaved

Structured Streaming: the unbounded tableSearch on YouTubeBookmarkSaved

Structured Streaming word count from KafkaSearch on YouTubeBookmarkSaved

Output modes and triggersSearch on YouTubeBookmarkSaved

Event-time windows with a watermarkSearch on YouTubeBookmarkSaved

Window counts on a timeline (recap)Search on YouTubeBookmarkSaved

Apache KafkaSearch on YouTubeBookmarkSaved

Core vocabularySearch on YouTubeBookmarkSaved

Partitions and keysSearch on YouTubeBookmarkSaved

Consumer groupsSearch on YouTubeBookmarkSaved

Replication and acknowledgementsSearch on YouTubeBookmarkSaved

Kafka from the command lineSearch on YouTubeBookmarkSaved

Producer and consumer in PythonSearch on YouTubeBookmarkSaved

Retention versus log compactionSearch on YouTubeBookmarkSaved

The streaming ecosystemSearch on YouTubeBookmarkSaved

Kafka Connect, Streams, ksqlDB, Schema RegistrySearch on YouTubeBookmarkSaved

End to end: IoT temperature monitoringSearch on YouTubeBookmarkSaved

Labs on the slidesSearch on YouTubeBookmarkSaved

Quick self-testSearch on YouTubeBookmarkSaved

On this page