Back to Streaming Mini: Redpanda + Spark

Redpanda & Structured Streaming in 10 Minutes

Broker, topics, connector options, and the windowed-count query.

6 minutesVideo LessonPDF notes

Watch Redpanda & Structured Streaming in 10 Minutes on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Loading progress...

Key moments

Direct Service Calls — Services calling each other creates coupling, failures, and traffic spikes.
Append-Only Log — Kafka writes events to a log and consumers read at their own pace.
Topics and Partitions — Topics are split into partitions for scaling; order is only guaranteed within a partition.
Keys and Ordering — Keys are hashed to partitions; same key preserves order, but adding partitions breaks mapping.
Consumer Offsets and Groups — Consumers track offsets; groups assign partitions for parallel, independent reading.
Replication and Failover — Partitions are replicated; leader failure promotes an in-sync replica.
Performance and Retention — Sequential writes, page cache, zero-copy; retention keeps data for a window.
KRaft and Summary — Kafka 4 uses KRaft instead of Zookeeper; summary of append-only log benefits.

PDF notes

Bottom line: You'll be able to explain how an append-only log broker like Kafka decouples producers and consumers, scales through topics and partitions, and preserves per-key order. You'll also be able to configure consumer groups, offsets, replication, and retention for reliable streaming pipelines.

This lesson on Redpanda & Structured Streaming in 10 Minutes is hands-on and example-driven. You'll be able to explain how an append-only log broker like Kafka decouples producers and consumers, scales through topics and partitions, and preserves per-key order. You'll also be able to configure consumer groups, offsets, replication, and retention for reliable streaming pipelines.

What You'll Be Able To Do

Describe how an append-only commit log decouples producers from consumers.
Explain how topics and partitions enable horizontal scaling.
Choose a message key to preserve per-entity ordering.
Configure a consumer group to parallelize reads without duplicate processing.
Trace how offsets enable replay and independent consumers.
Identify how replication and KRaft keep the cluster available.

Detailed Concept Walkthrough

1. Append-Only Commit Log

Kafka is not a traditional message queue; it is a durable, append-only log where producers add events to the end and consumers read at their own pace by tracking an offset.

Mechanism: When a producer sends an event, Kafka appends it to the end of a partition log and assigns a monotonically increasing offset. The next event gets the next offset, and no event is ever rewritten or deleted just because it was read.
Under the Hood: Appending to the end of a file is a sequential write, which is far faster than random disk seeks. Kafka also relies on the operating system's page cache, so recently written data is often still in memory when a consumer reads it.
Zero Copy: When shipping data to consumers, Kafka uses zero-copy transfer, sending bytes directly from the system's memory to the network socket without copying them into Kafka's own memory.
Best Practice: Treat the log as immutable. Retention is controlled by time or size limits, not by whether a consumer has read a message. This design enables replay and independent consumers.

# Produce events to an append-only log
kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092
> {"order_id": "A1", "status": "created"}
> {"order_id": "A1", "status": "paid"}

# Consume from the beginning and print offsets
kafka-console-consumer.sh --topic orders --from-beginning \
  --property print.offset=true --bootstrap-server localhost:9092

Key Takeaway: Kafka is a fast, durable append-only log; consumers read by tracking an offset, not by receiving per-message acknowledgements.

2. Topics, Partitions, and Keys

A topic is a named stream of events, and it is split into partitions so the data can scale across machines. A message key determines which partition an event lands in, preserving order for that key.

Mechanism: A topic can grow too large for one machine, so Kafka breaks it into partitions. Each partition is an ordered log, and Kafka only guarantees order within a single partition.
Key Hashing: Every event has a value and an optional key. Kafka hashes the key to a number and uses that number to choose a partition. The same key always lands on the same partition, so all events for one order ID stay in order.
Ordering Guarantee: If you need every event for a specific entity processed in sequence, you must use that entity's ID as the key. Without a key, events are distributed round-robin and order is not preserved.
Best Practice: Create extra partitions upfront. If you add partitions later, the hash calculation changes and the same key can start landing on a different partition, breaking ordering.

# Create a topic with 6 partitions and replication factor 3
kafka-topics.sh --create --topic orders --partitions 6 \
  --replication-factor 3 --bootstrap-server localhost:9092

# Produce with a key so all events for order A1 go to the same partition
kafka-console-producer.sh --topic orders --property parse.key=true \
  --property key.separator=: --bootstrap-server localhost:9092
> A1:{"order_id":"A1","status":"created"}
> A1:{"order_id":"A1","status":"paid"}

Key Takeaway: Partition by key to scale horizontally while preserving per-entity order; never add partitions if key ordering matters.

3. Consumer Groups and Offsets

Consumers track a single number—the offset—to know where they are in a partition. Consumer groups let multiple consumers share the work of reading a topic without processing the same message twice.

Mechanism: A consumer periodically commits its offset back to Kafka, which stores it in a small internal topic. After a crash or restart, the consumer resumes from the last committed offset.
Consumer Groups: A consumer group is a set of consumers working together on the same topic. Inside a group, each partition is assigned to exactly one consumer, so all consumers work in parallel without duplicate processing.
Independence: Different consumer groups are completely independent. A fraud detection system can read every order in real time while an analytics job reads the same orders three days later; they never interfere.
Replay: Because data is not deleted on read, you can move an offset backwards to reprocess data. For example, if you shipped a bug, you can replay the last week of orders.
Best Practice: You can never have more active consumers than partitions in a group. If you have 10 consumers and 6 partitions, 4 consumers will be idle.

# Start a consumer group; Kafka assigns partitions automatically
kafka-console-consumer.sh --topic orders --group email-service \
  --bootstrap-server localhost:9092

# Scale by adding another consumer to the same group
kafka-console-consumer.sh --topic orders --group email-service \
  --bootstrap-server localhost:9092

Key Takeaway: Use consumer groups to parallelize reads; each partition is processed by exactly one consumer in the group, and offsets enable replay.

4. Replication and Fault Tolerance

Each partition is copied across several brokers so the log survives machine failures. One broker is the leader for a partition, and the others hold in-sync replicas.

Mechanism: Producers always write to the leader for a partition. By default, consumers read from the leader too, but they can be configured to read from a nearby replica.
Failover: If the leader's machine fails, one of the in-sync replicas is promoted to become the new leader. The cluster keeps running, and consumers continue from the offsets they already committed.
Durability: A replication factor greater than one ensures that data is not lost if a single broker goes down. For critical topics, a replication factor of three is common.
Best Practice: Monitor the in-sync replica (ISR) set. If a replica falls too far behind, it is removed from the ISR and cannot be promoted until it catches up.

# Create a topic with replication factor 3
kafka-topics.sh --create --topic payments --partitions 6 \
  --replication-factor 3 --bootstrap-server localhost:9092

# Describe the topic to see leader and replicas per partition
kafka-topics.sh --describe --topic payments --bootstrap-server localhost:9092

Key Takeaway: Replication keeps the log available; leader election is transparent to consumers because they only track offsets.

5. Retention and KRaft Metadata

Kafka keeps data for a configurable retention window, regardless of whether it has been read. In Kafka 4, cluster metadata is managed by KRaft, a built-in Raft consensus protocol, replacing Zookeeper.

Mechanism: You set retention by time (e.g., 7 days) or by size. Everything inside that window stays, whether or not it has been consumed. This is what enables replay and multiple independent consumers.
KRaft: Kafka used to rely on a separate system called Zookeeper to track brokers and partition leaders. As of Kafka 4, the cluster manages this itself using KRaft, its own implementation of the Raft consensus protocol.
Best Practice: Set retention based on how far back you might need to replay. If you need to reprocess a week of data, retention must be at least a week.
Nuance: Retention is independent of consumer offsets. A consumer can still read data that is older than its committed offset as long as the data has not been deleted by retention.

# Set topic retention to 7 days (604800000 ms)
kafka-configs.sh --alter --topic orders \
  --add-config retention.ms=604800000 \
  --bootstrap-server localhost:9092

Key Takeaway: Retention is time- or size-based and independent of consumption; KRaft removes the Zookeeper dependency in Kafka 4.

Topics Covered in Redpanda & Structured Streaming in 10 Minutes

Direct Service Calls (0:00 - 1:30) — Services calling each other creates coupling, failures, and traffic spikes.
Append-Only Log (1:30 - 3:00) — Kafka writes events to a log and consumers read at their own pace.
Topics and Partitions (3:00 - 4:30) — Topics are split into partitions for scaling; order is only guaranteed within a partition.
Keys and Ordering (4:30 - 6:00) — Keys are hashed to partitions; same key preserves order, but adding partitions breaks mapping.
Consumer Offsets and Groups (6:00 - 7:30) — Consumers track offsets; groups assign partitions for parallel, independent reading.
Replication and Failover (7:30 - 9:00) — Partitions are replicated; leader failure promotes an in-sync replica.
Performance and Retention (9:00 - 10:30) — Sequential writes, page cache, zero-copy; retention keeps data for a window.
KRaft and Summary (10:30 - 12:00) — Kafka 4 uses KRaft instead of Zookeeper; summary of append-only log benefits.

Spark Lakehouse Cheat Sheet

Append-Only Log — Durable log where events are appended and never rewritten

kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092

Topic — Named stream of events, split into partitions

kafka-topics.sh --create --topic orders --partitions 6 --replication-factor 3 --bootstrap-server localhost:9092

Partition — Ordered chunk of a topic for horizontal scaling

kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092

Message Key — Hashed to choose partition; preserves per-key order
```
> A1:{"order_id":"A1","status":"created"}
```

Consumer Group — Set of consumers sharing partition assignments without duplicates

kafka-console-consumer.sh --topic orders --group email-service --bootstrap-server localhost:9092

Offset — Consumer's position in a partition; enables replay

kafka-console-consumer.sh --topic orders --property print.offset=true --from-beginning

Replication Factor — Number of copies of each partition across brokers
```
--replication-factor 3
```
Retention — Time or size limit for keeping events in the log
```
retention.ms=604800000
```

Comparison Table

Aspect	Kafka	Traditional Message Queue
Data model	Append-only commit log	Queue of messages
Read tracking	Consumer offset	Per-message acknowledgement
Replay	Rewind offset to reprocess	Usually not possible
Scaling	Partitions + consumer groups	Often single consumer
Retention	Time or size based	Delete on read

Common Pitfalls

Mistake: Adding partitions after keyed data is written. Avoid: Create extra partitions upfront to keep key-to-partition mapping stable.
Mistake: Assuming global ordering across a topic. Avoid: Use a key and rely on per-partition ordering only.
Mistake: Putting more consumers than partitions in a group. Avoid: Match consumer count to partition count for full parallelism.
Mistake: Treating Kafka as a queue that deletes on read. Avoid: Set retention and use offsets to replay data.
Mistake: Forgetting replication factor for critical topics. Avoid: Use replication factor 3 and monitor in-sync replicas.

FAQs

What happens if the email service is down? With Kafka, the order service just writes an event; the email service can catch up later from its committed offset.
Can two consumers in the same group read the same partition? No, each partition is assigned to exactly one consumer within a group.
How do I reprocess last week's orders? Move the consumer offset backwards and read again; data remains until retention expires.
What is KRaft? Kafka 4's built-in Raft consensus for cluster metadata, replacing Zookeeper.
Does Kafka delete messages after they are read? No, messages stay until the retention window or size limit is reached.
Why is Kafka fast if it writes to disk? Sequential appends, page cache, and zero-copy avoid random disk I/O and extra memory copies.

Common pitfalls

Mistake: Adding partitions after keyed data is written. Avoid: Create extra partitions upfront to keep key-to-partition mapping stable.
Mistake: Assuming global ordering across a topic. Avoid: Use a key and rely on per-partition ordering only.
Mistake: Putting more consumers than partitions in a group. Avoid: Match consumer count to partition count for full parallelism.
Mistake: Treating Kafka as a queue that deletes on read. Avoid: Set retention and use offsets to replay data.
Mistake: Forgetting replication factor for critical topics. Avoid: Use replication factor 3 and monitor in-sync replicas.

Let's say you're running a web application, and behind it you've got a lot of services. There's an order service, a payment service, an email service, an analytics engine, and a warehouse system that needs to know what item to ship. A customer places one order, and five different parts of your system need to react to it. So, how do they find out about it? The obvious answer is to wire them together. When an order comes in, the order service calls the payment service, then calls the email service, then notifies analytics, and then pings the warehouse. For a small system, this is completely fine, and I would say that you should not reach for anything fancier. But, the problems show up as you grow. Your order service now has to know about every other system that cares about orders. Every time someone builds a new feature that needs order data, they come to you and ask you to add one more call. The service that's supposed to handle orders is slowly turning into a hub that every other system plugs into. There's also the question of what happens when something goes wrong. If the email service is down, should the whole order fail because of it? When analytics is running slow, why is the customer waiting three extra seconds for a response completely unrelated to their transaction? Or even worse, the moment you hit a traffic spike, every downstream service gets hit with that same spike at the same moment, because each call goes straight through with nothing in between to absorb it. In that situation, what you really want is for the order service to stop caring about others. It should be able to announce that an order happened, and then immediately take care of its stuff. Anyone who's interested can deal with that event in their own time, and that's the exact problem Kafka was built to solve. Instead of services calling each other, producers write events into Kafka, and consumers read them back out. The producer has no idea who's listening, or whether anyone is listening at all. It sits in the middle and holds onto everything that comes through. So far, this probably sounds like an ordinary message queue, and there are plenty of those already. What makes Kafka different is one decision about what it actually is. Basically, Kafka is an append-only commit log. When a producer sends an event, Kafka writes it to the end of a file and stamps it with a number called an offset. The next event gets the next offset, and the one after that gets the next, on and on. You never go back and rewrite something in the middle, and you never delete a message just because somebody read it. Events pile up in the order they arrived. Oldest at the beginning, newest at the end. That sounds almost too simple, but actually that's the whole point. A normal message queue has to track every message for every consumer, remembering which ones were delivered and which ones got acknowledged. Kafka doesn't do any of that per message bookkeeping, it just keeps the log. As your application grows, keeping everything on one machine isn't enough anymore. You can probably guess what I'll say, and you'd be right. We have to spread the data across multiple machines. Kafka does this under the hood, and it starts with how it organizes your events. We have two ideas here, topics and partitions. A topic is just one stream of events, for example, all of your orders or all of your transactions. A single topic can quickly become way too big for one machine, so Kafka breaks each topic into smaller chunks, and each of those is a partition. By splitting data into partitions, you get the ability to scale, but there is one problem. Kafka only promises that events stay in order within a single partition. So, when ordering matters to you, say you need every event for one specific order to be processed in sequence, those events have to land in the same partition. To solve that problem, every event you send has two parts, a value and a key. The value is the actual content. The key is usually an ID that identifies what the event is about, like the order ID. When a new event appears, Kafka takes its key, runs it through a hash function to turn it into a number, and that number decides which partition the event goes to. The important thing is that the same key always lands on the same partition. So, every event tagged with the same order ID goes to one partition, and thanks to that, order is preserved. There's one catch you should know about. This only works as long as the number of partitions stays the same. If you add partitions to the topic later, the hash calculation changes and the same key can suddenly start landing somewhere else. That's why people usually create extra partitions from the start rather than adding them later. Now, let's look at the reading side. When a consumer reads data from partition, it only has to track one single thing, the offset. Kafka doesn't track individual messages for each consumer and there are no per message acknowledgements anywhere. The only thing that gets saved is that one number. Periodically, the consumer commits its offset back and Kafka stores it in a small internal topic. So, after crash or a restart, the consumer can pick up exactly where it left off. Because each consumer keeps its own offset, your consumers are completely independent of each other. Your fraud detection system can read every order the moment it comes in while your analytics job reads those same orders 3 days later for a report. They never interfere with each other. It also lets you replay data when you need to. For example, you shipped a bug and want to reprocess the last week of orders. You just move your offset backwards and read everything again. The data is still sitting there because nothing gets deleted when a consumer reads it. To scale the reading itself, Kafka uses consumer groups. A consumer group is just a set of consumers working together on the same topic. Inside a group, each partition goes to exactly one consumer. So, if a topic has six partitions and your group has three consumers, each consumer handles two partitions and all three work in parallel. If you add more consumers, Kafka reassigns the partitions to share the load. One important thing is that you can never have more active consumers than partitions. So, if you put 10 consumers in a group reading from six partitions, four of them will have nothing to do. To see why this matters, let's say your email service gets overwhelmed by a spike in orders. In that case, you might want to spin up a second copy to help handle the load. But if both copies read from the topic at the same time, your customer would get two identical confirmation emails. To fix this, you just put them both into the same consumer group. Kafka automatically splits the work, so each order is handled exactly once, and your customer gets only one email. For any of this to survive a machine going down, each partition is copied across several brokers. One broker is a leader for a given partition, and the others hold replicas that stay in sync behind it. Producers always write to the leader, and by default, consumers read from it, too. But, they can also be set up to read from a nearby replica. If the leader's machine falls over, one of the in-sync replicas gets promoted to take its place, and the cluster keeps running. You don't lose a log, and the consumers simply carry on from the offsets they already had. The part that surprised people is how fast all of this is. Kafka writes everything to disk, and that's normally slow. But, appending to the end of a log is a sequential write, and sequential writes are far faster than hopping around the disk to random spots. Additionally, Kafka relies on the operating system's memory called page cache. So, data that was just written is usually still sitting in memory when a consumer comes along to read it. When Kafka does ship that data out, it uses a technique called zero copy, where the bytes go straight from the system's memory to the network socket without ever touching Kafka's memory. But, how long does Kafka actually keep all this data? Basically, you just set a retention window. For example, 7 days or a size limit, and everything inside that window stays whether or not it has been read. The last piece is how the cluster keeps track of itself. It needs to know which brokers exist, and which one is the leader for each partition. Kafka used to hand this job to a separate system called Zookeeper, but as of Kafka 4, they got rid of it. The cluster now manages all of that itself using KRaft, its own built-in take on the Raft consensus protocol. To summarize everything, Kafka is basically just a really fast append-only log. Instead of a messy web of direct calls, you just drop an event into the stream and move on. The rest of your system handles it whenever it's ready. If you enjoyed this video, drop a like and subscribe. And please tell me in the comments what topic you want to see next. I'll pick the most liked one and make a video about it. Thanks for watching and see you in the next one.

Frequently asked questions

What happens if the email service is down?

With Kafka, the order service just writes an event; the email service can catch up later from its committed offset.

Can two consumers in the same group read the same partition?

No, each partition is assigned to exactly one consumer within a group.

How do I reprocess last week's orders?

Move the consumer offset backwards and read again; data remains until retention expires.

What is KRaft?

Kafka 4's built-in Raft consensus for cluster metadata, replacing Zookeeper.

Does Kafka delete messages after they are read?

No, messages stay until the retention window or size limit is reached.

Why is Kafka fast if it writes to disk?

Sequential appends, page cache, and zero-copy avoid random disk I/O and extra memory copies.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.