Recap — Streaming Without Drowning

# Streaming Without Drowning Streaming infrastructure (Kafka, Pub/Sub, Kinesis) is powerful and seductive. It's also operationally expensive. Use it deliberately. ## When you actually need streaming Streaming is justified when: 1. **End-to-end latency < 1 minute is required.** Fraud detection, live dashboards for ops, real-time alerts. 2. **Many consumers need the same data.** Pub/sub fan-out makes streaming natural. 3. **Stream-as-data-source.** Some sources are inherently streaming (web events, IoT, logs). 4. **Reprocessing requires replay.** Kafka retains messages; you can replay history. Streaming is NOT justified when: 1. **Hourly batches are fine.** Most analytics don't need < 1 minute lag. 2. **You're chasing trends.** "Real-time analytics" sounds modern but adds cost without business value. 3. **One downstream consumer.** Batch is simpler if it's a single sink. 4. **Small team without ops capacity.** Kafka takes operational effort. Common mistake: building streaming infra "because we might need it" — pays operational tax forever without benefit. ## Kafka concepts (the universal pattern) Kafka is the dominant streaming platform. AWS Kinesis and GCP Pub/Sub are similar with vendor-specific differences. Concepts: ### Topic A named stream. "orders", "page-views", "transactions". ### Partition A topic is split into partitions for parallelism. Each partition is an ordered log. ``` Topic: orders ├── Partition 0: [msg1, msg2, msg3, ...] ├── Partition 1: [msg1, msg2, msg3, ...] └── Partition 2: [msg1, msg2, msg3, ...] ``` Order is guaranteed within a partition, not across. ### Producer Writes messages to a topic. Specifies the partition (by key hash) or lets Kafka decide. ```python from confluent_kafka import Producer p = Producer({"bootstrap.servers": "kafka:9092"}) p.produce( topic="orders", key="user_123", value=json.dumps({"order_id": 456, "amount": 100}), ) ``` ### Consumer Reads messages from a topic. Tracks its offset (position in the log). ```python from confluent_kafka import Consumer c = Consumer({ "bootstrap.servers": "kafka:9092", "group.id": "warehouse-loader", "auto.offset.reset": "earliest", }) c.subscribe(["orders"]) while True: msg = c.poll(1.0) if msg: process(msg.value()) c.commit() ``` ### Consumer group Multiple consumers cooperating; each partition is read by exactly one consumer in the group. ``` Topic: orders (3 partitions) Consumer Group: warehouse-loader ├── Consumer A: partition 0 ├── Consumer B: partition 1 └── Consumer C: partition 2 ``` Adding more consumers parallelizes reading. Limit: one consumer per partition. ### Offset Position in the partition log. Tracks "what have I already processed". Committed offsets persist; on restart, consumer resumes from the last committed offset. ### Retention How long Kafka keeps messages. Default 7 days; can be days, weeks, infinite. Long retention = replay capability ("reprocess all of last month's data"). ## Delivery guarantees Three modes: - **At-most-once**: commit offset before processing. May lose messages. - **At-least-once**: process before commit. May see duplicates. Default for most systems. - **Exactly-once**: requires transactional support; complex; expensive. Most teams operate at-least-once and dedupe downstream. Easier than exactly-once. ## Streaming → warehouse patterns Two common patterns: ### Pattern 1: Kafka → S3 → warehouse Kafka Connect writes events to S3 (every N minutes); warehouse loader picks up files. Pros: simple, decoupled, replayable from S3. Cons: not really real-time (latency = batch interval). ### Pattern 2: Kafka → stream processor → warehouse Flink, Spark Streaming, or Materialize processes the stream and writes to warehouse continuously. Pros: true real-time, supports complex transformations. Cons: more operational complexity. For most analytics use cases, pattern 1 is sufficient and simpler. ## Stream processing frameworks - **Apache Flink** — most mature stream processor; SQL or Java/Python API. Operational complexity. - **Apache Spark Structured Streaming** — Spark's streaming module. Familiar for Spark users; micro-batch model. - **Materialize** — incremental materialized views over Kafka. Postgres-like; simpler than Flink. - **RisingWave** — similar to Materialize; open source. - **Kafka Streams** — Java library inside your app. No separate cluster. For most teams in 2026: Materialize or RisingWave for analytics use cases (simple, SQL-based). Flink for complex stateful processing. ## When NOT to use streaming Symptoms streaming is wrong for you: - Your business case is "we want fresher data" without specifying how fresh. - Hourly batches would be fine but feel slow. - You don't have an engineer who knows Kafka. - You have one consumer of the stream. Most teams that adopt Kafka spend more on operating it than they save in latency. ## Operating Kafka Self-host options: - Apache Kafka (open source) — full control