Recap — Async Messaging Patterns

# Async Messaging Patterns Synchronous calls couple services tightly. Async messaging (via queues, topics) decouples them — at the cost of complexity. Pick deliberately. ## Why async Sync: A calls B; A waits for B's response. ``` Service A → (HTTP) → Service B A blocks until B responds. ``` If B is slow or down, A is stuck. Async: A puts a message on a queue; B processes when ready. ``` Service A → [Queue] ← Service B (worker) A returns immediately. ``` A doesn't wait. B processes at its own pace. Use async when: - Operation can be deferred (email, notifications, batch processing). - B might be slow or temporarily unavailable. - Multiple consumers should react to the same event. - Work needs to be load-balanced across many workers. Use sync when: - Caller needs an immediate result (showing data, checkout flow). - Operation must succeed or fail in the request lifetime. - Simpler is fine and queues add complexity for no gain. ## Major messaging systems ### Apache Kafka Distributed log; topics partitioned across brokers; consumers track their position. Strengths: - Massive throughput (millions of messages/sec). - Replay — consumers can re-read history. - Decoupling — many consumers per topic. - Persistence — messages stay for days/weeks. Use cases: event streaming, log aggregation, change-data-capture, real-time analytics. Operational cost: significant (cluster ops, ZooKeeper or KRaft, schema registry). ### RabbitMQ Traditional message broker. Queues, exchanges, routing. Strengths: - Flexible routing (topic exchanges, fanout, direct). - Lower operational complexity than Kafka. - Per-message acknowledgments. Use cases: task queues, traditional async work, microservice communication. ### AWS SQS Managed queue. No ops; pay per message. Strengths: zero ops; integrated with AWS. Cons: AWS-only; less flexible than Kafka or RabbitMQ. ### Redis Streams / Pub-Sub Lightweight pub-sub built into Redis. Strengths: simple; fast; already there if you use Redis. Cons: not as durable; not as feature-rich. ### Google Pub/Sub / Azure Service Bus Cloud equivalents to SQS; managed pub-sub. ## Patterns ### Pattern 1: Work queue Many producers; one queue; many workers consume. Each message processed once. ``` [Web] → [Queue] → [Worker 1, Worker 2, Worker 3] → DB ``` Use case: send emails; process uploads; render reports. Scaling: add more workers. ### Pattern 2: Pub/sub (fan-out) One producer; one topic; many subscribers. Each subscriber gets a copy. ``` [Order Service] → [orders topic] ↓ [Email Service] [Analytics Service] [Inventory Service] ``` Use case: notify multiple services of an event. ### Pattern 3: Event-driven architecture Services emit events; other services react. Loosely coupled. ``` User signed up → "user.signed_up" event → Email Service: send welcome → Analytics: track signup → CRM: create lead ``` Pros: services don't know about each other; new consumers added easily. Cons: harder to trace request flow; eventual consistency. ### Pattern 4: CQRS (Command Query Responsibility Segregation) Separate write path (commands) from read path (queries). ``` Write: command → write DB (normalized, ACID) ↓ events Read: query → read DB (denormalized for fast reads) ``` Use case: read-heavy workloads where read model differs from write model. ### Pattern 5: Event sourcing Store all state changes as events; reconstruct state by replaying. ``` Events: user_created, name_updated, email_changed, ... State: replay all events to get current user ``` Pros: full audit trail; time-travel debugging; CQRS-friendly. Cons: complexity; not for every app. ## Delivery guarantees ### At-most-once Message may be lost; never delivered twice. When OK: metrics, low-importance logs. ### At-least-once Message guaranteed to arrive; may arrive multiple times. Consumer must handle duplicates (idempotency). Most common in production. ### Exactly-once Each message processed once. Hard to implement; expensive. Kafka offers "exactly-once" semantics within Kafka (transactional). Across systems: harder. In practice: design idempotent consumers; use at-least-once. ## Idempotency in consumers Critical for at-least-once. Consumer must handle duplicate delivery without side effects. Patterns: - **Dedup table**: store message IDs; skip if seen. - **Upserts**: write so re-processing is no-op (UPDATE where ID matches). - **Idempotency key**: client provides key; server enforces uniqueness. ## Common async-messaging mistakes 1. **Async when sync is simpler.** Don't overuse. 2. **No idempotency in consumers.** Duplicates corrupt state. 3. **No DLQ (dead letter queue).** Failing messages block the queue forever. 4. **Ignoring ordering.** Some patterns require order (FIFO queues, Kafka partition order). 5. **No monitoring of queue depth.** Backlogs grow silently. ## Takeaway Async messaging decouples services; sync for immediate-response needs. Kafka for