Recap — Load Balancer Strategies and Gotchas

# Load Balancer Strategies and Gotchas Load balancers (LBs) are the front door to most scaled systems. They distribute incoming requests across backend instances. Sounds simple; many details matter. ## Where LBs sit ``` Internet ↓ [Load Balancer] ← public-facing ↓ [App Server 1] [App Server 2] [App Server 3] ``` Or in tiers: ``` External LB → Web Tier → Internal LB → API Tier → Internal LB → DB Tier ``` LBs at multiple layers; each scales its tier. ## Layer 4 (L4) vs Layer 7 (L7) ### Layer 4 (transport) Operates on TCP/UDP. Sees IP addresses and ports, not the HTTP content. ``` Client → LB sees: TCP packet from 1.2.3.4 → forward to backend X ``` Pros: - Fast. Doesn't parse content. - Works with any protocol (HTTP, WebSocket, gRPC, custom). - Lower latency. Cons: - Can't route based on URL or headers. - Can't terminate TLS centrally (each backend does it). Tools: AWS NLB, HAProxy (L4 mode), nginx (L4). ### Layer 7 (application) Operates on HTTP. Parses URL, headers, cookies. ``` Client → LB sees: GET /api/users → route to api-service-X GET /admin → route to admin-service ``` Pros: - URL-based routing (/api → API service; /static → CDN). - Header-based decisions (geo, A/B tests). - TLS termination at LB; backends use plain HTTP. - Can cache, compress, modify requests. Cons: - More CPU per request (parsing). - Tied to HTTP/HTTPS protocols. Tools: AWS ALB, nginx (HTTP mode), Envoy, HAProxy (L7), Traefik. **In practice**: most public-facing LBs are L7 for the routing flexibility. Internal LBs can be L4 for speed. ## Load balancing algorithms ### Round robin Distribute requests in turn: 1, 2, 3, 1, 2, 3... Simple; assumes all backends are equally capable. Fine default. ### Least connections Send to the backend with fewest active connections. Better when request durations vary. Long-running requests don't pile up on one server. ### Weighted (round robin or least connections) Assign weights. Backend with weight 3 gets 3x as many requests as weight 1. Use when machines have different capacities. ### IP hash / consistent hash Hash the client IP (or session ID); always route to the same backend. Used for: session stickiness, stateful caching by user, certain hash-based partitioning. Trade-off: when a backend dies, its share redistributes; with naive hashing, ALL hashes shift. **Consistent hashing** minimizes the redistribution (only the dead backend's share moves to its neighbor). ### Least response time Send to the fastest-responding backend. Adaptive; useful for varied workloads. ### Random Random backend. Works surprisingly well in practice; less coordination overhead. ## Sticky sessions (session affinity) When a user's requests must keep going to the same backend (in-memory session, cache): - LB sets a cookie identifying the backend. - Or hashes client IP / session ID. Cons: - Uneven load (some users heavy; their backend overloaded). - Backend failure loses the session. - Defeats horizontal scaling somewhat. Modern preference: externalize session state (Redis, DB); make backends stateless; don't need stickiness. ## Health checks LB pings backends regularly: - Active: HTTP request to `/health` endpoint. - Passive: monitor response codes; unhealthy = remove from pool. Important details: - **Frequency**: every 5-30 seconds typical. - **Threshold**: 3 consecutive failures = unhealthy (avoid flapping). - **Endpoint**: separate from main logic; cheap to compute. - **Deep vs shallow**: shallow checks "process alive"; deep checks "DB reachable" — the latter cascades failures (DB outage → all backends marked unhealthy → no traffic at all). ## High availability for the LB itself The LB is a single point of failure if not redundant. Solutions: - **Multiple LBs** behind a virtual IP (failover). - **Anycast** (DNS routes to nearest LB; multiple in different regions). - **Cloud-managed LBs** handle this automatically (AWS ELB, GCP LB). ## Caching at the LB L7 LBs can cache responses (HTTP cache headers). Saves backend hits for static or repeatable content. Less common in 2026 — usually delegated to CDN. ## Common LB mistakes 1. **No health checks.** Dead backends still receive traffic. 2. **Deep health checks.** DB outage cascades through whole system. 3. **Sticky sessions when not needed.** Limits horizontal scaling. 4. **Single LB.** Single point of failure. 5. **Naive hashing for cache sharding.** Adding/removing a node shifts everything; use consistent hashing. ## Takeaway LB sits between clients and backends; distributes requests. L4 = fast, content-agnostic; L7 = HTTP-aware, routing flexibility. Algorithms: round robin (default), least connections (variable durations), IP hash (stickiness), consistent hashing (cache sharding). Stateless backends + externalized session = no need for sticky sessions. Always health-check; avoid deep checks that cascade. LB itself needs HA (or use managed).