Load Balancers

This lesson on Load Balancers is hands-on and example-driven. You will design, position, and configure load balancers across multi-tier distributed architectures. You can evaluate routing algorithms, manage server health states, and choose between software-defined and hardware appliances to scale traffic reliably.

What You'll Be Able To Do

• Position load balancers across heterogeneous architecture tiers including web servers, caching layers, and database replicas.
• Implement routing policies based on host headers, URL paths, and content types to direct traffic across microservices.
• Select and configure traffic distribution algorithms such as Round Robin, Least Connections, and Least Response Time based on workload characteristics.
• Evaluate deployment trade-offs between cloud software load balancers and physical on-premise hardware appliances.
• Design continuous health check mechanisms to remove failing instances from active traffic rotation dynamically.

Detailed Concept Walkthrough

1. Architectural Placement and Tiered Balancing

Load balancers act as reverse proxies that distribute incoming client traffic across multiple backend instances to ensure high availability and prevent single points of failure. They can be deployed at multiple distinct tiers within a distributed system.

• Mechanism: Position load balancers at client-facing entry points in front of web application servers, as well as internally between web services, caching layers, and database replica pools.
• Under the Hood: Client DNS records resolve to the public IP of the external load balancer, which terminates or forwards the TCP/HTTP connection and opens a separate backend connection to an internal node.
• Best Practice: Use layered load balancing so that traffic between stateless compute nodes and stateful storage tiers is evenly balanced without exposing internal topology to public networks.

# Example NGINX upstream configuration for layered web nodes
upstream backend_web_cluster {
    server web1.internal.net:8080 weight=3;
    server web2.internal.net:8080 weight=2;
    server web3.internal.net:8080 backup;
}
server {
    listen 80;
    location / {
        proxy_pass http://backend_web_cluster;
    }
}

Key Takeaway: Placing load balancers at multiple architectural boundaries isolates failures and allows each infrastructure tier to scale independently.

2. Attribute-Based Traffic Routing

Advanced load balancers inspect application-layer (Layer 7) packet attributes to steer requests to specialized backend clusters. This enables domain separation, API versioning, and static asset offloading.

• Host-Based Routing: The load balancer parses the HTTP 'Host' header to direct traffic targeting distinct subdomains or domain names to isolated server clusters.
• Path-Based Routing: The load balancer inspects the URL path (such as '/v1/users' vs. '/v2/users') to route traffic to specific microservices or updated API releases.
• Content-Based Routing: Requests are forwarded based on headers, query parameters, or file extensions, separating streaming media or heavy image processing from low-latency transactional endpoints.

# NGINX L7 routing by URI path and domain
http {
    server {
        listen 80;
        server_name api.example.com;
        location /v1/ {
            proxy_pass http://api_v1_cluster;
        }
        location /v2/ {
            proxy_pass http://api_v2_cluster;
        }
        location /media/ {
            proxy_pass http://media_storage_cluster;
        }
    }
}

Key Takeaway: Layer 7 routing inspects request metadata to distribute traffic to purpose-built service pools based on paths, headers, and hosts.

3. Software Versus Hardware Load Balancers

Load balancing can be implemented using dedicated physical rack appliances or flexible, software-defined virtual instances and cloud services. The choice depends on performance demands, regulatory environments, and elasticity needs.

• Software Balancers: Deployed as virtual machines, containers, or managed cloud services (e.g., AWS ALB, HAProxy, NGINX), allowing rapid scaling, programmatic reconfiguration, and low initial capital expenditure.
• Hardware Appliances: Proprietary rack-mounted devices with specialized ASICs designed for extreme raw throughput and physical isolation in on-premise data centers.
• Nuance: Software balancers are preferred in cloud-native and elastic architectures, whereas hardware balancers frequently serve legacy enterprise environments or strict government compliance mandates.

# Deploying HAProxy via Docker CLI as a software load balancer
docker run -d \
  --name haproxy-lb \
  -p 80:80 -p 443:443 \
  -v $(pwd)/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro \
  haproxy:latest

Key Takeaway: Software load balancers provide flexible elasticity for modern clouds, while hardware appliances cater to specialized on-premise throughput and regulatory needs.

4. Traffic Distribution Algorithms and Health Ledgers

Load balancers use algorithmic strategies to select the next destination node, pairing distribution logic with real-time health checks to avoid degraded or dead servers.

• Static Algorithms: Round Robin and Weighted Round Robin cycle through nodes sequentially without evaluating dynamic server load, ideal for uniform request sizes and identical hardware.
• Dynamic Algorithms: Least Connections and Least Response Time monitor backend metrics and assign new traffic to the server with the fewest active tasks or lowest latency.
• Health Ledgers: The load balancer continuously probes backend endpoints (e.g., HTTP GET /healthz) and maintains an internal ledger; unhealthy nodes are removed from the routing table until consecutive checks succeed.

# HAProxy configuration using Least Connections and active health checks
backend app_servers
    balance leastconn
    option httpchk GET /healthz
    http-check expect status 200
    server srv1 10.0.0.1:8080 check inter 2000ms fall 3 rise 2
    server srv2 10.0.0.2:8080 check inter 2000ms fall 3 rise 2
    server srv3 10.0.0.3:8080 check inter 2000ms fall 3 rise 2

Key Takeaway: Dynamic balancing algorithms rely on active health ledgers to steer requests away from saturated or failing instances.

Topics Covered in Load Balancers

1. Context and Architectural Role (0:03 - 1:05) — Covers the fundamental purpose of load balancers as mandatory components for distributed system stability and scalability.
2. Placement and Tiering (1:06 - 2:22) — Examines where load balancers reside across infrastructure layers, including web servers, caching tiers, and databases.
3. L7 Traffic Routing Capabilities (2:23 - 3:52) — Explains host-based, path-based, and content-based routing policies to manage microservices and API versions.
4. Software vs. Hardware Balancers (3:52 - 4:35) — Compares cloud-native software load balancers with dedicated on-premise hardware appliances.
5. Algorithms and Health Checks (4:35 - 7:25) — Details static and dynamic traffic distribution algorithms and dynamic health ledger monitoring.

System Design Fundamentals Cheat Sheet

• Round Robin — Cycles requests evenly across available servers in sequential order

  balance roundrobin
  

• Least Connections — Routes incoming traffic to the server with fewest active connections

  balance leastconn
  

• Weighted Round Robin — Distributes requests proportionally based on assigned server weight capacities

  server app1 10.0.0.1:80 weight 3
  

• Path-Based Routing — Directs traffic to target groups matching the request URI path

  location /api/v2 { proxy_pass http://v2_backend; }
  

• Host-Based Routing — Directs traffic based on the HTTP Host header domain name

  server_name auth.example.com;
  

• Active Health Check — Probes backend instances periodically to verify operational readiness

  option httpchk GET /healthz
  

Comparison Table

Common Pitfalls

• Mistake: Assuming load balancers only belong at the public network edge. Avoid: Place internal load balancers between application servers, caches, and database replicas to balance internal tiers.
• Mistake: Using simple Round Robin for long-running connections or uneven tasks. Avoid: Switch to Least Connections so saturated nodes do not accumulate heavy processing backlogs.
• Mistake: Relying on TCP connection checks instead of application-level health endpoints. Avoid: Configure HTTP health checks querying endpoints that validate database and dependency health.

FAQs

• Can load balancers balance database connections directly? Yes, internal load balancers can distribute read queries across database read replicas, provided the application handles write-to-primary and read-from-replica routing logic.
• What is the difference between path-based and host-based routing? Host-based routing inspects the domain in the Host header (e.g., api.domain.com vs web.domain.com), whereas path-based routing inspects the URI string (e.g., domain.com/v1 vs domain.com/v2).
• How does an active health check remove a node? The load balancer sends periodic synthetic requests to a health endpoint; if a node fails a predetermined threshold of consecutive probes, it is marked down and skipped.