Scaling — Autoscaling, Latency Budgets, Caching

This lesson on Scaling — Autoscaling, Latency Budgets, Caching is hands-on and example-driven. You will monitor live Kubernetes resource telemetry, diagnose CPU starvation under load, and size pod resource requests. You will deploy metrics-server in a local kind cluster, generate synthetic load using wrk, and scale workloads horizontally to balance compute utilization.

What You'll Be Able To Do

• Deploy and configure metrics-server within a local kind cluster by patching container TLS arguments.
• Generate concurrent HTTP traffic against internal cluster workloads using wrk inside a diagnostic pod.
• Monitor live pod and node CPU utilization using kubectl top commands.
• Calculate cluster pod density limits using CPU core to millicore conversions.
• Scale deployments manually to distribute compute load below declared baseline resource requests.
• Query Kubernetes API versions to verify cluster support for the autoscaling resource group.

Detailed Concept Walkthrough

1. Cluster Telemetry with Metrics Server

Metrics Server scrapes resource utilization metrics directly from node Kubelets and exposes them via the Kubernetes aggregated API. This provides the telemetry backbone required for both operator visibility and automated scaling decisions.

• Mechanism: Metrics Server queries container runtime metrics on each node and exposes them at metrics.k8s.io for consumption by the control plane and CLI tools.
• Under the Hood: In development environments like kind, Kubelet certificates are self-signed, causing default Metrics Server TLS validation to fail unless explicitly passed the --kubelet-insecure-tls flag.
• Best Practice: Always deploy environment-compatible manifests and remove test-only insecure TLS arguments before promoting deployment configurations to production clusters.

# Patch Metrics Server deployment to bypass self-signed cert validation in kind
kubectl patch deployment metrics-server -n kube-system --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/args/-",
    "value": "--kubelet-insecure-tls"
  }
]'

# Validate resource telemetry
kubectl top nodes
kubectl top pods

Key Takeaway: Metrics Server requires an established trust chain or an explicit insecure flag to collect Kubelet metrics in local virtualized clusters.

2. Traffic Generation and Load Profiling

Synthetic load generation creates artificial compute demand to expose bottlenecks and validate cluster behavior before deploying autoscaling policies.

• Mechanism: A dedicated benchmarking pod running wrk executes sustained, multi-threaded HTTP requests against the cluster-internal service DNS endpoint.
• Under the Hood: Heavy concurrent client traffic increases server compute consumption, causing pods without sufficient CPU allocations to starve and spike in request latency.
• Best Practice: Run benchmarking utilities from dedicated pods inside the cluster network to eliminate external ingress bottlenecks and isolate container compute limits.

# Launch an interactive shell inside the traffic generation pod
kubectl exec -it traffic-generator -- sh

# Run wrk load test: 5 threads, 5 concurrent connections against internal service
wrk -t5 -c5 -d60s http://go-web-service.default.svc.cluster.local:8080/

Key Takeaway: Internal load testing with wrk reveals real-world container CPU spikes without ingress network interference.

3. Millicore Math and Pod Density

Kubernetes measures CPU allocation in millicores (1 core = 1,000m) to allow fine-grained scheduling and prevent cluster node over-subscription.

• Mechanism: Declaring requests.cpu reserves a dedicated millicore slice for container scheduling, while unreserved compute remains available for burst workloads.
• Under the Hood: If a pod's CPU request is set to 500m (0.5 cores) but receives heavy traffic without strict limits, its actual usage can burst to 1,500m (300% of request), consuming capacity from neighbor workloads.
• Best Practice: Calculate node capacity by multiplying total host cores by 1,000m to determine maximum guaranteed pod density before scheduling workloads.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-web-server
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: web
        image: go-web-server:v1
        resources:
          requests:
            cpu: "500m"       # 500 millicores (0.5 physical CPU cores)
            memory: "128Mi"

Key Takeaway: CPU requests represent scheduler guarantees in millicores, not strict limits on what an unconstrained pod can consume.

4. Horizontal Load Distribution

Scaling the number of active pod replicas redistributes incoming network traffic across multiple containers to bring individual pod CPU consumption below requested thresholds.

• Mechanism: Increasing the replica count creates additional pod endpoints behind the Service, splitting the incoming connection stream among all healthy instances.
• Under the Hood: As traffic spreads across 4 replicas instead of 1, individual pod CPU consumption drops proportionally (e.g., from ~1,500m down to under 400m per pod).
• Best Practice: Check supported cluster API groups with kubectl api-versions to verify that autoscaling/v2 is active before defining dynamic autoscaling manifests.

# Scale deployment to 4 replicas to distribute incoming load
kubectl scale deployment go-web-server --replicas=4

# Observe distributed CPU utilization drop across replicas
kubectl top pods -l app=go-web-server

# Verify cluster support for autoscaling API resources
kubectl api-versions | grep autoscaling

Key Takeaway: Scaling replicas horizontally divides aggregate traffic volume, lowering per-pod compute consumption below requested limits.

Topics Covered in Scaling — Autoscaling, Latency Budgets, Caching

1. Static Replica Limitations (0:04 - 1:16) — Reviewing declarative deployment manifests and identifying why static replica counts cause over-provisioning or traffic starvation.
2. Cluster Provisioning (1:17 - 2:03) — Bootstrapping a 6-core local kind Kubernetes cluster and deploying a sample Go web server with 500m CPU requests.
3. Traffic Generation with wrk (2:04 - 3:10) — Deploying an Alpine traffic generation pod and running multi-threaded HTTP benchmarking tests using wrk.
4. Metrics Server Deployment (3:11 - 5:26) — Installing metrics-server, patching Kubelet TLS arguments for kind, and verifying live node and pod telemetry.
5. Manual Scale Optimization (5:27 - 8:38) — Calculating node millicore capacity, identifying a 300% CPU bottleneck, and scaling replicas to 4 to balance traffic.
6. Autoscaling API Verification (8:39 - 9:12) — Querying cluster API versions to verify autoscaling group availability in preparation for HPA configuration.

ML in Practice Cheat Sheet

• kubectl top nodes — Displays CPU and memory utilization across all cluster nodes

  kubectl top nodes
  

• kubectl top pods — Displays live CPU and memory metrics for running pods

  kubectl top pods -l app=go-web-server
  

• kubectl scale deployment <name> --replicas=<n> — Imperatively sets the desired replica count for a deployment

  kubectl scale deployment go-web-server --replicas=4
  

• kubectl exec -it <pod> -- <cmd> — Executes an interactive command inside a running container

  kubectl exec -it traffic-pod -- wrk -t5 -c5 http://web-service
  

• kubectl api-versions — Lists all Kubernetes API groups and versions enabled on cluster

  kubectl api-versions | grep autoscaling
  

• spec.containers[*].resources.requests.cpu — Declares guaranteed CPU scheduling allocation in millicores

  resources:
    requests:
      cpu: "500m"
  

Comparison Table

Common Pitfalls

• Mistake: Deploying standard metrics-server manifests to kind clusters without insecure TLS flags. Avoid: Add the insecure Kubelet TLS argument to metrics-server container arguments in local test environments.
• Mistake: Assuming a CPU request acts as a hard execution ceiling for a container. Avoid: Understand that requests reserve scheduling capacity while unconstrained containers can burst to 300% or more.
• Mistake: Hardcoding static replica counts for variable traffic production workloads. Avoid: Plan resource request baselines and verify autoscaling API support to implement dynamic horizontal autoscaling.

FAQs

• Why does metrics-server fail to scrape metrics out of the box in kind? Kind clusters use self-signed Kubelet certificates by default, which causes Metrics Server TLS certificate verification to fail unless patched with --kubelet-insecure-tls.
• What does 500m CPU mean in a deployment manifest? 500m represents 500 millicores, which is exactly equivalent to 0.5 physical or virtual CPU cores reserved for container scheduling.
• Why did CPU drop below 500m when scaling from 1 to 4 replicas under constant wrk load? Kubernetes services distribute incoming connections across all ready pod endpoints, cutting the traffic and compute load processed by each individual pod by approximately four.
• How does Kubernetes decide when a pod is at 100% CPU utilization? Autoscaling metrics calculate utilization percentage relative to the declared CPU request value, meaning a 500m request consuming 500m is evaluated at 100% capacity.