Monitoring & Observability

This lesson on Monitoring & Observability is hands-on and example-driven. You will master the three pillars of observability—logs, metrics, and distributed traces—to diagnose system failures and evaluate production health. You will learn to configure structured event streams, instrument the Four Golden Signals, and establish realistic percentile-based Service Level Objectives.

What You'll Be Able To Do

• Differentiate between logs, metrics, and traces during incident root-cause analysis.
• Structure application logs using JSON key-value pairs for automated querying at scale.
• Instrument services to track the Four Golden Signals across distributed systems.
• Formulate percentile-based Service Level Objectives (P50, P95, P99) to reflect actual user experience.
• Identify and mitigate high-cardinality traps within time-series metric databases.

Detailed Concept Walkthrough

1. The Three Pillars of Observability

Observability relies on three distinct telemetry primitives—logs, metrics, and traces—that collectively explain what happened, overall system health, and request journeys across boundaries.

• Hospital Analogy: Logs act like detailed nurse notes documenting specific discrete events; metrics represent real-time patient vitals measured continuously; traces map the complete patient journey through different departments.
• Telemetry Synergy: Metrics provide low-cost alerting on anomalies, traces pinpoint the specific failing downstream microservice, and logs deliver detailed contextual state for root-cause analysis.
• Storage Trade-offs: Metrics are compact and aggregated in time-series databases, while logs and distributed traces demand high-throughput ingestion pipelines and scalable search indices.

# Conceptual relationship of the three pillars during an incident
# 1. Metric Alert fires: P99 Latency > 500ms on /checkout
# 2. Distributed Trace identifies bottleneck: payment-service spans take 450ms
# 3. Structured Log provides exact cause: {"error": "Gateway timeout", "trace_id": "4bf92f3577b34da6"}

Key Takeaway: Metrics detect that a problem exists, traces isolate where it occurs, and logs explain what caused it.

2. Structured vs Unstructured Logging

Logs are discrete, timestamped records of events containing severity levels and context, designed to explain exact system states at specific points in time.

• Parsing Mechanics: Unstructured plaintext logs require complex, fragile regular expressions to parse at runtime, causing query bottlenecks during incidents.
• Machine Readability: Structured logs emit data as serialized key-value pairs (typically JSON), allowing log aggregators to automatically index fields without custom extractors.
• Context Propagation: Structured formats enforce standard fields like timestamp, log level (INFO, WARN, ERROR), correlation IDs, and execution metadata for seamless filtering across distributed nodes.

import json
import logging
from datetime import datetime, timezone

# Structured JSON logging implementation
def log_event(level: str, message: str, **context):
    payload = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "level": level,
        "message": message,
        **context
    }
    print(json.dumps(payload))

log_event("ERROR", "Payment processing failed", user_id=4821, order_id="ord-993", error_code="INSUFFICIENT_FUNDS")

Key Takeaway: Always emit structured JSON logs with contextual metadata to enable high-speed filtering across distributed services.

3. Metrics and the Four Golden Signals

Metrics are numeric measurements tracked periodically over time in time-series databases to evaluate real-time health and capacity trends.

• The Four Golden Signals: Modern observability standardizes on Latency (request duration), Traffic (demand/request rate), Errors (rate of failed requests), and Saturation (resource capacity utilization).
• Time-Series Storage: Metrics are stored as time-stamped float values associated with metric names and label sets, optimized for fast aggregations and mathematical operations over time windows.
• Cardinality Constraints: Adding dimensions with unlimited unique values (e.g., raw user IDs or order IDs) causes an exponential explosion in time-series index size, degrading database performance.

# Prometheus metric exposition for the Four Golden Signals
# HELP http_requests_total The total number of HTTP requests (Traffic & Errors)
# TYPE http_requests_total counter
http_requests_total{method="POST",handler="/checkout",status="200"} 10243
http_requests_total{method="POST",handler="/checkout",status="500"} 12

# HELP http_request_duration_seconds Latency histogram
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1"} 9800
http_request_duration_seconds_bucket{le="0.5"} 10200

Key Takeaway: Monitor the Four Golden Signals to guarantee minimum viable observability without blowing up storage through high cardinality.

4. Latency Percentiles and SLOs

Averages conceal extreme tail latency experienced by outliers, requiring percentile metrics (P50, P95, P99) to establish meaningful Service Level Objectives.

• Flaw of Averages: An average latency of 50ms can mask the fact that 1% of your highest-value users endure multi-second timeouts on complex database queries.
• Percentile Distribution: P50 represents median user experience, P95 highlights elevated latency trends, and P99 captures the worst-impacted 1% of requests.
• SLO Formulation: Define precise, measurable targets using tail percentiles (e.g., '99% of valid HTTP requests to /checkout must return within 200ms over a 30-day rolling window').

# Prometheus PromQL queries for Percentile SLO evaluation
# Calculate P99 latency across all web instances over a 5m rate window
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Calculate P50 (median) latency for baseline comparison
histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Key Takeaway: Never use mean latency for user experience targets; construct SLOs based on tail percentiles like P95 and P99.

Topics Covered in Monitoring & Observability

1. Defining Three Pillars (0:42 - 1:28) — The instructor contrasts logs, metrics, and traces using a hospital analogy.
2. Logs and Structured Formats (1:29 - 4:16) — The instructor demonstrates why structured JSON logging is essential over unstructured text for distributed systems.
3. Metrics and Golden Signals (4:16 - 5:39) — The instructor covers time-series metrics and introduces the Four Golden Signals for core service monitoring.
4. Percentiles and Cardinality (5:39 - 7:20) — The instructor explains why percentiles must replace averages in SLOs and warns against high-cardinality label explosion.

System Design Fundamentals Cheat Sheet

• Structured JSON Log — Formats log events for automated parsing

  {"timestamp": "2023-10-27T10:00:00Z", "level": "INFO", "event": "auth_success"}
  

• Latency Signal — Tracks duration taken to service requests

  http_request_duration_seconds{handler="/api/v1/orders"}
  

• Traffic Signal — Measures demand placed on the system

  rate(http_requests_total[1m])
  

• Error Signal — Monitors rate of failed requests explicitly

  rate(http_requests_total{status=~"5.."}[1m])
  

• Saturation Signal — Quantifies resource utilization and remaining capacity

  node_memory_Active_bytes / node_memory_MemTotal_bytes
  

• P99 Percentile Query — Calculates 99th percentile response duration

  histogram_quantile(0.99, sum(rate(http_duration_bucket[5m])) by (le))
  

Comparison Table

Common Pitfalls

• Mistake: Relying on average latency to define Service Level Objectives. Avoid: Measure P95 and P99 percentiles to detect severe tail-latency impacts on users.
• Mistake: Writing unstructured string logs across microservices. Avoid: Emit structured JSON with standard keys including level, timestamp, and context.
• Mistake: Injecting high-cardinality values like user IDs into metric label dimensions. Avoid: Store high-cardinality attributes exclusively in structured logs and distributed traces.

FAQs

• Why is average latency considered harmful in system design? Averages smooth out outliers, hiding tail-latency spikes that severely degrade the experience for the unluckiest percentage of users.
• What is the high-cardinality trap in time-series metrics? Attaching unique identifiers like UUIDs as metric labels creates millions of distinct time-series streams, exhausting metric database memory.
• When should an engineer look at traces instead of logs? Use traces to isolate which service in a call graph introduced latency or errors before diving into logs for exact stack traces.