Recap — Metrics, Logs, Traces

# Metrics, Logs, Traces Observability = "can I figure out what's happening from the outside?" In production, you live and die by this. Three pillars. ## Pillar 1: Metrics Numeric measurements over time. Pre-aggregated, fast to query, cheap to store. ``` http_requests_total{method="GET", status="200"} 12345 api_latency_p99_seconds 0.8 db_connections_active 42 ``` ### Types of metrics - **Counter**: monotonically increases (requests handled, errors). - **Gauge**: point-in-time value (current memory usage, queue length). - **Histogram**: distribution (latency percentiles). - **Summary**: similar to histogram, computed differently. ### What to measure (RED + USE) RED for services: - **R**ate: requests per second. - **E**rrors: error rate. - **D**uration: latency (p50, p95, p99). USE for resources: - **U**tilization: % busy. - **S**aturation: queue length, blocking. - **E**rrors: hardware errors, faults. ### Tools - **Prometheus**: pull-based metrics store; standard for K8s. - **Datadog**: managed, full observability platform. - **Grafana**: visualization (works with many backends). - **StatsD**: push-based; older but widely deployed. - **CloudWatch**: AWS native. ## Pillar 2: Logs Discrete events with context. Searchable; high-detail; expensive at scale. ``` 2026-05-26 13:45:23 INFO user_id=123 action=login session=abc result=success 2026-05-26 13:45:25 ERROR user_id=124 action=login session=def result=invalid_password 2026-05-26 13:45:30 WARN user_id=125 db_connection=timeout retries=3 ``` ### Structured vs unstructured Unstructured (avoid): ``` 2026-05-26 user 123 logged in successfully from 1.2.3.4 ``` Structured (preferred): ```json {"ts":"2026-05-26T13:45:23Z","level":"INFO","user_id":123,"action":"login","ip":"1.2.3.4","result":"success"} ``` Structured logs are queryable, filterable, aggregatable. Use JSON or key=value format. ### Tools - **ELK / OpenSearch**: Elasticsearch + Logstash + Kibana. Industry standard. - **Loki**: Grafana's logs system; cheaper than ELK. - **Splunk**: enterprise; expensive. - **Datadog Logs**: integrated with their platform. - **CloudWatch Logs**: AWS native. ### Log levels - DEBUG: verbose; disabled in production usually. - INFO: notable events. - WARN: unexpected but recoverable. - ERROR: failures requiring attention. - FATAL: process about to die. Use INFO and above in production; DEBUG only when investigating. ### Log volume problems Logs at scale: millions per second, petabytes per month, expensive storage. Strategies: - Sample (keep 1% of access logs; keep all errors). - Retention tiers (7 days hot, 30 days warm, 1 year cold). - Log to file → ship → archive cycle. ## Pillar 3: Traces End-to-end view of a single request across services. Shows the full call tree and timing. ``` Request: POST /api/orders ├── api-gateway (5ms) │ ├── auth-service (10ms) │ ├── catalog-service (25ms) │ │ └── postgres (15ms) │ ├── payment-service (200ms) ← bottleneck │ │ └── stripe-api (180ms) │ └── notification-service (15ms) Total: 255ms ``` ### Why traces matter In microservices, "why is this slow?" is hard to answer without traces. Traces show: - Which service is slow. - Which DB call is slow. - Where time is actually spent. - Cascading failures. ### Distributed tracing Each request gets a **trace ID** that propagates across services. Each operation is a **span** with parent-child relationships. ``` Trace ID: abc123 Span 1: api-gateway (parent: -) Span 2: auth (parent: 1) Span 3: catalog (parent: 1) Span 4: postgres-query (parent: 3) ``` ### Tools - **OpenTelemetry**: open standard for traces, metrics, logs. The future. - **Jaeger**: open-source tracing backend. - **Zipkin**: older open-source tracer. - **Datadog APM**: integrated traces. - **AWS X-Ray**: AWS native. OpenTelemetry is becoming the universal way to instrument; backends are vendor-specific. ## How the three combine A real outage investigation: 1. **Alert fires**: "API error rate exceeds 1%" (metric). 2. **Look at dashboard**: errors started 10 min ago; specific endpoint affected. 3. **Search logs**: filter `endpoint=/api/orders level=ERROR` → see "database timeout". 4. **Look at traces**: requests show 5-second DB queries. 5. **Drill into DB**: locks on `orders` table; long-running query blocking. 6. **Fix**: kill the rogue query; add index. Metrics tell WHAT broke. Logs tell WHY. Traces tell WHERE. ## SLI, SLO, SLA Three terms often confused: - **SLI (Service Level Indicator)**: a measured metric, e.g., "p99 latency". - **SLO (Service Level Objective)**: internal target, e.g., "p99 latency < 500ms 99% of the time". - **SLA (Service Level Agreement)**: contractual commitment with customer (and penalties if violated). SLO drives engineering priorities; SLA drives contracts. Error budget: 1% downtime tolerance = ~7h/month. If you've used 5h this month, slow down on risky deploys. ## Alerting Three principles: 1. **Alert on symptoms, not ca