Caching

This lesson on Caching is hands-on and example-driven. You will be able to design and implement caching tiers across modern system architectures to reduce database load and cut request latency. You will select optimal eviction policies and freshness strategies like TTLs based on workload access patterns.

What You'll Be Able To Do

• Analyze memory hierarchy trade-offs to justify cache placement between fast storage and persistent databases.
• Select appropriate eviction policies (LRU, LFU, FIFO) based on access recency and frequency patterns.
• Implement cache invalidation mechanisms using Time-To-Live (TTL) and manual eviction to prevent stale data reads.
• Deploy specialized caching layers including CDNs for static assets and Redis for high-throughput application data.

Detailed Concept Walkthrough

1. Memory Hierarchy and Locality

Caching exploits the principle of locality by placing high-speed, temporary storage between fast execution units and slower persistent storage. It balances the trade-off between expensive, ultra-fast memory and cheap, slower disk storage.

• Mechanism: Data requests check the cache tier first before hitting downstream origin systems. If found, a cache hit returns data in sub-millisecond time; otherwise, a cache miss fetches from persistent storage and populates the cache.
• Under the Hood: Hardware registers and CPU caches sit at the top of the hierarchy with minimal capacity and highest cost per byte. Main memory (RAM) acts as an intermediary layer before mechanical or solid-state disk storage.
• Best Practice: Position caches as close to the requester as possible to minimize network hops and origin database saturation.

# Example: Basic memory hierarchy lookup pattern
def get_user_profile(user_id):
    # 1. Check in-memory cache first
    profile = memory_cache.get(f"user:{user_id}")
    if profile:
        return profile  # Cache hit (fast path)
    
    # 2. Cache miss: fallback to persistent disk/database
    profile = database.query_user(user_id)
    memory_cache.set(f"user:{user_id}", profile)
    return profile

Key Takeaway: Caches trade off expensive memory capacity to shield slow origin databases from high-frequency queries.

2. Distributed Caching: CDNs and Redis

Specialized caching infrastructure handles distinct workloads across global network edges and application backend layers. Content Delivery Networks (CDNs) distribute static assets globally, while in-memory data stores like Redis handle dynamic application data.

• CDN Architecture: Edge servers distributed across points of presence (PoPs) terminate user connections geographically close to clients, serving cached images, videos, and static files.
• In-Memory Data Stores: Redis stores key-value structures entirely in RAM, functioning as an application cache, primary database, or message broker with sub-millisecond read/write latency.
• Execution Flow: Static requests route through edge CDN proxies, whereas backend application services interact with centralized Redis clusters to cache database queries and compute-heavy results.

# Python Redis application cache implementation
import redis

client = redis.Redis(host='localhost', port=6379, db=0)

def get_cached_search(query):
    cache_key = f"search:{query}"
    # Attempt Redis read
    cached_result = client.get(cache_key)
    if cached_result:
        return cached_result.decode('utf-8')
    
    # Fetch from search index
    result = run_expensive_search(query)
    client.set(cache_key, result, ex=300) # Cache for 5 minutes
    return result

Key Takeaway: Use CDNs to cache static assets at the network edge and Redis to accelerate dynamic backend queries.

3. Cache Eviction Policies

Eviction policies govern how a cache frees memory space when it hits its predefined capacity limit. Choosing the correct algorithm depends on whether access patterns favor recently touched or frequently requested data.

• LRU (Least Recently Used): Evicts the item that has not been accessed for the longest time, prioritizing temporal recency.
• LFU (Least Frequently Used): Tracks access counters to evict items with the lowest cumulative hit counts, prioritizing historical popularity.
• FIFO (First In First Out): Evicts items in the strict order they were inserted, disregarding subsequent read access patterns entirely.

# Redis maxmemory configuration for LRU/LFU eviction
# /etc/redis/redis.conf

# Set hard cap on memory
maxmemory 2gb

# Configure eviction policy when memory cap is reached
# Options: allkeys-lru, volatile-lru, allkeys-lfu, volatile-lfu, noeviction
maxmemory-policy allkeys-lru

Key Takeaway: Select LRU for workloads where recent items are re-requested, and LFU for items with sustained high popularity.

4. Cache Invalidation and TTL Strategies

Cache invalidation ensures client reads receive fresh data after updates occur at the source of truth. Strategies combine proactive programmatic invalidation with reactive Time-To-Live (TTL) expiration.

• Time-To-Live (TTL): Sets an expiration timestamp on cached keys, automatically evicting stale data once the duration elapses.
• Manual Invalidation: Application code explicitly deletes or updates corresponding cache keys immediately after committing writes to the origin database.
• Under the Hood: Proactive invalidation eliminates stale reads instantly on mutation, while TTL acts as a safety net against leaked or orphaned cache entries.

def update_user_email(user_id, new_email):
    # 1. Update source of truth
    database.execute("UPDATE users SET email = %s WHERE id = %s", (new_email, user_id))
    
    # 2. Explicitly invalidate cache entry
    client.delete(f"user:{user_id}")
    
    # Optional: warm cache with new TTL
    # client.set(f"user:{user_id}", new_email, ex=3600)

Key Takeaway: Combine explicit cache invalidation on writes with short-to-medium TTLs to guarantee eventual freshness.

Topics Covered in Caching

1. Memory Hierarchy Basics (0:00 - 1:26) — Explains how caching sits between registers and main memory to balance cost and speed.
2. Business Value & Use Cases (1:26 - 3:13) — Covers benefits including latency reduction and offloading database traffic with practical examples.
3. CDNs and Redis Tooling (3:13 - 5:07) — Contrasts edge distribution via CDNs with centralized backend caching using Redis.
4. Capacity & Eviction Policies (5:07 - 6:14) — Compares LRU, LFU, and FIFO strategies for freeing memory when caches fill up.
5. Invalidation & TTL Strategies (6:14 - 7:44) — Demonstrates maintaining data freshness using automatic TTL expiration and proactive invalidations.

System Design Fundamentals Cheat Sheet

• SET key value EX seconds — Store key-value pair with automatic expiration TTL

  client.set("query:top10", json_data, ex=60)
  

• DEL key — Explicitly invalidate and remove a cached key

  client.delete("user:1042")
  

• LRU Policy — Evict data based on least recent access timestamp

  maxmemory-policy allkeys-lru
  

• LFU Policy — Evict data based on lowest request frequency

  maxmemory-policy allkeys-lfu
  

• CDN Edge Caching — Serve static assets geographically close to clients

  curl -I https://cdn.example.com/static/logo.png
  

Comparison Table

Common Pitfalls

• Mistake: Relying entirely on infinite cache lifespans without TTLs. Avoid: Always attach a sensible TTL to prevent orphaned stale records when invalidation events fail.
• Mistake: Using LFU for bursty traffic where old popular keys block new items. Avoid: Use LRU when recent access patterns matter more than historical popularity.
• Mistake: Querying origin databases for unchanging assets repeatedly. Avoid: Offload static assets and repetitive compute results to CDNs and Redis layers.

FAQs

• What is the difference between cache eviction and cache invalidation? Eviction occurs when the cache runs out of memory and removes items based on a policy (e.g., LRU). Invalidation occurs when data changes at the source and the cache entry is removed to avoid serving stale data.
• Why not use the fastest memory tier (registers) for everything? Registers and fast hardware memory are prohibitively expensive and physically limited in capacity, requiring a multi-tiered hierarchy balancing speed and cost.
• When should I choose Redis over a CDN? Use Redis for dynamic backend application caching, database query results, and key-value state. Use CDNs to distribute static assets like images, scripts, and video globally.