Memory Architecture — Short-Term, Working & Semantic

This lesson on Memory Architecture — Short-Term, Working & Semantic is hands-on and example-driven. You will design a multi-tiered agent memory architecture separating in-process scratchpads, short-term session caches, and persistent synthesized long-term attributes. You will implement cache eviction to prevent stale retrieval bias and enforce privacy boundaries across multi-agent handoffs.

What You'll Be Able To Do

• Categorize agent data into working, short-term, or long-term tiers based on lifespan and scope.
• Implement low-latency session caching and state propagation using high-speed key-value stores.
• Evict stale or erroneous RAG retrieval artifacts from short-term context to prevent reasoning bias.
• Synthesize durable user attributes from raw session logs into long-term persistent storage.
• Scrub sensitive user data using de-identification pipelines before transmitting state across multi-agent boundaries.

Detailed Concept Walkthrough

1. Three-Tier Agent Memory Architecture

Agent memory maps directly onto application state management, separated into distinct temporal tiers to balance operational latency, retention, and token costs.

• Mechanism: Working memory acts as an ephemeral in-process scratchpad for intermediate tool execution and mathematical reasoning steps.
• Under the Hood: Short-term memory captures conversation context and cached retrieval outputs across multiple turns within an active user session.
• Best Practice: Long-term memory stores durable user attributes and recurring preferences across sessions, synthesized from prior interaction logs rather than raw transcripts.

# Conceptual schema mapping memory tiers to agent state
agent_state = {
    "working_memory": {"intermediate_step": 42, "tool_scratchpad": []},
    "short_term_memory": {"session_id": "sess_123", "history": []},
    "long_term_memory": {"user_id": "usr_99", "preferences": {"theme": "dark"}}
}

Key Takeaway: Separate data across working, short-term, and long-term tiers to balance runtime performance with persistence.

2. Working and Short-Term Session Caching

Short-term memory provides rapid state propagation across multi-step execution flows using high-speed key-value data stores.

• Mechanism: Low-latency caching services (such as Memorystore or Redis) hold active turn histories and recently retrieved RAG payloads.
• Under the Hood: In-memory key-value lookups eliminate database I/O bottlenecks during rapid multi-turn agent interactions.
• Best Practice: Always scope short-term cache keys by session identifier with explicit TTL policies to prevent unbound memory growth.

import redis

# Connect to high-speed session datastore (e.g., Memorystore/Redis)
cache = redis.Redis(host='localhost', port=6379, db=0)

def save_session_context(session_id: str, context_data: str, ttl_seconds: int = 3600):
    cache.setex(name=f"session:{session_id}", time=ttl_seconds, value=context_data)

Key Takeaway: Use fast in-memory key-value caching to deliver low-latency state propagation across active session turns.

3. Context Eviction and RAG Artifact Management

Retaining incorrect or outdated retrieval-augmented generation (RAG) context in short-term memory biases subsequent agent reasoning steps.

• Mechanism: Context overweighting occurs when an agent treats stale or erroneous retrieved documents in its session history as ground truth for future actions.
• Under the Hood: LLM attention layers disproportionately weigh earlier conversational context, compounding errors across chained tool steps.
• Best Practice: Implement explicit pruning routines that evict invalidated RAG artifacts and failed tool outputs from the active prompt window.

def prune_stale_rag_context(messages: list[dict], failed_doc_id: str) -> list[dict]:
    # Filter out retrieved artifacts that produced validation errors
    return [
        msg for msg in messages
        if msg.get("metadata", {}).get("doc_id") != failed_doc_id
    ]

Key Takeaway: Actively evict erroneous retrieval artifacts from short-term memory to prevent compounding reasoning failures.

4. Long-Term Memory Synthesis and Cost Optimization

Synthesizing session transcripts into concise, durable attributes avoids the token overhead of injecting full conversation histories into every prompt.

• Mechanism: A post-session summarization routine processes short-term logs and extracts high-value traits, user preferences, and code corrections.
• Under the Hood: Extracted key-value attributes are persisted in a long-term database and loaded selectively at session initialization.
• Best Practice: Hydrate the system prompt with synthesized profile summaries rather than replaying raw multi-turn conversation logs.

def extract_durable_attributes(session_summary: str) -> dict:
    # Ingest session summary and persist key attributes to database
    user_profile = {"preferred_language": "Python", "enforce_type_hints": True}
    # Save user_profile to persistent long-term storage
    return user_profile

Key Takeaway: Compress interaction history into structured attributes to minimize prompt token usage and compute costs.

5. Multi-Agent State Isolation and Privacy Controls

Multi-agent systems require explicit boundaries between shared communication channels and isolated private execution states.

• Mechanism: Partition agent states so specialized sub-agents only access the minimum necessary context for their specific subtasks.
• Under the Hood: Data de-identification pipelines sanitize sensitive user data, such as payment credentials or PII, before inter-agent handoffs.
• Best Practice: Define strict marshaling schemas between collaborating agents to prevent leaking unscrubbed memory artifacts across service boundaries.

def sanitize_for_handoff(agent_data: dict) -> dict:
    scrubbed = agent_data.copy()
    # Strip sensitive credentials prior to cross-agent transmission
    scrubbed.pop("payment_credentials", None)
    scrubbed["user_email"] = "[REDACTED]"
    return scrubbed

Key Takeaway: Partition state per agent and de-identify sensitive data before transmitting context across agent boundaries.

Topics Covered in Memory Architecture — Short-Term, Working & Semantic

1. Foundational Memory Tiers (0:06 - 1:40) — Categorizes agent memory into working scratchpad, short-term session context, and persistent long-term storage.
2. Working and Session Memory (1:40 - 2:30) — Explains low-latency session caching and the necessity of purging stale RAG retrieval artifacts.
3. Long-Term Memory Synthesis (2:30 - 3:13) — Covers extracting durable attributes via post-session summarization to minimize prompt token overhead.
4. Multi-Agent State Isolation (3:13 - 4:18) — Details state partitioning between agents and de-identifying sensitive data prior to message handoffs.

LangChain & LangGraph for AI Agents Cheat Sheet

• Working Memory — Stores ephemeral intermediate tool outputs and calculations

  scratchpad = {'step_1_calc': 105.4, 'raw_output': 'Pending'}
  

• Short-Term Memory — Maintains low-latency context across multi-turn sessions

  cache.setex(f"session:{sid}", 3600, json.dumps(history))
  

• Long-Term Memory — Persists synthesized attributes and user preferences

  db.profiles.update_one({'uid': uid}, {'$set': {'role': 'admin'}})
  

• Context Eviction — Removes stale or invalid RAG artifacts from state

  history = [m for m in history if not m.get('is_stale')]
  

• State De-identification — Scrubs sensitive fields before cross-agent handoff

  payload = {k: v for k, v in state.items() if k != 'token'}
  

Comparison Table

Common Pitfalls

• Mistake: Retaining incorrect RAG retrieval outputs in the active session buffer. Avoid: Implement explicit eviction logic to purge invalid retrieved context before subsequent steps.
• Mistake: Storing raw uncompressed chat logs as persistent long-term memory. Avoid: Run an asynchronous summarization pass to extract only high-value user attributes.
• Mistake: Sharing a single global state object across all collaborating agents. Avoid: Partition agent states and isolate internal scratchpads behind explicit communication interfaces.
• Mistake: Passing sensitive user credentials across inter-agent communication channels. Avoid: Apply automated de-identification pipelines before marshaling data between agent workflows.

FAQs

• What causes context overweighting in short-term agent memory? Retaining incorrect or outdated RAG search results causes the model to prioritize faulty historical artifacts over valid reasoning in subsequent turns.
• Why should long-term memory store synthesized attributes instead of raw transcripts? Synthesizing attributes eliminates repetitive computation and prevents prompt context overflow, significantly reducing API token costs.
• When should multi-agent systems use isolated state versus shared state? Use isolated state for internal reasoning and scratchpads, while using scrubbed, schema-validated shared messages for collaborative handoffs.
• How does working memory differ technically from short-term memory? Working memory is in-process ephemeral scratchpad data for a single step, whereas short-term memory is cached session context spanning turns.