Recap — Debugging and Monitoring Agents in Production

# Debugging and Monitoring Agents in Production A 1-LLM-call app is debuggable from logs. A 20-step agent is not. Production agents need three layers of observability. ## Layer 1 — Streaming state changes Stream each node's state change as it happens: ```python for chunk in graph.stream({"messages": [user_msg]}, config): print(chunk) # {"agent": {"messages": [AIMessage(...)]}} # {"tools": {"messages": [ToolMessage(...)]}} # {"agent": {"messages": [AIMessage(...)]}} ``` For UX: stream to the user too. "Researching... Found 3 documents... Generating answer..." → better than a 30-second silent wait. For debugging: see exactly where execution is at any moment. In production: - Stream events to a queue (Kafka, Redis Streams). - Log each event. - Power a real-time debug UI. ## Layer 2 — LangSmith tracing LangSmith automatically traces every LLM call, tool invocation, and chain/graph execution: ```python import os os.environ["LANGCHAIN_TRACING_V2"] = "true" os.environ["LANGCHAIN_API_KEY"] = "ls-..." os.environ["LANGCHAIN_PROJECT"] = "support-agent-prod" # Now every call is traced automatically ``` LangSmith UI shows: - Per-trace view: each step of an agent execution. - Inputs, outputs, timing, token counts per step. - Error stack traces. - Searchable history across all runs. - Cost breakdown. For multi-step agents, this is THE debugging tool. Without it, you're squinting at logs trying to reconstruct what happened. ### Tracing structure Each "run" in LangSmith corresponds to a top-level invoke. Nested runs (LLM calls, tool calls, sub-graph executions) appear as children. Example trace: ``` Run: graph.invoke (8.3s, $0.034) ├── agent (1.2s, $0.008) │ └── llm call (1.1s, $0.008) ├── tools (0.5s, $0) │ └── get_order_status (0.4s, $0) ├── agent (1.0s, $0.006) │ └── llm call (0.9s, $0.006) └── (3 more iterations...) ``` Click any step → see its input, output, prompt, tools available, raw API response. ### Searching traces Find specific runs: - By user_id (custom metadata). - By thread_id. - By error. - By model version. - By latency or cost outliers. ```python graph.invoke( {"messages": [user_msg]}, config={ "configurable": {"thread_id": "user-123"}, "metadata": { "user_id": "user-123", "version": "v1.2.0", "feature_flag": "new_routing", } } ) ``` Tags and metadata make traces searchable in LangSmith UI. ### Datasets and eval LangSmith doubles as an eval platform: - Build a dataset of (input, expected_output) examples. - Run your agent on the dataset. - Score outputs (deterministic, LLM-as-judge, human). - Compare runs (e.g., before and after a prompt change). ```python from langsmith import Client client = Client() dataset = client.create_dataset("support-agent-eval") client.create_examples( inputs=[{"messages": [...]} for ex in eval_data], outputs=[{"answer": ex.expected} for ex in eval_data], dataset_id=dataset.id, ) # Run evals from langsmith.evaluation import evaluate results = evaluate( lambda inputs: graph.invoke(inputs)["messages"][-1].content, data=dataset, evaluators=[my_judge_evaluator], ) ``` CI/CD integration: run evals on every PR; fail if scores drop. ## Layer 3 — Structured logging Independent of LangSmith, log key events to your standard logging infra: ```python import structlog logger = structlog.get_logger() def call_model(state): logger.info("agent_decide", thread_id=state.get("thread_id"), iteration=state["iteration"], message_count=len(state["messages"])) response = llm_with_tools.invoke(state["messages"]) logger.info("agent_decided", tool_calls=len(getattr(response, "tool_calls", []))) return {"messages": [response], "iteration": state["iteration"] + 1} ``` For dashboards (Grafana, Datadog): - Requests per minute. - p50/p95/p99 latency. - Error rate. - Cost per request (aggregate from token usage). - Hit rate of specific tools. LangSmith is your "look at one specific run". Logs + dashboards are your "is the system healthy overall?". ## Monitoring production agents What to watch: ### Latency p99 latency. Multi-step agents have variable latency; alert when p99 spikes. ### Error rates - LLM API errors. - Tool execution errors. - Validation failures. - Max-iteration-reached signals. ### Cost Per request and aggregate. Spikes mean something changed (longer prompts, more iterations, model swap). ### Tool usage Which tools fire most? Which never fire (dead code)? Which fire too often (loop bug)? ### Conversation length Average and outliers. Long conversations might be a UX issue or a memory bug. ### User feedback Thumbs up/down, complaints, escalations. Direct quality signal. ## Debugging a specific failure User reports: "The agent gave me a wrong answer." Process: 1. Get the thread_id from your app's logs. 2. Open LangSmith, search by thread_id. 3. Look at the trace: what tools fired? what did the LLM see at each step? what went wrong? 4. R