Industry Context: An e-commerce platform needs an autonomous customer support agent to resolve customer inquiries regarding order status, shipping delays, and refund requests. The agent must enforce strict determinism: querying order tracking via API, verifying refund eligibility before triggering a refund, gating destructive actions behind human approval thresholds, and preventing infinite loops.
Deliverables
support_agent.py: Complete agent loop with tool definitions (get_order_status,check_refund_eligibility,issue_refund,escalate_to_human).tool_schemas.json: Strict JSON schema definitions with semantic descriptions of WHEN to call each tool.test_conversations.py: Test suite verifying execution traces across happy paths, multi-step dependency orders, out-of-scope banter, and unauthorized refund attempts.
Starter Code Scaffold
import json
from typing import Dict, Any, List
TOOLS = [
{
"name": "get_order_status",
"description": "Look up order tracking and delivery status. Use when the customer provides or asks about an order_id.",
"parameters": {
"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"]
}
},
{
"name": "check_refund_eligibility",
"description": "Verify whether an order qualifies for a refund. MUST be called BEFORE calling issue_refund.",
"parameters": {
"type": "object",
"properties": {"order_id": {"type": "string"}},
"required": ["order_id"]
}
}
]
def agent_loop(user_query: str, chat_history: List[Dict[str, str]], max_iterations: int = 5) -> str:
"""Execute agent tool loop with cycle detection and maximum iteration guard."""
# TODO: Implement tool execution, state tracking, and exit conditions
pass
100-Point Evaluation Rubric
- Criterion 1: Tool Schema Engineering (25 pts) — Descriptive docstrings clarifying parameter semantics, required attributes, and explicit trigger contexts.
- Criterion 2: Multi-Step Precondition Gating (25 pts) — Architectural barrier preventing
issue_refundexecution without prior validatedcheck_refund_eligibility. - Criterion 3: Loop Safety & State Management (25 pts) — Deterministic
max_iterations=5termination, handlingtool_calls is None, graceful degradation on API failures. - Criterion 4: Test Suite & Trace Verification (25 pts) — Comprehensive automated unit tests verifying execution traces across normal, adversarial, and boundary flows.
Implementation Guide & Case Steps
"Build a support agent that can look up order status, check refund eligibility, create tickets, and escalate to humans. Use Claude's tool-use feature."
Step 1 — Define the tools
tools = [
{
"name": "get_order_status",
"description": "Look up the current status, tracking info, and estimated delivery of a customer's order. Use when the customer asks about their order, shipping, delivery, or tracking. Requires order_id.",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string", "description": "Order ID, e.g., '12345'"}
},
"required": ["order_id"]
}
},
{
"name": "check_refund_eligibility",
"description": "Check if an order is eligible for refund. Returns eligibility status and reasoning. Use BEFORE promising a refund.",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"}
},
"required": ["order_id"]
}
},
{
"name": "initiate_refund",
"description": "Initiate a refund for an eligible order. Only call after check_refund_eligibility returns eligible=true. Returns the refund ID and ETA.",
"input_schema": {
"type": "object",
"properties": {
"order_id": {"type": "string"},
"reason": {"type": "string", "description": "Customer's stated reason for refund"}
},
"required": ["order_id", "reason"]
}
},
{
"name": "create_support_ticket",
"description": "Create a support ticket for issues that need human attention. Use when: customer expresses frustration, request is outside scope, or you're unsure how to proceed.",
"input_schema": {
"type": "object",
"properties": {
"customer_id": {"type": "string"},
"category": {"type": "string", "enum": ["complaint", "feature_request", "billing", "technical", "other"]},
"summary": {"type": "string"},
"priority": {"type": "string", "enum": ["low", "medium", "high", "urgent"]}
},
"required": ["customer_id", "category", "summary", "priority"]
}
},
]
Notice: descriptions explicitly state WHEN to call each tool, and constraints (e.g., "only call initiate_refund after eligibility check").
Step 2 — Implement the tools
def get_order_status(order_id):
order = db.query("SELECT * FROM orders WHERE id = ?", order_id)
if not order:
return {"error": "Order not found"}
return {
"order_id": order.id,
"status": order.status,
"carrier": order.carrier,
"tracking_number": order.tracking,
"estimated_delivery": order.eta.isoformat(),
"placed_at": order.created_at.isoformat(),
}
def check_refund_eligibility(order_id):
order = db.query("SELECT * FROM orders WHERE id = ?", order_id)
if not order:
return {"eligible": False, "reason": "Order not found"}
days_since = (datetime.utcnow() - order.created_at).days
if days_since > 30:
return {"eligible": False, "reason": f"Order is {days_since} days old; refund window is 30 days"}
if order.status == "delivered" and order.delivered_at:
days_since_delivery = (datetime.utcnow() - order.delivered_at).days
if days_since_delivery > 30:
return {"eligible": False, "reason": f"Delivered {days_since_delivery} days ago; return window expired"}
return {"eligible": True, "reason": "Within refund window"}
def initiate_refund(order_id, reason):
eligibility = check_refund_eligibility(order_id)
if not eligibility["eligible"]:
return {"success": False, "error": eligibility["reason"]}
refund = refund_service.create(order_id=order_id, reason=reason)
return {
"success": True,
"refund_id": refund.id,
"eta_days": 3,
}
def create_support_ticket(customer_id, category, summary, priority):
ticket = support_service.create_ticket(
customer_id=customer_id,
category=category,
summary=summary,
priority=priority,
)
return {"ticket_id": ticket.id, "created": True}
TOOL_REGISTRY = {
"get_order_status": get_order_status,
"check_refund_eligibility": check_refund_eligibility,
"initiate_refund": initiate_refund,
"create_support_ticket": create_support_ticket,
}
Step 3 — The agent loop
import anthropic
claude = anthropic.Anthropic()
SYSTEM_PROMPT = '''You are a customer support agent for Acme. You can look up orders, check refund eligibility, initiate refunds, and create support tickets.
Guidelines:
- Be empathetic and concise.
- ALWAYS check refund eligibility before promising a refund.
- If unsure, create a support ticket and tell the customer a human will follow up.
- If the customer is frustrated, prioritize de-escalation.
- Don't fabricate information; use tools to look up real data.'''
def run_agent(user_message, customer_id, max_iterations=10):
messages = [{"role": "user", "content": f"[Customer ID: {customer_id}] {user_message}"}]
for iteration in range(max_iterations):
response = claude.messages.create(
model="claude-sonnet-4-6",
system=SYSTEM_PROMPT,
tools=tools,
messages=messages,
max_tokens=1024,
)
# If no tool use, we're done
if response.stop_reason != "tool_use":
final_text = "".join([block.text for block in response.content if block.type == "text"])
return {"answer": final_text, "iterations": iteration + 1}
# Process tool calls
messages.append({"role": "assistant", "content": response.content})
tool_results = []
for block in response.content:
if block.type == "tool_use":
tool_fn = TOOL_REGISTRY.get(block.name)
if not tool_fn:
result = {"error": f"Unknown tool: {block.name}"}
else:
try:
result = tool_fn(**block.input)
except Exception as e:
result = {"error": str(e)}
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": json.dumps(result),
})
messages.append({"role": "user", "content": tool_results})
return {"answer": "I'm having trouble completing this request. Let me create a ticket for human support.", "iterations": max_iterations}
Step 4 — Test it
Test scenarios:
# Scenario 1: Simple order lookup
agent.run("Where's my order #12345?", customer_id="cust_001")
# Expected: tool call → get_order_status → response with tracking info
# Scenario 2: Refund request, within window
agent.run("I want to return my shoes from order #12345", customer_id="cust_001")
# Expected: check_refund_eligibility → initiate_refund → confirmation
# Scenario 3: Refund request, outside window
agent.run("I want to return order #99999 from last year", customer_id="cust_001")
# Expected: check_refund_eligibility → declined → empathetic explanation + ticket
# Scenario 4: Frustrated customer
agent.run("This is ridiculous, I've been waiting 3 weeks for my package", customer_id="cust_001")
# Expected: order lookup + de-escalation + ticket creation
# Scenario 5: Out-of-scope
agent.run("Can I get a job at Acme?", customer_id="cust_001")
# Expected: politely decline + ticket to HR
For each scenario, verify:
- Correct tools were called.
- Tools were called in correct order.
- Response addresses the customer's concern.
- No fabricated data.
Step 5 — Production safeguards
def run_agent_safe(user_message, customer_id, max_iterations=10):
# Rate limit
if rate_limiter.too_many(customer_id):
return {"error": "Rate limit exceeded"}
# Log request
request_id = str(uuid.uuid4())
log_request(request_id, customer_id, user_message)
try:
result = run_agent(user_message, customer_id, max_iterations)
log_response(request_id, result)
return result
except Exception as e:
log_error(request_id, str(e))
create_support_ticket(customer_id, "technical", f"Agent error: {e}", "high")
return {"answer": "Sorry, I'm having technical trouble. A human will follow up."}
Safeguards include:
- Rate limiting per customer.
- Logging every request, tool call, and response.
- Capped iterations.
- Auto-escalation on error.
- Tool-call timing (alert if a tool is repeatedly called or takes too long).