Offline Evaluations, Golden Datasets & LLM Judges

This lesson on Offline Evaluations, Golden Datasets & LLM Judges is hands-on and example-driven. You will construct curated golden datasets, implement deterministic code and reference-anchored LLM-as-a-judge evaluators, and execute automated regression experiments using the LangSmith Python SDK and UI.

What You'll Be Able To Do

• Construct structured golden datasets with explicit input and reference output schemas in LangSmith.
• Promote single and bulk production telemetry traces into persistent evaluation test suites.
• Implement deterministic Python code evaluators to validate output constraints such as word counts.
• Author reference-backed LLM-as-a-judge rubric evaluators for open-ended qualitative scoring.
• Execute automated test suites using the LangSmith SDK evaluate function and custom input adapters.
• Compare multi-model experiment runs side-by-side in the UI to isolate factual regressions.

Detailed Concept Walkthrough

1. Golden Datasets and Population Strategies

Golden datasets act as static, verifiable ground-truth benchmarks used to measure application quality across prompt, model, or code iterations.

• Schema Architecture: Datasets define explicit input keys (such as question) alongside reference output keys (such as answer) to set expected baseline behaviors.
• Telemetry Promotion: Developers populate datasets through manual authoring, synthetic generation, or by promoting recorded production traces directly from project telemetry.
• Ground-Truth Anchoring: Storing high-quality reference targets alongside inputs ensures downstream qualitative evaluators have an objective standard for comparison.

from langsmith import Client

client = Client()
dataset = client.create_dataset(
    dataset_name="ELI5 Golden Dataset",
    description="Curated explain-like-I'm-five question and answer pairs."
)
client.create_examples(
    inputs=[{"question": "Why is the sky blue?"}],
    outputs=[{"answer": "Air molecules scatter blue light more than other colors."}],
    dataset_id=dataset.id
)

Key Takeaway: Curating production traces into golden datasets establishes fixed, repeatable ground-truth baselines for continuous regression testing.

2. Deterministic and LLM-as-a-Judge Evaluators

Comprehensive evaluation pairs deterministic code rules for structural criteria with LLM judges for open-ended semantic accuracy.

• Deterministic Code Evaluators: Python functions inspect run output dictionaries to score verifiable constraints like length limits, schema conformance, or regex matches without model inference costs.
• LLM-as-a-Judge Rubrics: Prompt-based judges compare candidate model outputs against reference answers and inputs to score nuanced qualities like factual correctness and simplicity.
• Reference Context Nuance: Passing ground-truth reference outputs to the judge anchors evaluation rubrics and prevents subjective scoring drift across model variants.

def brevity_evaluator(run, example) -> dict:
    # Deterministic check for word count limit
    output_text = run.outputs.get("output", "")
    passed = len(output_text.split()) <= 200
    return {"key": "brevity_pass", "score": int(passed)}

def correctness_judge(run, example) -> dict:
    # Reference-backed qualitative evaluation
    reference = example.outputs.get("answer")
    prediction = run.outputs.get("output")
    # Evaluator compares prediction to reference using a rubric
    return {"key": "correctness", "score": 1}

Key Takeaway: Use deterministic code functions for strict constraints and anchor LLM judges with reference outputs for qualitative scoring.

3. SDK Experiment Execution and Input Adapters

The SDK evaluate function orchestrates batch test runs across datasets, mapping inputs to candidate models and recording metrics.

• Input-Adapter Mapping: Wrapper functions adapt dataset input keys to match the exact argument signature expected by the target application chain or model.
• Execution Lifecycle: The evaluate() function iterates through dataset records, invokes the adapter pipeline, passes outputs to evaluators, and streams results to LangSmith.
• Run Namespacing: Assigning an experiment_prefix automatically namespaces test batches, allowing clean categorization across different prompts or model architectures.

from langsmith.evaluation import evaluate

def target_runner(inputs: dict) -> dict:
    # Adapter mapping dataset 'question' to application input
    query = inputs["question"]
    response = my_agent.invoke({"input": query})
    return {"output": response.content}

results = evaluate(
    target_runner,
    data="ELI5 Golden Dataset",
    evaluators=[brevity_evaluator, correctness_judge],
    experiment_prefix="o3-mini-eval"
)

Key Takeaway: Input adapters bridge dataset schemas to target applications, allowing the SDK evaluate function to execute automated test batches.

4. Dataset Auto-Evaluators and Side-by-Side Analysis

Dataset-level auto-evaluators automate quality checks on all runs, enabling side-by-side differential analysis in the LangSmith UI.

• Persistent Auto-Evaluators: Configuring evaluator rubrics directly on a dataset in the UI ensures they execute automatically on every subsequent experiment without SDK script changes.
• Differential Experiment Comparison: Selecting multiple experiment runs in the UI isolates regressions by comparing aggregate pass rates and metric scores.
• Trace-Level Diagnostics: Drilling down into individual example diffs reveals specific model failures, such as omitted facts or hallucinations between model candidates.

# UI Auto-Evaluators run automatically during SDK evaluation
# No local evaluator list required if auto-evaluators are configured
benchmark_run = evaluate(
    target_runner,
    data="ELI5 Golden Dataset",
    experiment_prefix="gpt-3.5-benchmark"
)

Key Takeaway: Dataset auto-evaluators enforce permanent quality standards across all experiments to power side-by-side regression comparisons.

Topics Covered in Offline Evaluations, Golden Datasets & LLM Judges

1. Golden Dataset Setup (0:00 - 2:16) — Exploration of dataset architecture, schema definitions, and an existing Explain Like I'm Five evaluation set.
2. Dataset Population (2:16 - 4:18) — Demonstration of populating evaluation datasets via manual entry, synthetic generation, and promoting production telemetry traces.
3. SDK Configuration (4:18 - 5:43) — Setting up the LangSmith Python SDK client in a notebook and switching models to o3-mini.
4. Evaluator Implementation (5:43 - 8:06) — Writing a deterministic word-count brevity evaluator and an LLM-as-a-judge factual correctness scoring function.
5. SDK Evaluation Execution (8:06 - 8:44) — Executing batch test runs across the golden dataset using input adapters and the evaluate function.
6. Dataset Auto-Evaluators (8:44 - 10:35) — Navigating experiment results in the UI and inspecting persistent dataset-level simplicity auto-evaluators.
7. Model Comparison (10:35 - 12:16) — Performing side-by-side regression analysis between o3-mini and GPT-3.5 to isolate factual omissions in trace logs.

LangChain & LangGraph for AI Agents Cheat Sheet

• client.create_dataset(dataset_name, description) — Creates a named evaluation dataset in LangSmith

  client.create_dataset(dataset_name="ELI5 Dataset")
  

• client.create_examples(inputs, outputs, dataset_id) — Adds structured input and reference examples to dataset

  client.create_examples(inputs=[{"q": "hi"}], outputs=[{"a": "hello"}], dataset_id=ds.id)
  

• def custom_evaluator(run, example) -> dict — Defines custom scoring function inspecting run and example

  def length_eval(run, example): return {"key": "len", "score": len(run.outputs["output"])}
  

• evaluate(target, data, evaluators, experiment_prefix) — Executes evaluation suite against target application and dataset

  evaluate(runner, data="ELI5 Dataset", evaluators=[length_eval], experiment_prefix="test-run")
  

• def input_adapter(inputs: dict) -> dict — Maps dataset dictionary keys to model invocation inputs

  def runner(inputs: dict): return {"output": chain.invoke({"query": inputs["question"]})}
  

Comparison Table

Common Pitfalls

• Mistake: Omitting reference outputs from LLM judge prompts. Avoid: Always include ground-truth reference outputs in judge rubrics to prevent subjective scoring drift.
• Mistake: Passing raw dataset inputs directly to applications with mismatched parameter signatures. Avoid: Wrap model calls in an adapter function that maps dataset schema keys to application inputs.
• Mistake: Redefining identical evaluators across separate local testing scripts. Avoid: Configure persistent auto-evaluators at the dataset level in the LangSmith UI for global execution.

FAQs

• Why should I convert production traces into a golden dataset instead of using synthetic data alone? Production traces capture real-world edge cases, user phrasing, and failure modes that synthetic data generators frequently overlook.
• What is the primary difference between local SDK evaluators and UI auto-evaluators? Local SDK evaluators execute only during that specific script run, whereas UI auto-evaluators persist on the dataset and execute across every future experiment.
• How does an input adapter function prevent evaluation runtime errors? Input adapters translate the dataset's stored schema keys into the exact parameter dictionary expected by the candidate agent pipeline.