Source: Notebook: LLM Evaluation & Observability

The owner's tutorial on eval datasets, offline graders, LLM-as-judge, online observability and the eval flywheel.

Contents

Summary

A practical introduction to proving an LLM app works: treat evaluation as a dataset problem, gate deploys with offline evals, monitor production with tracing and sampled online evals, and feed every failure back into the dataset.

Key claims

  • LLM evaluation is hard because outputs are open-ended, non-deterministic, prone to silent regressions on prompt or model changes, and compounding across agent steps.
  • Start with 20-50 hand-labelled cases covering happy paths, hard cases, edge cases and every past failure; never let eval cases leak into few-shot prompts.
  • Four grader families: exact/rule, reference-based similarity, LLM-as-judge, human.
  • An offline harness loops over the dataset, grades, and fails CI if the pass rate drops below baseline minus a tolerance.
  • LLM judges need a concrete rubric, a reason with each score, preference for pairwise comparison where possible, bias awareness, and validation against human labels.
  • For RAG, separate retrieval quality (recall, MRR) from generation quality (groundedness given the retrieved chunks).
  • Change one variable at a time and re-score against the baseline.
  • Production observability: trace every request (full prompt, retrieved context, tool calls, model and params, output, latency, tokens, cost), with agent chains as nested spans.
  • Online signals to watch: implicit feedback (retries, edits, thumbs, abandonment), sampled LLM-as-judge scores on live traffic, guardrail hits, and drift in latency, cost and refusal rate; ops metrics include latency p50/p95; alert on regressions.
  • A prompt tweak or model upgrade can quietly break cases that used to work.

Feeds

LLM evaluation · LLM-as-judge · LLM observability

Suggest changes →