Source: LangSmith evaluation concepts and SDK docs
Official LangSmith docs on datasets, examples, splits, versions, experiments, evaluator types, offline vs online evaluation, and the evaluate() SDK call.
Contents
- What: official product documentation
- Publisher: LangChain
- Date: accessed 2026-11-14
- Original: Evaluation concepts, Evaluate an LLM application
Summary
LangSmith evaluation runs a target function over a versioned dataset, scores each output with evaluators, and stores the results as an experiment that can be compared side by side with others. The same evaluators can also run online against production traces.
Key claims
- An example is
inputs+ optional reference outputs (seen only by evaluators) + optional metadata. Splits group examples; datasets version automatically on change and versions can be tagged and pinned in CI. - An experiment is one application version evaluated on one dataset, capturing outputs, scores and traces per example.
- Evaluator types: code (deterministic), LLM-as-judge (reference-free or reference-based), human (annotation queues, single-run or pairwise), pairwise comparison, and a decision-model judge with typed questions.
- Evaluators return feedback:
key,scoreorvalue, optionalcomment. - Offline runs on datasets with references (benchmarking, regression testing, backtesting); online runs on production runs without references, so only reference-free evaluators apply.
- SDK:
Client.create_dataset,create_examples(langsmith>=0.3.13),client.evaluate(target, data=..., evaluators=[...], experiment_prefix=..., max_concurrency=..., metadata=...);aevaluatefor large async jobs. Python evaluators can takeinputs,outputs,reference_outputsand return a bool. openevalsis LangChain’s open-source package of prebuilt evaluators.