Source: Braintrust docs: Eval SDK quickstart

Braintrust's Eval() abstraction of data, task and scores, producing comparable experiments.

Contents

Summary

Braintrust is a hosted eval and observability platform. Its SDK defines an eval as three parts: data (inputs and expected outputs), task (the function under test) and scores (scorer functions). Each run is stored as an experiment that can be compared with earlier ones.

Key claims

  • Scorers come from the open-source autoevals library (e.g. ExactMatch), custom code, or LLM-as-judge.
  • bt eval <file> runs an eval and links to the stored experiment.
  • Experiments record inputs, outputs, scores and metadata and are compared side by side (e.g. before and after a prompt change).
  • Datasets can be uploaded (e.g. CSV mapped to input/expected) and versioned; evals can run in CI to catch regressions.

Feeds

LLM evaluation

Suggest changes →