Source: openai/evals (GitHub)
OpenAI's open-source eval framework and registry, where basic and model-graded evals are defined in YAML and JSONL.
Contents
- What: open-source repository README
- Publisher: OpenAI
- Date: accessed 2026-11-14 (repository first released March 2023)
- Original: github.com/openai/evals
Summary
A framework and public registry of evals. Basic (match-style) and model-graded evals need no code: a JSONL dataset plus YAML parameters in the registry. The README points users to configuring evals in the OpenAI dashboard.
Key claims
- Registry data is stored with Git LFS under
evals/registry/data; eval definitions are YAML underevals/registry/evals. - Basic and model-graded templates need only data and YAML; the project no longer accepts evals with custom code.
- Install with
pip install evals(Python 3.9+). - The hosted dashboard version of Evals is being deprecated: read-only from 31 October 2026, shut down 30 November 2026 (per OpenAI’s eval guide).