Source: Evaluating Large Language Models Trained on Code (Chen et al., 2021)
The Codex/HumanEval paper, which defined pass@k with an unbiased estimator and showed repeated sampling boosts success.
Contents
- What: research paper, arXiv:2107.03374 (OpenAI)
- Authors: Mark Chen, Jerry Tworek, Heewoo Jun and 55 others
- Date: 7 July 2021 (v2 14 July 2021)
- Original: arxiv.org/abs/2107.03374
Summary
Introduces HumanEval (hand-written Python problems graded by unit tests) and the functional-correctness metric pass@k: the probability that at least one of k samples passes. Gives the standard unbiased estimator still used today.
Key claims
- pass@k is estimated by drawing n ≥ k samples per problem (the paper uses n = 200, k ≤ 100), counting c correct, and averaging 1 − C(n−c, k)/C(n, k) over problems.
- The plug-in estimate 1 − (1 − p̂)^k is biased; the paper gives a numerically stable product form instead of computing binomials directly.
- Optimal temperature depends on k: for a 679M model, T = 0.2 was best for pass@1 and T = 0.8 for pass@100, because diversity helps when only one success is needed.
- Codex-12B solved 28.8% of HumanEval with one sample and 70.2% with 100 samples per problem.
- BLEU correlates poorly with functional correctness: correct and incorrect solutions have overlapping BLEU distributions.