Source: Scaling LLM Test-Time Compute Optimally (Snell et al., 2024)

Paper showing that difficulty-adaptive test-time compute beats best-of-N by ~4x and can beat a 14x larger model on easy and medium problems.

Contents
  • What: research paper (arXiv 2408.03314)
  • Authors: Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar (UC Berkeley / Google DeepMind)
  • Date: 2024-08-06
  • Original: arXiv:2408.03314

Summary

Asks how much a fixed inference budget can improve a model on hard prompts, and how best to spend it. It studies two mechanisms: search against a process reward model (PRM) verifier, and sequential self-revision. It finds that the best strategy depends on question difficulty.

Key claims

  • Two levers: (1) search (best-of-N, beam search, lookahead) scored by a PRM; (2) changing the proposal distribution through sequential revisions.
  • Sequential revision acts as local refinement and parallel sampling as global search. Easy questions do best purely sequentially; harder ones need a mix. With naive answer selection, about 38% of correct answers get revised back into incorrect ones, so a selection step is needed.
  • Beam search beats best-of-N at low budgets and on medium-hard questions, but over-optimizes the PRM on easy ones. Nothing helps much on the hardest bin.
  • “Compute-optimal” allocation by estimated difficulty matches best-of-N with up to ~4x less compute.
  • FLOPs-matched against a ~14x larger model, extra test-time compute wins on easy/medium questions and when inference load is low relative to pretraining. Pretraining wins on the hardest questions and under heavy inference load. The two are not fully interchangeable.

Feeds

Test-time compute · Reasoning models

Suggest changes →