Source: RULER: What's the Real Context Size of Your Long-Context Language Models? (NVIDIA)

NVIDIA benchmark extending needle-in-a-haystack with multi-needle, multi-hop and aggregation tasks; about half of 32K+ models fail at 32K.

Contents
  • What: benchmark paper (arXiv:2404.06654), COLM 2024
  • Authors: Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg (NVIDIA)
  • Date: 9 April 2024 (v3 6 August 2024)
  • Original: arxiv.org/abs/2404.06654 · code on GitHub

Summary

RULER is a synthetic, configurable long-context benchmark (13 tasks) that goes beyond vanilla NIAH: multiple needles and needle types, multi-hop variable tracing, aggregation (e.g. common-word extraction) and QA with distractors. It separates a model’s claimed context length from its effective length, the longest input at which it still meets a quality bar.

Key claims

  • 17 long-context models evaluated, all claiming 32K or more.
  • Almost all score near perfect on vanilla NIAH, yet show large drops as length and task complexity grow.
  • Only about half keep satisfactory performance at 32K.
  • Yi-34B, advertised at 200K, shows large headroom for improvement as length increases.

Feeds

Long-context degradation

Suggest changes →