Source: EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

Paper on the third generation of EAGLE draft heads, which fuse multi-layer target features and predict tokens directly.

Contents

Summary

EAGLE drafts using the target model’s own hidden features. EAGLE-3 drops feature prediction, predicts tokens directly from a fusion of low, middle and top layer features, and trains with a technique the authors call training-time test so the draft scales with more training data.

Key claims

  • Up to 6.5x speedup over vanilla decoding, about 1.4x over EAGLE-2.
  • 1.38x throughput gain at batch size 64 inside SGLang.
  • Multi-layer feature fusion replaces top-layer-only features.

Feeds

Speculative decoding

Suggest changes →