Source: Hybrid LLM, cost-efficient and quality-aware query routing (Ding et al., ICLR 2024)
Paper routing each query to a small or large model by predicted difficulty, with a quality knob tunable at test time; up to 40% fewer large-model calls without quality loss.
Contents
- What: research paper, ICLR 2024 (arXiv:2404.14618)
- Authors: Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan, Ahmed Hassan Awadallah
- Date: 2024-04-22 (arXiv)
- Original: arXiv:2404.14618
Summary
A pre-generation router predicts whether a query is easy enough for a small model given a desired quality level, so the expensive model is called only when needed.
Key claims
- Routing is based on predicted query difficulty and a target quality level.
- Up to 40% fewer calls to the large model with no drop in response quality.
- The quality level can be changed at test time to trade quality for cost without retraining.