Source: GPT Semantic Cache (Regmi & Pun, 2024)

Reports that caching query embeddings in Redis and serving similar queries from cache cut LLM API calls by up to 68.8% with high precision on their test categories.

Contents
  • What: arXiv preprint 2411.05276 (v3, revised 2024-12-09)
  • Authors: Sajal Regmi, Chetan Phakami Pun
  • Date: 2024-11-08
  • Original: arXiv:2411.05276

Summary

Stores embeddings of user queries in an in-memory store (Redis); when a new query is semantically close to a cached one, the stored response is returned without calling the LLM.

Key claims

  • Up to 68.8% fewer API calls across query categories.
  • Cache hit rates of 61.6% to 68.8%.
  • Over 97% of hits judged correct (“positive hits”) on their evaluation sets.

Feeds

Cost and latency optimization

Suggest changes →