Source: Dated Data: Tracing Knowledge Cutoffs in Large Language Models

Johns Hopkins paper showing a model's effective knowledge cutoff often differs from its reported cutoff, and why.

Contents
  • What: research paper (arXiv:2403.12958), COLM 2024
  • Authors: Jeffrey Cheng, Marc Marone, Orion Weller, Dawn Lawrie, Daniel Khashabi, Benjamin Van Durme
  • Date: 19 March 2024 (v2 17 September 2024)
  • Original: arxiv.org/abs/2403.12958

Summary

Vendors publish a single “knowledge cutoff” date. The paper defines an effective cutoff per resource or topic and estimates it by probing a model against many time-stamped versions of the same documents (e.g. Wikipedia revisions) and finding which version it fits best. Effective cutoffs often lag the reported date.

Key claims

  • Effective cutoffs frequently differ from reported cutoffs, and vary by source.
  • New CommonCrawl dumps contain a large share of old content, so “recent” crawls skew the model toward older facts.
  • Deduplication misses semantic and near-lexical duplicates, over-weighting older versions of documents.
  • Users should not assume uniform knowledge up to the stated date.

Feeds

LLM limitations · Hallucination

Suggest changes →