Source: Kamphuis et al. 2020: Which BM25 Do You Mean?

ECIR 2020 reproducibility study finding no significant effectiveness differences among eight BM25 variants, including Lucene's lossy length encoding.

Contents
  • What: reproducibility paper, ECIR 2020
  • Author: Chris Kamphuis, Arjen P. de Vries, Leonid Boytsov, Jimmy Lin
  • Date: 2020
  • Original: Preprint PDF

Summary

“BM25” names a family of scoring functions. The authors implement eight variants (Robertson et al., Lucene default, Lucene with exact lengths, ATIRE, BM25L, BM25+, BM25-adpt, TFl∘δ∘p×IDF) over the same Lucene/Anserini index, exported to a relational database, and compare them on three TREC newswire collections (Robust04, Core17, Core18).

Key claims

  • The Robertson et al. IDF, log((N - df + 0.5)/(df + 0.5)), goes negative when a term occurs in more than half the documents; Lucene adds 1 inside the log to avoid that.
  • Lucene (as of version 8) stores document length lossily in one byte, so only 256 distinct lengths exist and the length-normalization term can be precomputed per length.
  • ATIRE uses log(N/df) as IDF and multiplies tf by (k1 + 1), which does not change rankings.
  • BM25L and BM25+ add a constant δ to counter BM25’s over-penalizing of long documents; BM25-adpt picks a k1 per term.
  • With k1 = 0.9 and b = 0.4 (Anserini’s defaults), ANOVA and Tukey’s HSD find no significant differences between any variants on any collection.
  • Mundane choices such as stopword lists often move results more than the choice of BM25 variant.
  • Lucene’s lossy length approximation costs negligible effectiveness and its speed benefit is also negligible.
  • Using a variant that avoids negative scores may still be worthwhile.

Feeds

BM25

Suggest changes →