Source: ColPali (Faysse et al., ICLR 2025)
Retrieves document pages by embedding page images with a vision-language model into multi-vector late-interaction embeddings, skipping OCR and layout parsing; introduces the ViDoRe benchmark.
Contents
- What: research paper (arXiv:2407.01449, ICLR 2025)
- Author/publisher: Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, Pierre Colombo
- Date: 2024-06-27 (v6 2025-02-28)
- Original: arXiv:2407.01449
Summary
ColPali questions whether retrieval over visually rich documents needs a parsing pipeline at all. It fine-tunes PaliGemma-3B to emit one embedding per image patch of a page and scores queries with ColBERT-style late interaction. On the new ViDoRe benchmark it beats text pipelines built on Unstructured parsing plus captioning, and indexes pages far faster.
Key claims
- Each page becomes about 1,030 vectors of dimension 128 (1,024 patches plus a few prompt tokens); about 257.5 KB per page in float16.
- Offline indexing on an NVIDIA L4: 0.39 s per page for ColPali vs 7.22 s per page for an Unstructured pipeline (layout 0.81 s, OCR 2.67 s, captioning 3.71 s).
- ViDoRe average nDCG@5: 81.3 for ColPali vs 67.0 for the best text baseline (Unstructured + captioning + BGE-M3).
- Retrieval only: the generator still needs either the page image or extracted text to answer.