Source: Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5)
Microsoft paper introducing the E5 open embedding models, trained contrastively on a large curated set of weakly supervised text pairs.
Contents
- What: research paper, arXiv:2212.03533
- Authors: Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei
- Date: submitted 7 December 2022; v2 revised 22 February 2024
- Original: arxiv.org/abs/2212.03533
Summary
E5 shows that a large, carefully filtered corpus of naturally occurring text pairs (CCPairs) is enough to pre-train strong general-purpose embeddings with a contrastive objective, then fine-tune on labelled data. It is the origin of the widely used open E5 family (and later multilingual-e5 and e5-mistral).
Key claims
- Weak supervision from CCPairs plus contrastive pre-training yields single-vector embeddings for retrieval, clustering and classification.
- Zero-shot, E5 is described as the first model to beat the BM25 baseline on BEIR retrieval without labelled data.
- Fine-tuned, it topped MTEB at the time, beating models with 40x more parameters.
- Evaluated on 56 datasets from BEIR and MTEB.