This post explores the history and advancements in text embedding models, including the tricks used to build state-of-the-art models and the role of contrastive loss in fine-tuning. It also highlights the importance of standardized evaluation and provides insights into the training recipe for top-performing models.
Table of contents
How to Build a State-of-the-art Text Embedding ModelA (very not comprehensive) history of embeddingsLanguage-model-based embeddings have taken over information retrievalThe modern state-of-the-art recipe for building embeddingsTrick 1: Start with a pre-trained general-purpose language modelTrick 2: Fine-tune for information retrieval with contrastive lossTrick 3: Prefix your queriesTrick 4: Scale to large batch sizes to optimally leverage in-batch negativesTrick 5: Finish training with some hard negativesWhy does this work?ConclusionDefine the future of AI with usAcknowledgementsReferences24 Impressions