Constructing a complicated native LLM RAG pipeline by combining dense embeddings with BM25
The essential Retrieval-Augmented Technology (RAG) pipeline makes use of an encoder mannequin to seek for comparable paperwork when given a question.
That is additionally known as semantic search as a result of the encoder transforms textual content right into a high-dimensional vector illustration (known as an embedding) during which semantically comparable texts are shut collectively.
Earlier than we had Massive Language Fashions (LLMs) to create these vector embeddings, the BM25 algorithm was a very talked-about search algorithm. BM25 focuses on necessary key phrases and appears for precise matches within the obtainable paperwork. This method is named key phrase search.
If you wish to take your RAG pipeline to the following degree, you may need to strive hybrid search. Hybrid search combines the advantages of key phrase search and semantic search to enhance search high quality.
On this article, we are going to cowl the speculation and implement all three search approaches in Python.
Desk of Contents
· RAG Retrieval
∘ Key phrase Search With BM25
∘ Semantic Search With Dense Embeddings
∘ Semantic Search or Hybrid Search?
∘ Hybrid Search
∘ Placing It All Collectively
·…
