Hybrid Retrieval: BM25 and Vector Search Combined

Retrieval Augmented Generation (RAG) systems often struggle with a fundamental challenge: accurately fetching relevant documents for diverse user queries. Pure keyword search, like BM25, excels at exact matches but fails with synonyms or conceptual queries. Conversely, pure vector search (semantic search) captures conceptual similarity but can miss precise keyword occurrences, especially for named entities or specific product codes. This limitation leads to suboptimal context for Large Language Models (LLMs), resulting in less accurate responses or hallucinations.

Hybrid retrieval, specifically combining BM25 and vector search, was invented to address this dual challenge. By leveraging the strengths of both methods, it ensures that both lexical and semantic relevance are considered during document retrieval. Ignoring this approach in production can lead to brittle RAG systems that perform inconsistently across query types, increasing operational costs due to poor LLM output quality and requiring more human intervention. For developers, it means building more robust and reliable agent systems that can handle a wider range of user inputs effectively. From an enterprise perspective, this translates to higher user satisfaction, reduced operational overhead, and more trustworthy AI agent deployments. Use hybrid retrieval when your application requires high precision for specific terms alongside a deep understanding of query intent, especially in domains with varied terminology or complex information needs. For simple, highly structured data, a single retrieval method might suffice, but for general-purpose agents, hybrid is often superior.

Hybrid retrieval combines multiple search techniques to enhance relevance. This approach typically merges lexical search and semantic search results.

• Lexical Search (e.g., BM25): Focuses on keyword matching between query and document text. It calculates a score based on term frequency (how often a term appears in a document) and inverse document frequency (how rare a term is across the entire corpus). BM25 is effective for precise keyword recall but struggles with synonyms or conceptual queries.

• Semantic Search (Vector Search): Utilizes embedding models to convert text into high-dimensional numerical vectors. Documents and queries are represented as vectors, and relevance is determined by the cosine similarity (or other distance metrics) between these vectors. This method excels at understanding the meaning and context of a query, even with different phrasing, but can sometimes miss exact keyword matches.

• Reciprocal Rank Fusion (RRF): An algorithm used to combine ranked lists from multiple retrieval methods into a single, unified ranked list. RRF assigns a score to each document based on its rank in each individual list, giving higher weight to documents that appear high in multiple lists. It is robust to different scoring scales and does not require score normalization from individual rankers.

• Retrieval Augmented Generation (RAG): An architecture where an LLM retrieves relevant information from an external knowledge base before generating a response. Hybrid retrieval directly improves the 'retrieval' component of RAG, providing richer, more accurate context to the LLM.

• Document Chunks: Large documents are often split into smaller, semantically coherent chunks before indexing. This practice improves retrieval granularity, ensuring that only the most relevant portions of a document are fetched, reducing context window pressure and noise for the LLM.

Real BM25 Implementation

In production, replace MockLexicalIndex with the rank_bm25 library:

from rank_bm25 import BM25Okapi

tokenized_corpus = [doc.split() for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)
scores = bm25.get_scores(query.split())

Hybrid retrieval operates by executing both lexical and semantic search queries in parallel, then merging their results into a single, optimized list. The process ensures comprehensive coverage for diverse query types.

1. Indexing Phase

Documents are first processed for both lexical and semantic indexing. For lexical search, text is tokenized and inverted indices are built, typically for BM25. Simultaneously, document chunks are converted into vector embeddings using a pre-trained embedding model, which are then stored in a vector database. This dual indexing is crucial for enabling parallel retrieval.

2. Query Execution

When a user submits a query, it is simultaneously sent to both the lexical search engine (e.g., Elasticsearch, Solr) and the vector database. The lexical engine processes the query to identify keyword matches and returns a ranked list of document chunks based on BM25 scores. In parallel, the query is embedded into a vector, which is then used to perform a similarity search against the document embeddings in the vector database, yielding another ranked list based on semantic similarity.

3. Result Merging (Reciprocal Rank Fusion)

Each retrieval method produces a ranked list of document chunks. These lists, often with different scoring scales, are then fed into a Reciprocal Rank Fusion (RRF) algorithm. RRF combines these lists by assigning a score to each unique document based on its rank in each individual list. The formula for RRF for a document d across N rankers is typically RRF_score(d) = sum(1 / (k + rank_i(d))) where rank_i(d) is the rank of document d in the i-th ranker's list, and k is a constant (commonly 60) to prevent division by zero and smooth scores. Documents appearing high in multiple lists receive significantly higher RRF scores.

4. Final Ranking and Context Selection

The RRF algorithm outputs a single, unified ranked list of document chunks. This list is then truncated to the top-N most relevant chunks, which are subsequently used as context for the LLM. This final selection ensures that the LLM receives a balanced perspective, incorporating both keyword precision and semantic understanding.

Failure Cases

• Empty Results: If both lexical and semantic searches return no relevant documents, the RRF will also yield an empty list. Agents should implement fallback mechanisms, such as broadening the search scope, querying a general knowledge base, or directly informing the user of the lack of information.
• Disparate Relevance: In cases where one search method yields highly relevant results and the other yields very poor ones, RRF can still provide a reasonable combined list. However, if the k parameter is not tuned, or if one ranker consistently provides noise, the overall quality can degrade. Monitoring retrieval metrics is essential to identify such imbalances.
• Indexing Lag: If the lexical and vector indices are not synchronized, newly added or updated documents might only appear in one index, leading to incomplete retrieval. Robust indexing pipelines with transactional updates or eventual consistency guarantees are necessary.

The conceptual architecture for hybrid retrieval involves distinct components for indexing, retrieval, and fusion. At its core, a Document Store holds the raw text content. This store feeds into two parallel indexing pipelines: a Lexical Indexer and a Vector Indexer. The Lexical Indexer processes text for keyword search, creating an inverted index and storing it in a Lexical Search Engine (e.g., Elasticsearch, OpenSearch). The Vector Indexer uses an Embedding Model to transform document chunks into vectors, which are then stored in a Vector Database (e.g., Pinecone, Weaviate, Chroma).

When a user query arrives, it first hits an API Gateway or Query Orchestrator. This orchestrator dispatches the query in parallel to both the Lexical Search Engine and the Vector Database. The Lexical Search Engine returns a ranked list of document IDs and BM25 scores. Concurrently, the Query Orchestrator sends the query to the Embedding Model to generate a query vector, which is then used by the Vector Database to return a ranked list of document IDs and similarity scores. These two ranked lists flow back to the Query Orchestrator.

The Query Orchestrator then invokes a Reciprocal Rank Fusion (RRF) Module. This module takes the two ranked lists as input, applies the RRF algorithm, and produces a single, unified ranked list of document IDs. Finally, the Query Orchestrator retrieves the actual content for the top-ranked documents from the Document Store and delivers this combined context to the downstream AI Agent or LLM for generation. Execution starts with the user query and ends with the LLM receiving the optimized context.

1. Setting Up the Indexing Environment

Implementing hybrid retrieval requires a robust indexing pipeline capable of handling both lexical and semantic representations of your data. For lexical search, open-source solutions like Elasticsearch or Apache Solr are common. For vector search, dedicated vector databases or vector search capabilities within existing databases (e.g., PostgreSQL with pgvector) are used. The choice depends on scale, existing infrastructure, and feature requirements.

First, define your document structure. Each document should have a unique ID, content, and potentially metadata. Documents are typically chunked into smaller, semantically coherent units (e.g., 200-500 tokens with overlap) to improve retrieval granularity and fit within LLM context windows.

import os
import logging
from typing import List, Dict
from collections import defaultdict

# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

class DocumentChunk:
    def __init__(self, id: str, text: str, metadata: Dict = None):
        self.id = id
        self.text = text
        self.metadata = metadata if metadata is not None else {}

    def __repr__(self):
        return f"DocumentChunk(id='{self.id}', text='{self.text[:50]}...', metadata={self.metadata})"

# In a real system, this would be a connection to Elasticsearch/Pinecone
class MockLexicalIndex:
    def __init__(self):
        self.documents = []
        self.inverted_index = defaultdict(list)

    def add_document(self, chunk: DocumentChunk):
        self.documents.append(chunk)
        # Simulate simple inverted index for BM25-like behavior
        for word in chunk.text.lower().split():
            self.inverted_index[word].append(chunk.id)
        logging.info(f"Added document {chunk.id} to lexical index.")

    def search(self, query: str) -> List[Dict]:
        # Simulate BM25: simple keyword matching for demonstration
        query_words = query.lower().split()
        scores = defaultdict(float)
        for word in query_words:
            for doc_id in self.inverted_index[word]:
                scores[doc_id] += 1 # Simple term frequency score
        
        ranked_results = sorted(scores.items(), key=lambda item: item[1], reverse=True)
        return [{'id': doc_id, 'score': score} for doc_id, score in ranked_results]

class MockVectorIndex:
    def __init__(self, embedding_model):
        self.documents = []
        self.embeddings = {}
        self.embedding_model = embedding_model

    def add_document(self, chunk: DocumentChunk):
        embedding = self.embedding_model.encode(chunk.text)
        self.documents.append(chunk)
        self.embeddings[chunk.id] = embedding
        logging.info(f"Added document {chunk.id} to vector index.")

    def search(self, query: str) -> List[Dict]:
        query_embedding = self.embedding_model.encode(query)
        similarities = {}
        for doc_id, doc_embedding in self.embeddings.items():
            # Simulate cosine similarity
            similarity = self._cosine_similarity(query_embedding, doc_embedding)
            similarities[doc_id] = similarity
        
        ranked_results = sorted(similarities.items(), key=lambda item: item[1], reverse=True)
        return [{'id': doc_id, 'score': score} for doc_id, score in ranked_results]

    def _cosine_similarity(self, vec1, vec2):
        dot_product = sum(v1 * v2 for v1, v2 in zip(vec1, vec2))
        magnitude1 = sum(v**2 for v in vec1)**0.5
        magnitude2 = sum(v**2 for v in vec2)**0.5
        if magnitude1 == 0 or magnitude2 == 0:
            return 0.0
        return dot_product / (magnitude1 * magnitude2)

# Placeholder for a real embedding model (e.g., Sentence Transformers)
class MockEmbeddingModel:
    def encode(self, text: str) -> List[float]:
        # Return a dummy vector for demonstration
        # In production, use a model like 'all-MiniLM-L6-v2'
        return [hash(c) % 1000 / 1000.0 for c in text[:10]] # Simple hash-based vector

2. Implementing Reciprocal Rank Fusion (RRF)

RRF is a robust method for combining ranked lists without requiring score normalization. It assigns a score based on the rank of an item in each list. The standard formula for RRF is 1 / (k + rank), where k is a constant (typically 60) and rank is the 1-based position of the item in a list. Higher ranks (smaller numbers) contribute more to the final score.

def reciprocal_rank_fusion(ranked_lists: List[List[Dict]], k: int = 60) -> List[Dict]:
    fused_scores = defaultdict(float)
    
    for ranked_list in ranked_lists:
        for rank, item in enumerate(ranked_list, start=1):
            doc_id = item['id']
            # RRF formula: 1 / (k + rank)
            fused_scores[doc_id] += 1.0 / (k + rank)
            
    # Sort documents by their fused scores in descending order
    sorted_fused_scores = sorted(fused_scores.items(), key=lambda item: item[1], reverse=True)
    
    # Reconstruct the list with original document content (assuming we can retrieve it)
    # For this example, we'll just return the IDs and scores
    return [{'id': doc_id, 'score': score} for doc_id, score in sorted_fused_scores]

3. Orchestrating Hybrid Retrieval

The final step involves orchestrating the parallel calls to both search indexes and applying RRF. This component acts as the central point for a hybrid retrieval system.

class HybridRetriever:
    def __init__(self, lexical_index: MockLexicalIndex, vector_index: MockVectorIndex):
        self.lexical_index = lexical_index
        self.vector_index = vector_index

    def retrieve(self, query: str, top_k: int = 5) -> List[DocumentChunk]:
        logging.info(f"Executing hybrid retrieval for query: '{query}'")
        
        # 1. Execute lexical search
        lexical_results = self.lexical_index.search(query)
        logging.debug(f"Lexical results: {lexical_results}")

        # 2. Execute vector search
        vector_results = self.vector_index.search(query)
        logging.debug(f"Vector results: {vector_results}")

        # 3. Apply Reciprocal Rank Fusion
        fused_results = reciprocal_rank_fusion([lexical_results, vector_results])
        logging.debug(f"Fused results: {fused_results}")

        # 4. Retrieve original document chunks for the top_k results
        retrieved_chunks = []
        for item in fused_results[:top_k]:
            doc_id = item['id']
            # In a real system, fetch the full chunk from a document store
            # For this example, we'll find it from our mock index's documents
            chunk = next((d for d in self.lexical_index.documents if d.id == doc_id), None)
            if chunk:
                retrieved_chunks.append(chunk)
            else:
                logging.warning(f"Document chunk with ID {doc_id} not found in store.")

        logging.info(f"Successfully retrieved {len(retrieved_chunks)} chunks.")
        return retrieved_chunks

4. Tuning and Optimization

• RRF k parameter: The k constant in RRF influences how quickly the contribution of lower-ranked items diminishes. A higher k gives more weight to lower-ranked items, while a lower k emphasizes top ranks. Tuning k requires empirical evaluation against a representative dataset.
• Number of results from each ranker: While RRF is robust, sending an excessive number of low-relevance results from individual rankers can dilute the quality of the fused list. It's common to limit the number of results from each individual search (e.g., top_100 from lexical, top_100 from vector) before fusion.
• Embedding Model Selection: The quality of the vector search is directly dependent on the chosen embedding model. Select a model that is well-suited for your domain and language. Consider fine-tuning if domain-specific performance is critical.
• Chunking Strategy: The size and overlap of document chunks significantly impact retrieval. Overlapping chunks can help capture context that spans chunk boundaries, while smaller chunks can improve precision. Experimentation is key.