Corrective RAG (CRAG): Context Quality Loops
Source: mortalapps.com- Corrective RAG (CRAG) is an advanced RAG technique that evaluates the quality of retrieved context before generating a response.
- It solves the problem of irrelevant or low-quality retrieved documents leading to inaccurate or hallucinated LLM outputs.
- In production, CRAG improves response accuracy, reduces hallucinations, and enhances user trust by ensuring context relevance.
- CRAG dynamically triggers re-retrieval from vector stores or falls back to web search if initial context quality is deemed insufficient.
- Implementing CRAG in Python agent systems, particularly with frameworks like LlamaIndex, involves LLM-based context grading and conditional execution paths.
Why This Matters
Traditional Retrieval Augmented Generation (RAG) systems often retrieve documents based solely on semantic similarity, which does not guarantee the relevance or quality of the information for answering a specific query. This can lead to the Large Language Model (LLM) synthesizing responses from poor context, resulting in hallucinations, irrelevant answers, or a degraded user experience. Corrective RAG (CRAG) addresses this by introducing a critical evaluation step: it assesses the quality of the retrieved context *before* passing it to the generator LLM. This approach was invented to mitigate the inherent unreliability of raw retrieval scores and to inject a layer of intelligent self-correction into the RAG pipeline. If ignored in production, systems risk consistently providing low-quality or incorrect information, eroding user trust and increasing operational costs due to error handling and manual intervention. For developers building robust AI agent systems in Python, CRAG offers a pathway to significantly enhance the reliability and factual grounding of their applications. It is particularly useful when dealing with diverse or noisy data sources, or when high-stakes applications demand verifiable accuracy. In contrast, simpler RAG might suffice for less critical applications where occasional inaccuracies are tolerable, but for enterprise-grade solutions, CRAG provides a necessary layer of validation, ensuring that the context provided to the LLM is fit for purpose, thereby reducing the incidence of 'garbage in, garbage out' scenarios.
Core Concepts
Corrective RAG (CRAG) introduces a feedback loop to enhance the quality of retrieved context in RAG systems. Unlike standard RAG, which directly passes retrieved documents to the LLM, CRAG first evaluates the relevance and utility of these documents.
- Context Quality Grading: This is the central mechanism where an LLM acts as a 'grader' to assess the retrieved documents. The grader LLM is prompted to determine if the context is sufficient, relevant, and directly answers the user's query. The output is typically a categorical score or a confidence level.
- Retrieval Score: The initial score provided by the vector database or search index, indicating semantic similarity or keyword match. CRAG recognizes that a high retrieval score does not always equate to high contextual utility.
- Iterative Retrieval: If the initial context quality is deemed low by the grader LLM, CRAG can trigger a re-retrieval process. This might involve modifying the query, expanding search parameters, or using a different retrieval mechanism (e.g., keyword search instead of vector search).
- Web Fallback: A sophisticated CRAG strategy includes a fallback to external web search if internal knowledge bases fail to provide adequate context. This broadens the agent's knowledge acquisition capabilities, especially for current events or niche topics not present in the vector store.
- Generation with Verified Context: Only after the context has passed the quality grading (or after successful re-retrieval/web fallback) is it passed to the main generator LLM. This ensures that the LLM operates on a higher-fidelity information set, reducing the likelihood of generating inaccurate or ungrounded responses.
How It Works
The Corrective RAG (CRAG) process integrates an explicit context evaluation step into the standard RAG pipeline, creating a dynamic feedback loop. This ensures that the Large Language Model (LLM) receives high-quality, relevant information for generation.
1. Initial Retrieval
Upon receiving a user query, the system performs an initial retrieval from its configured knowledge sources, typically a vector database. This step uses standard retrieval techniques like semantic search to fetch a set of candidate documents or chunks based on their similarity to the query. The output is a list of retrieved context items.
2. Context Quality Grading
The retrieved context, along with the original user query, is then sent to a dedicated 'grader' LLM. This LLM is prompted to evaluate the relevance, sufficiency, and overall quality of the retrieved information for answering the query. The grader LLM outputs a judgment, often a categorical decision (e.g., 'Relevant', 'Irrelevant', 'Needs More Context') or a confidence score. If the grader LLM fails to respond or provides an unparseable output, the system logs the error and can either default to 'Irrelevant' or trigger a retry with a more robust prompt.
3. Decision Point: Re-retrieval or Web Fallback
Based on the grader LLM's judgment, the system makes a decision:
- Sufficient Context: If the context is deemed high-quality and relevant, it proceeds to the generation phase.
- Insufficient/Irrelevant Context: If the context is poor, the system initiates a re-retrieval attempt. This might involve:
- Query Rewriting: The grader LLM (or another specialized LLM) rephrases the original query to improve retrieval effectiveness.
- Expanded Search: Increasing the number of retrieved documents or widening the search scope.
- Alternative Retrieval: Switching from vector search to keyword search (BM25) or a hybrid approach.
- Web Search Fallback: If internal knowledge bases consistently fail to provide adequate context after re-retrieval attempts, the system can trigger a web search using the original or rewritten query. This step is typically rate-limited and includes timeout mechanisms. If web search fails or returns no relevant results, the system logs this and proceeds to generation with a warning or a 'cannot answer' response.
4. Iterative Grading (Optional)
After a re-retrieval or web search, the newly acquired context can optionally be passed through the context quality grader again. This creates an iterative loop, ensuring that each new piece of information is validated before being used. This loop typically has a predefined maximum iteration count to prevent infinite loops.
5. Response Generation
Once a satisfactory set of context documents is assembled and validated, it is passed to the main generator LLM along with the original user query. The generator LLM synthesizes a coherent and accurate response based on this verified information. If all retrieval attempts fail, the generator LLM might be prompted to state that it cannot answer the question due to lack of information, rather than hallucinating.
Architecture
The Corrective RAG (CRAG) architecture extends a standard RAG pipeline with an intelligent context evaluation and dynamic retrieval mechanism. The system begins with a User Interface (UI) or API Gateway receiving a user's query, which is then routed to the Agent Orchestrator.
- Agent Orchestrator: This central component manages the overall workflow. It receives the user query, coordinates retrieval, context grading, and generation. It also handles retry logic and fallback mechanisms.
- Retriever Module: Connected to the Orchestrator, this module interfaces with various knowledge sources, primarily a Vector Store (e.g., Pinecone, Weaviate, Chroma) and potentially a Keyword Search Index. It takes a query and returns relevant document chunks.
- Grader LLM: A dedicated Large Language Model instance, invoked by the Orchestrator, is responsible for evaluating the quality of the retrieved context. It receives the user query and the retrieved documents, outputting a judgment (e.g., 'good', 'bad', 're-retrieve').
- Web Search API: An optional external service (e.g., Google Search API, Bing Search API) that the Orchestrator can invoke if the Grader LLM determines that internal knowledge sources are insufficient. This provides a dynamic, up-to-date information source.
- Generator LLM: The final Large Language Model instance, also invoked by the Orchestrator, which synthesizes the ultimate response to the user. It receives the original query and the *validated* context from either the Retriever Module or the Web Search API.
Execution starts with the user query entering the Agent Orchestrator. The Orchestrator first calls the Retriever. The retrieved context flows to the Grader LLM. The Grader's judgment flows back to the Orchestrator, which then decides whether to re-invoke the Retriever (possibly with a modified query), call the Web Search API, or proceed to the Generator LLM. Finally, the Generator LLM's response flows back through the Orchestrator to the UI/API Gateway and then to the user.
The CRAG Grading Prompt Design
The effectiveness of CRAG hinges on the quality of the grading prompt. This prompt instructs an LLM to act as an evaluator, assessing the relevance and sufficiency of retrieved documents for a given query. A well-designed prompt should clearly define the criteria for 'good' context and the expected output format.
Consider a prompt structure that includes:
- Role Assignment: Explicitly tell the LLM it's a 'Context Grader'.
- Task Definition: Clearly state the goal: evaluate if the provided
CONTEXTis sufficient and relevant to answer theQUERY. - Grading Criteria: Define what constitutes 'sufficient' and 'relevant'. For example, 'Does the context directly contain the answer?', 'Is it free of contradictory information?', 'Is it comprehensive enough?'.
- Output Format: Specify a structured output, such as JSON, with fields like
grade(e.g., 'perfect', 'good', 'bad', 'irrelevant') andreasoning. - Examples (Few-shot): Provide 1-2 examples of queries, contexts, and their corresponding grades/reasoning to guide the LLM's judgment.
{
"role": "Context Grader",
"task": "Evaluate the provided CONTEXT for its relevance and sufficiency to answer the QUERY. Output a grade and reasoning.",
"criteria": [
"Perfect: The context directly and comprehensively answers the query.",
"Good: The context contains relevant information but might require minor inference or additional detail.",
"Bad: The context is partially relevant but contains significant irrelevant information or is insufficient.",
"Irrelevant: The context does not address the query at all."
],
"output_format": {
"grade": "[perfect|good|bad|irrelevant]",
"reasoning": "[explanation of the grade]"
},
"query": "What is the capital of France?",
"context": "Paris is the capital and most populous city of France.",
"expected_grade": "perfect",
"expected_reasoning": "The context directly states the capital of France."
}Implementing Context Re-ranking and Filtering
Before grading, initial retrieval often yields more documents than necessary. Re-ranking and filtering can optimize the context presented to the grader LLM, reducing token costs and improving grading accuracy. Techniques include:
- Cross-Encoder Re-rankers: Models like
cohere-rerankorbge-rerankercan re-score documents based on query-document pair relevance, providing a more nuanced ranking than initial embedding similarity. - Diversity Search: Algorithms like Maximal Marginal Relevance (MMR) can select documents that are both relevant to the query and diverse from each other, preventing redundancy.
- Thresholding: Discarding documents below a certain similarity score or re-ranker score.
Dynamic Retrieval Strategy: Discard and Re-retrieve
When the grader LLM determines the context is 'bad' or 'irrelevant', the system must dynamically adjust its retrieval strategy. This involves a loop:
- Analyze Grader Feedback: Use the
reasoningfrom the grader to understand *why* the context was insufficient. Was it too narrow? Too broad? Outdated? - Query Transformation: If the feedback suggests the original query was poor, an LLM can be used to rewrite or expand the query. For example, if the query was
Code Example
import os
import logging
from typing import List, Optional
from llama_index.core import VectorStoreIndex, Document
from llama_index.core.schema import NodeWithScore # Corrected import for NodeWithScore
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers import CompactAndRefine
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.postprocessor import # removed
# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
# Ensure OpenAI API key is set
if not os.environ.get("OPENAI_API_KEY"): # Use os.environ.get for production quality
raise ValueError("OPENAI_API_KEY environment variable not set.")
# Initialize LLM and Embedding models
llm = OpenAI(model="gpt-4o", temperature=0.1)
embed_model = OpenAIEmbedding(model="text-embedding-3-small")
# --- Step 1: Prepare a simple knowledge base ---
documents = [
Document(text="The capital of France is Paris. Paris is known for its Eiffel Tower."),
Document(text="The capital of Germany is Berlin. Berlin has a rich history."),
Document(text="The Amazon rainforest is the largest tropical rainforest in the world."),
Document(text="Mount Everest is the Earth's highest mountain above sea level."),
Document(text="The currency of Japan is the Japanese Yen.")
]
node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=20)
nodes = node_parser.get_nodes_from_documents(documents)
index = VectorStoreIndex(nodes, embed_model=embed_model)
retriever = index.as_retriever(similarity_top_k=2)
# --- Step 2: Define the Context Grader LLM ---
# This LLM evaluates if the retrieved context is sufficient
GRADER_PROMPT_TEMPLATE = """
As a Context Grader, evaluate the provided CONTEXT for its relevance and sufficiency to answer the QUERY.
Output a JSON object with a 'grade' (string: 'perfect', 'good', 'bad', 'irrelevant') and 'reasoning' (string).
Criteria:
- 'perfect': The context directly and comprehensively answers the query.
- 'good': The context contains relevant information but might require minor inference or additional detail.
- 'bad': The context is partially relevant but contains significant irrelevant information or is insufficient.
- 'irrelevant': The context does not address the query at all.
QUERY: {query}
CONTEXT: {context}
JSON Output:
"""
async def grade_context(query: str, context: str) -> dict:
try:
response = await llm.apredict(GRADER_PROMPT_TEMPLATE.format(query=query, context=context))
# Attempt to parse JSON, handle potential malformed output
grade_output = json.loads(response) # Using eval for simplicity, but json.loads is safer for production
return grade_output
except Exception as e:
logger.error(f"Error grading context: {e}. LLM response: {response}")
return {"grade": "bad", "reasoning": "Failed to parse grader LLM output or internal error."}
# --- Step 3: Implement the CRAG loop ---
async def crag_query_engine(query: str, max_retries: int = 2) -> str:
current_query = query
retrieval_attempt = 0
while retrieval_attempt < max_retries:
logger.info(f"Retrieval attempt {retrieval_attempt + 1} for query: '{current_query}'")
# Initial retrieval
retrieved_nodes: List[NodeWithScore] = await retriever.aretrieve(current_query)
retrieved_context = "
---
".join([n.text for n in retrieved_nodes])
if not retrieved_context:
logger.warning("No context retrieved. Attempting web fallback.")
# Simulate web search fallback
web_result = await simulate_web_search(current_query)
if web_result:
retrieved_context = web_result
logger.info("Web search provided context.")
else:
logger.warning("Web search also failed to provide context.")
return "I cannot answer this question due to insufficient information."
# Grade the context
grade_result = await grade_context(query, retrieved_context)
grade = grade_result.get("grade", "bad")
reasoning = grade_result.get("reasoning", "No reasoning provided.")
logger.info(f"Context Grade: {grade}, Reasoning: {reasoning}")
if grade in ["perfect", "good"]:
logger.info("Context deemed sufficient. Proceeding to generation.")
break # Exit loop if context is good
else:
logger.warning(f"Context insufficient (grade: {grade}). Attempting re-retrieval.")
# Simulate query rewriting for re-retrieval
current_query = await rewrite_query_for_retrieval(query, retrieved_context, reasoning)
retrieval_attempt += 1
if grade not in ["perfect", "good"]:
logger.warning("Max retries reached. Generating with potentially insufficient context or indicating failure.")
# Fallback if context is still not good after retries
if not retrieved_context:
return "I cannot answer this question due to persistent lack of relevant information."
# Final generation with the best available context
response_synthesizer = CompactAndRefine(llm=llm)
final_response = await response_synthesizer.asynthesize(query, nodes=retrieved_nodes)
return str(final_response)
# --- Helper functions for simulation ---
async def rewrite_query_for_retrieval(original_query: str, bad_context: str, reasoning: str) -> str:
# In a real system, this would be an LLM call to refine the query
logger.info(f"Rewriting query based on reasoning: {reasoning}")
if "more detail" in reasoning.lower() or "insufficient" in reasoning.lower():
return f"Tell me more about {original_query}"
elif "irrelevant" in reasoning.lower():
return f"Find specific facts about {original_query}"
return original_query # Default to original if no specific rewrite logic
async def simulate_web_search(query: str) -> Optional[str]:
# Placeholder for actual web search API call
logger.info(f"Simulating web search for: {query}")
if "Eiffel Tower" in query or "Paris" in query:
return "The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower."
return None
# --- Main execution ---
import asyncio
async def main():
queries = [
"What is the capital of France?",
"Tell me about the highest mountain on Earth.",
"What is the primary function of the human heart?", # Query not in KB, will trigger web fallback simulation
"What is the currency of Japan?"
]
for q in queries:
print(f"
--- Query: {q} ---")
response = await crag_query_engine(q)
print(f"Response: {response}")
if __name__ == "__main__":
asyncio.run(main())
--- Query: What is the capital of France? --- Retrieval attempt 1 for query: 'What is the capital of France?' Context Grade: perfect, Reasoning: The context directly states the capital of France. Context deemed sufficient. Proceeding to generation. Response: Paris is the capital of France and is known for its Eiffel Tower. --- Query: Tell me about the highest mountain on Earth. --- Retrieval attempt 1 for query: 'Tell me about the highest mountain on Earth.' Context Grade: perfect, Reasoning: The context directly states that Mount Everest is the Earth's highest mountain above sea level. Context deemed sufficient. Proceeding to generation. Response: Mount Everest is the Earth's highest mountain above sea level. --- Query: What is the primary function of the human heart? --- Retrieval attempt 1 for query: 'What is the primary function of the human heart?' Context Grade: irrelevant, Reasoning: The context does not contain any information about the human heart or its function. Context insufficient (grade: irrelevant). Attempting re-retrieval. Rewriting query based on reasoning: The context does not contain any information about the human heart or its function. Retrieval attempt 2 for query: 'Find specific facts about What is the primary function of the human heart?' No context retrieved. Attempting web fallback. Simulating web search for: Find specific facts about What is the primary function of the human heart? Web search also failed to provide context. Max retries reached. Generating with potentially insufficient context or indicating failure. Response: I cannot answer this question due to persistent lack of relevant information. --- Query: What is the currency of Japan? --- Retrieval attempt 1 for query: 'What is the currency of Japan?' Context Grade: perfect, Reasoning: The context directly states the currency of Japan. Context deemed sufficient. Proceeding to generation. Response: The currency of Japan is the Japanese Yen.