Multi-Turn Agent Evaluation Ragas: Thread-Level Scoring Guide
Source: mortalapps.com- Multi-turn evaluation assesses conversational agents across entire interaction sessions rather than isolated single-turn inputs.
- It solves the problem of compounding errors, semantic drift, and state degradation in long-lived agent sessions.
- Using Ragas for multi-turn evaluation provides standardized metrics for context retention, coherence, and goal trajectory.
- Implementing this framework enables automated CI/CD guardrails that block regressions in agent reasoning and conversational stability.
Why This Matters
Evaluating conversational agents has evolved past single-turn question-answering paradigms, making a robust multi turn agent evaluation ragas framework essential for production systems. In real-world deployments, agents execute multi-step plans, maintain complex state machines, and interact over extended sessions. Traditional metrics like ROUGE, BLEU, or even single-turn LLM-as-a-judge evaluations fail to capture semantic drift, goal trajectory, and context retention across long conversation threads. When agents interact over multiple turns, errors compound: a minor misunderstanding in turn two can lead to a complete failure of the agent's objective by turn ten. This approach was invented to model the conversation as a unified stateful graph rather than isolated inputs and outputs. Without session-level evaluation, teams risk deploying agents that appear highly accurate on individual turns but fail to solve the user's ultimate goal, leading to silent failures, customer churn, and high token waste. Use this multi-turn evaluation approach when your agent uses stateful memory, maintains a multi-step task execution loop (such as ReAct or Plan-and-Execute), or handles complex customer support workflows. Avoid it for simple, stateless single-turn tasks like summarization or direct translation, where standard single-turn LLM-as-a-judge or Ragas rag-triad metrics are computationally cheaper and sufficient. By shifting to session-level scoring, engineering teams can establish automated CI/CD guardrails that guarantee conversational coherence and goal completion before production deployment.
Core Concepts
To effectively evaluate multi-turn agent systems, you must understand the core concepts that define stateful, long-context interactions.
- Multi-Turn Thread: A sequence of alternating user and agent messages representing a single continuous session. Unlike single-turn interactions, each message depends on the historical context of the preceding turns.
- Semantic Intent: The underlying objective or goal of the user at any given point in the conversation, which may evolve as the agent provides new information.
- Goal Trajectory: The path the agent takes across turns to resolve the user's initial and evolving requests. Trajectory evaluation measures whether the agent is moving closer to or further from the goal.
- Coherence Score: A metric measuring how logically and contextually connected each turn is to the preceding history, ensuring the agent does not contradict itself or lose the thread.
- Topic Drift: The gradual, unintended shift of conversational context away from the primary objective, often caused by poor context management or hallucination.
- Ragas Multi-Turn Schema: The structured dataset format required by Ragas to evaluate conversations, mapping user inputs, agent responses, retrieved contexts, and reference ground truths across a timeline.
How It Works
Multi-turn evaluation with Ragas processes complete conversation histories to assess session-level performance. The execution pipeline follows a structured path from state extraction to metric aggregation.
Phase 1: Thread Extraction and Normalization
The evaluation pipeline begins by querying the agent's state store (such as Redis or PostgreSQL) to extract raw conversation threads. These threads are normalized into a structured format where each turn is explicitly labeled with the speaker role (user or assistant), the raw message content, the retrieved context chunks used during that turn, and any intermediate tool calls.
Phase 2: Context Alignment and Windowing
Because evaluating an entire 20-turn conversation in a single LLM call can exceed context windows or dilute attention, the pipeline applies context windowing. Ragas analyzes the thread using sliding windows of turns (e.g., turns 1-3, 2-4, 3-5) to evaluate local coherence, while simultaneously analyzing the first and last turns to evaluate global goal trajectory. This ensures that both micro-level transitions and macro-level outcomes are scored.
Phase 3: Metric Computation
The normalized dataset is passed to the Ragas evaluation engine, which orchestrates calls to an evaluation LLM. The engine computes specific multi-turn metrics:
- Coherency: Measures if the agent's response logically follows the user's prompt given the history.
- Topic Adherence: Verifies that the agent remains focused on the user's core query without introducing irrelevant topics.
- Goal Attainment: Compares the final state of the conversation against a reference ground truth to determine if the user's objective was met.
Failure Path Handling
If an evaluation call fails due to rate limits, context window exhaustion, or malformed JSON responses from the evaluation LLM, the pipeline catches the exception, logs the state of the failed thread, applies an exponential backoff retry strategy, and falls back to a secondary LLM provider if the primary remains unavailable. Threads that cannot be evaluated are flagged in the final report to prevent skewing the aggregate scores.
Architecture
The multi-turn evaluation architecture consists of five decoupled components that manage state extraction, data transformation, metric execution, and reporting. Execution starts at the Thread Store and ends at the CI/CD Reporter.
First, the Thread Store (such as Redis or PostgreSQL) holds the raw, stateful conversation histories of active and completed agent sessions. The Thread Extractor queries this store, pulling target sessions based on metadata filters (e.g., sessions with specific user ratings or tool-use flags).
Second, the extracted threads flow into the Schema Transformer. This component maps the raw database schemas into the standardized Ragas Multi-Turn Schema, aligning user inputs, assistant responses, and retrieved contexts into sequential turn objects.
Third, the Ragas Evaluation Engine receives the normalized dataset. It coordinates the execution of turn-by-turn and session-level metrics by sending structured prompts to the LLM Judge Provider (e.g., OpenAI, Anthropic, or a local Ollama instance).
Fourth, the raw metric outputs from the LLM Judge are processed by the Metrics Aggregator, which calculates statistical summaries (mean, median, and variance) for the entire evaluation batch.
Finally, the aggregated scores are sent to the CI/CD Reporter or Observability Dashboard. If any metric falls below a predefined threshold, the CI/CD Reporter triggers an alert or fails the build pipeline, preventing regressive agent code from reaching production.
Modeling Conversation Threads for Ragas
Evaluating multi-turn interactions requires mapping conversational state into a format that Ragas can parse. In single-turn RAG evaluation, a dataset consists of flat arrays of question, answer, contexts, and ground_truth. In multi-turn evaluation, we must represent the temporal sequence of the interaction. Ragas models this using a specialized schema where each sample is a list of turns, and each turn contains the interaction state at that specific timestamp.
To prepare data for Ragas, you must construct a list of conversation dictionaries. Each dictionary represents a full session and contains a list of turns. Each turn must contain:
user_input: The prompt sent by the user in this turn.response: The agent's output for this turn.retrieved_contexts: The list of document chunks retrieved to generate the response.reference: The ground truth response or expected state (optional, used for supervised metrics).
Session-Level vs. Turn-Level Metrics
When evaluating multi-turn agents, you must distinguish between turn-level metrics and session-level metrics. Turn-level metrics evaluate the immediate transition from turn $N$ to turn $N+1$. Session-level metrics evaluate the global trajectory of the conversation from turn $1$ to turn $M$.
| Metric Type | Metric Name | Focus | Evaluation Method |
|---|---|---|---|
| Turn-Level | Coherency | Measures if the response is logical given the immediate history. | LLM-as-a-judge analyzes the transition between adjacent turns. |
| Turn-Level | Context Recall | Measures if the retrieved context in turn $N$ matches the user's intent. | Compares retrieved context against reference ground truth for that turn. |
| Session-Level | Goal Attainment | Measures if the user's primary objective was successfully resolved. | LLM-as-a-judge compares the final turn state against the initial user prompt. |
| Session-Level | Topic Adherence | Detects if the agent drifted into irrelevant topics across the session. | Analyzes the semantic similarity of topics across all turns in the thread. |
Implementing Coherence and Trajectory Evaluation
Coherence evaluation ensures that the agent maintains a consistent persona, does not contradict prior statements, and references historical information correctly. To calculate coherence, Ragas prompts the evaluation LLM with the entire history up to turn $N$, followed by the agent's response at turn $N$. The evaluator is asked to identify any logical contradictions or context omissions.
Trajectory evaluation, on the other hand, measures the efficiency of the agent's path to resolution. An agent that takes 15 turns to solve a problem that should take 3 turns has a poor trajectory score, even if the final goal is attained. The trajectory score penalizes repetitive loops, unnecessary tool executions, and redundant clarifying questions.
Handling Context Window Constraints in Evaluation LLMs
As conversation threads grow, passing the entire history, retrieved contexts, and evaluation prompts to the judge LLM can easily exceed context limits or lead to the "lost in the middle" phenomenon. To mitigate this, implement a sliding window evaluation strategy.
Instead of evaluating a 20-turn thread in one call, segment the thread into overlapping windows of 3 turns (e.g., turns 1-3, 2-4, 3-5). Run the turn-level evaluations on these smaller windows. For session-level metrics like Goal Attainment, compress the intermediate turns by generating a running summary of the conversation state, and pass only the initial prompt, the state summary, and the final response to the evaluator LLM. This reduces token consumption and improves evaluator accuracy.
Customizing Ragas Prompts for Domain-Specific Intent
Default Ragas prompts are designed for general-purpose assistant evaluations. For production agents operating in specialized domains (e.g., healthcare, finance, or database administration), you must customize the evaluation prompts to reflect domain-specific constraints.
For example, in a financial advising agent, a change in user intent (e.g., shifting from asking about stock prices to asking about tax implications) must be handled with strict compliance guardrails. You can inject custom criteria into the Ragas evaluation config, instructing the judge LLM to penalize any turn where the agent provides tax advice without outputting the required legal disclaimer. This is achieved by subclassing the standard Ragas metrics and overriding their internal prompt templates with domain-specific evaluation rubrics.
Code Example
import os
import logging
from typing import List, Dict, Any
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
answer_relevancy,
faithfulness,
context_recall,
context_precision
)
# Configure logging for production visibility
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("ragas_multi_turn_eval")
def verify_environment():
"""Ensure required API keys are present in the environment."""
if not os.environ.get("OPENAI_API_KEY"):
logger.error("Missing OPENAI_API_KEY environment variable.")
raise EnvironmentError("OPENAI_API_KEY must be set to run Ragas evaluations.")
def load_raw_thread_from_db(session_id: str) -> List[Dict[str, Any]]:
"""
Simulates retrieving a multi-turn conversation thread from a state store.
In production, this would query Redis or PostgreSQL.
"""
logger.info(f"Fetching thread data for session: {session_id}")
# Mocked multi-turn thread data
return [
{
"turn": 1,
"user_input": "How do I configure a LangGraph checkpointer with PostgreSQL?",
"response": "You can use the PostgresSaver class from langgraph.checkpoint.postgres.",
"retrieved_contexts": [
"Use PostgresSaver to persist state in LangGraph using a PostgreSQL database connection pool."
],
"reference": "To configure a LangGraph checkpointer with PostgreSQL, use the PostgresSaver class."
},
{
"turn": 2,
"user_input": "Can you show me a code example of that?",
"response": "Here is how you initialize it: with PostgresSaver.from_conn_string(conn_str) as saver:",
"retrieved_contexts": [
"Example: with PostgresSaver.from_conn_string(DB_URI) as memory: app = workflow.compile(checkpointer=memory)"
],
"reference": "Initialize it using PostgresSaver.from_conn_string(conn_string) and pass it to compile()."
}
]
def format_thread_for_ragas(thread_data: List[Dict[str, Any]]) -> Dataset:
"""
Transforms raw multi-turn thread data into a Hugging Face Dataset compatible with Ragas.
"""
logger.info("Transforming thread data into Ragas evaluation schema.")
# Flattening multi-turn context for standard Ragas metrics
# For advanced multi-turn, we evaluate turns sequentially or aggregate them
questions = []
answers = []
contexts = []
ground_truths = []
for turn in thread_data:
questions.append(turn["user_input"])
answers.append(turn["response"])
contexts.append(turn["retrieved_contexts"])
ground_truths.append(turn["reference"])
dataset_dict = {
"question": questions,
"answer": answers,
"contexts": contexts,
"ground_truth": ground_truths
}
return Dataset.from_dict(dataset_dict)
def run_evaluation(dataset: Dataset) -> Dict[str, float]:
"""
Executes the Ragas evaluation pipeline over the formatted dataset.
"""
logger.info("Starting Ragas evaluation pipeline.")
try:
# Define the metrics to evaluate
metrics = [
faithfulness,
answer_relevancy,
context_recall,
context_precision
]
# Run evaluation
result = evaluate(
dataset=dataset,
metrics=metrics
)
logger.info("Evaluation completed successfully.")
return result
except Exception as e:
logger.error(f"Evaluation pipeline failed: {str(e)}", exc_info=True)
raise
if __name__ == "__main__":
try:
verify_environment()
raw_thread = load_raw_thread_from_db("session_98765")
ragas_dataset = format_thread_for_ragas(raw_thread)
scores = run_evaluation(ragas_dataset)
print("
=== Evaluation Results ===")
for metric, score in scores.items():
print(f"{metric}: {score:.4f}")
except Exception as err:
print(f"Execution failed: {err}")
INFO:ragas_multi_turn_eval:Fetching thread data for session: session_98765 INFO:ragas_multi_turn_eval:Transforming thread data into Ragas evaluation schema. INFO:ragas_multi_turn_eval:Starting Ragas evaluation pipeline. INFO:ragas_multi_turn_eval:Evaluation completed successfully. === Evaluation Results === faithfulness: 1.0000 answer_relevancy: 0.9450 context_recall: 1.0000 context_precision: 1.0000