Braintrust: CI/CD Agent Evaluation Setup
Source: mortalapps.com- Braintrust-based automated agent evaluation establishes a programmatic loop to test agentic workflows against golden datasets during CI/CD.
- It solves the problem of silent regressions, drift, and non-deterministic failures in agent tool-calling and multi-turn reasoning.
- Integrating this into GitHub Actions or GitLab CI ensures that no agent code is deployed without passing strict performance, cost, and correctness thresholds.
- This tutorial delivers a complete, runnable pipeline that runs evaluations, logs experiments to Braintrust, and fails the build on regression alerts.
Why This Matters
Implementing a robust braintrust agent evaluation setup is the difference between deploying deterministic software and shipping unpredictable black boxes. In traditional software engineering, unit tests assert exact outputs for known inputs. AI agents, however, rely on non-deterministic LLMs, dynamic tool execution, and multi-turn planning loops where a minor prompt adjustment or model update can silently break downstream tasks. Without automated evaluation, teams are forced to rely on manual, ad-hoc testing, which fails to scale and misses edge cases.
This approach was invented because manual inspection cannot keep pace with continuous deployment. If you ignore automated agent evaluations in your CI/CD pipeline, you risk deploying agents that hallucinate, enter infinite tool-use loops, or suffer from severe cost and latency regressions. Braintrust provides a high-performance, developer-focused platform to manage evaluation datasets, run parallel test cases, score outputs using LLM-as-a-judge or heuristic metrics, and track performance over time.
You should use a dedicated evaluation platform like Braintrust when your agent moves beyond simple single-turn prompts into complex multi-turn workflows, custom tool-calling, or RAG pipelines. While lightweight local scripts work for initial prototyping, they lack the centralized experiment tracking, dataset versioning, and regression analysis required for enterprise-grade production deployments. Integrating Braintrust directly into your CI/CD pipeline ensures that every pull request is programmatically validated against a versioned golden dataset before merging.
Core Concepts
To build a production-grade evaluation pipeline, you must understand the core abstractions used by Braintrust:
- Golden Dataset: A curated, version-controlled collection of inputs, expected outputs, and metadata representing the core use cases and edge cases your agent must handle.
- Scorers: Functions that evaluate agent outputs. They can be heuristic (e.g., exact match, JSON validation, regex) or model-based (LLM-as-a-Judge assessing semantic alignment, tone, or safety).
- Experiment: A single run of an evaluation suite against a specific version of your agent code and prompt configuration.
- Regression Threshold: The minimum acceptable score or maximum allowable drop in performance (e.g., accuracy, latency, cost) compared to a baseline experiment (usually the production branch) before a CI/CD build is failed.
- Braintrust Eval: The execution framework that coordinates pulling the dataset, running the agent against each record, executing scorers, and logging results to the Braintrust cloud or self-hosted instance.
How It Works
The automated evaluation pipeline operates as a closed loop integrated directly into your software development lifecycle. The process follows five distinct phases:
Step 1: Dataset Synchronization
The evaluation pipeline begins by pulling the designated golden dataset from Braintrust. This dataset is versioned, ensuring that evaluations run on a consistent set of test cases. Any updates to the dataset in the Braintrust UI are automatically captured via version tags.
Step 2: Agent Execution Loop
The CI/CD runner executes the evaluation script. For each record in the dataset, the runner passes the input to the agent. The agent executes its reasoning loop, calls tools, and produces a final response. This execution is wrapped in Braintrust tracing spans to capture detailed step-by-step telemetry.
Step 3: Scoring and Metric Extraction
Once the agent completes a run, Braintrust executes the assigned scorers. These scorers analyze the agent's output, intermediate tool calls, and execution logs. Metrics such as correctness, tool-use accuracy, latency, and token cost are computed and attached to the run.
Step 4: Experiment Logging and Baseline Comparison
The results of all test cases are aggregated into a new Braintrust Experiment. The pipeline queries Braintrust's API to compare the current experiment's average scores against the baseline experiment (typically the latest run on the main branch).
Step 5: Regression Check and CI/CD Gate
If any critical metric falls below the defined regression threshold, the evaluation script exits with a non-zero code. This action fails the CI/CD pipeline, preventing the regression from being merged or deployed. Detailed reports and links to the Braintrust UI are posted back to the pull request.
Architecture
The CI/CD evaluation architecture consists of five primary components: the Git Repository, the CI/CD Runner (e.g., GitHub Actions), the Agent Application Code, the Braintrust Platform, and the LLM Providers.
Execution starts when a developer pushes code or opens a pull request. The CI/CD Runner triggers the workflow and pulls the agent code. It sets up the environment, injecting API keys for Braintrust and LLM providers via secure environment variables.
The runner executes the evaluation script, which initializes the Braintrust client. The client fetches the target Golden Dataset from the Braintrust Platform. The evaluation script then loops through the dataset records, feeding inputs into the Agent Application.
As the agent runs, it calls LLM Providers and executes local tools. All LLM calls, tool executions, and internal state transitions are wrapped in Braintrust tracing spans, which stream telemetry back to the Braintrust Platform in real-time.
Upon completion of the agent loop, the evaluation script invokes the Scorer functions. These scorers evaluate the outputs and send the scores to Braintrust. Finally, the evaluation script queries the Braintrust API to compare the current experiment's aggregated metrics against the baseline experiment. If a regression is detected, the script exits with an error, halting the CI/CD pipeline.
Dataset Management and Versioning
In a production evaluation setup, your golden dataset must be treated with the same rigor as application code. Braintrust allows you to manage datasets programmatically or via their web interface. For CI/CD, you should pin your evaluations to a specific dataset version or use a release tag (e.g., production-v1). This prevents changes made by non-technical team members in the UI from unexpectedly breaking your CI/CD pipeline.
When structuring your dataset, each record should contain at least an input field (the prompt or user query) and an expected field (the ground truth or target output). You can also include custom metadata, such as the expected tool calls or category tags, to enable granular filtering and analysis in your reports.
Designing Robust Scoring Functions
Scorers are the heart of your evaluation pipeline. A naive exact-match scorer is rarely sufficient for agentic workflows. Instead, you should implement a hybrid scoring strategy:
- Heuristic Scorers: Use these for deterministic checks. For example, if your agent is supposed to output valid JSON, use a scorer that attempts to parse the output. If the agent must call a specific tool, write a scorer that inspects the trace logs to verify the tool was invoked with the correct arguments.
- Model-Based Scorers (LLM-as-a-Judge): Use these for semantic correctness, tone, and safety. To ensure consistency, your LLM-as-a-judge prompts must use structured outputs (JSON schema) and have the temperature set to 0. Always provide a clear rubric with explicit scoring criteria to minimize variance.
CI/CD Integration and Baseline Comparison
To prevent regressions, your CI/CD pipeline must compare the results of the current pull request against a stable baseline. The baseline is typically the latest evaluation run on your main or production branch.
Using the Braintrust SDK, you can programmatically fetch the baseline experiment, extract its summary statistics, and compare them to the current run. If the average score of a critical metric (like correctness) drops by more than a predefined threshold (e.g., 0.05 or 5%), the script should raise an error and fail the build. This programmatic gate ensures that performance degradation is caught before deployment.
Handling Flakiness and Non-Determinism
Because LLMs are inherently non-deterministic, agent evaluations can suffer from flakiness. A test that passes on one run might fail on the next due to minor variations in the model's response. To mitigate this:
- Run Multiple Iterations: For critical test cases, configure your evaluation to run each input multiple times (e.g., 3 to 5 times) and average the scores.
- Statistical Significance: Instead of failing on any minor drop, use statistical tests (like a t-test) to determine if the performance drop is statistically significant.
- Soft Failures: Categorize your test cases. Critical path cases should fail the build, while edge cases or experimental features can trigger warnings without blocking the pipeline.
Code Example
import os
import sys
import logging
from braintrust import evaluate, Score
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("braintrust_eval")
# Ensure required environment variables are set
BRAINTRUST_API_KEY = os.environ.get("BRAINTRUST_API_KEY")
OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY")
if not BRAINTRUST_API_KEY or not OPENAI_API_KEY:
logger.error("Missing required environment variables: BRAINTRUST_API_KEY or OPENAI_API_KEY")
sys.exit(1)
# Define a mock agent representing your production agent workflow
def my_agent(input_text):
# In production, this would call your LangGraph, CrewAI, or custom agent loop
# We wrap the execution in a Braintrust logger span for tracing
logger.info(f"Running agent on input: {input_text}")
# Simple mock logic for demonstration
if "calculate" in input_text.lower():
return "The result is 42."
return "I cannot process this request."
# Define a heuristic scorer
def exact_match_scorer(output, expected):
score = 1.0 if output.strip() == expected.strip() else 0.0
return Score(
name="exact_match",
score=score,
metadata={"output": output, "expected": expected}
)
# Define a semantic scorer (heuristic fallback for demonstration)
def semantic_contain_scorer(output, expected):
# Simple check if key expected terms are in the output
terms = expected.lower().split()
matches = sum(1 for term in terms if term in output.lower())
score = matches / len(terms) if terms else 0.0
return Score(
name="semantic_contain",
score=score,
metadata={"matches": matches, "total_terms": len(terms)}
)
def run_evaluation():
project_name = "Agent-CI-CD-Eval"
dataset_name = "Golden-Dataset-v1"
logger.info("Starting Braintrust evaluation...")
# Run the evaluation using Braintrust's evaluate function
# In a real setup, Braintrust pulls the dataset directly from your account
# For this runnable example, we provide inline data representing the fetched dataset
try:
result = evaluate(
project_name=project_name,
data=[
{"input": "Calculate the meaning of life", "expected": "The result is 42."},
{"input": "Hello agent", "expected": "I cannot process this request."}
],
task=lambda input_data: my_agent(input_data["input"]),
scores=[exact_match_scorer, semantic_contain_scorer],
)
# Extract summary statistics
summary = result.summary
logger.info(f"Evaluation complete. Summary: {summary}")
# Define regression threshold (e.g., minimum acceptable average score of 0.8)
threshold = 0.8
# Check scores
for score_name, score_val in summary.scores.items():
mean_score = score_val.mean
logger.info(f"Metric: {score_name} | Mean Score: {mean_score}")
if mean_score < threshold:
logger.error(f"Regression detected! Metric '{score_name}' score {mean_score} is below threshold {threshold}")
sys.exit(1)
logger.info("All evaluation checks passed successfully.")
sys.exit(0)
except Exception as e:
logger.error(f"Evaluation failed with error: {str(e)}")
sys.exit(1)
if __name__ == "__main__":
run_evaluation()INFO:braintrust_eval:Starting Braintrust evaluation... INFO:braintrust_eval:Running agent on input: Calculate the meaning of life INFO:braintrust_eval:Running agent on input: Hello agent INFO:braintrust_eval:Evaluation complete. Summary: ... INFO:braintrust_eval:Metric: exact_match | Mean Score: 1.0 INFO:braintrust_eval:Metric: semantic_contain | Mean Score: 1.0 INFO:braintrust_eval:All evaluation checks passed successfully.