Tool Use Correctness Evaluation: Scoring and CI Validation
Source: mortalapps.com- Tool-use correctness scoring evaluates whether an agent called the correct tool with the correct argument payload.
- It solves the problem of silent agent failures, where incorrect tool invocations pass schema validation but execute with corrupted parameters.
- In production, this provides a deterministic quality gate to prevent regression when updating system prompts or underlying LLM models.
- Teams can run these evaluations as a CI/CD gate using schema diffing, exact-match validation, and semantic distance scoring.
- The primary tradeoff is the maintenance overhead of keeping a ground-truth dataset in sync with evolving tool schemas.
Why This Matters
Implementing a robust tool use correctness evaluation framework is critical for moving agentic systems from experimental prototypes to reliable production software. In production, agents frequently fail not by throwing explicit exceptions, but by silently invoking the wrong tools or passing corrupted, hallucinated arguments to API endpoints. This approach was invented because traditional software testing paradigms (like unit tests) cannot handle the non-deterministic nature of LLM tool-calling, while generic LLM-as-a-judge evaluations are too slow, expensive, and subjective for high-throughput validation.
If you ignore tool-use correctness scoring, your agentic pipeline will suffer from silent regressions. A minor tweak to a system prompt or a model upgrade can cause an agent to pass a string instead of an ISO timestamp, or call delete_user instead of archive_user, leading to data corruption, security breaches, and broken user experiences.
You should use deterministic tool-use correctness scoring (such as schema diffing and exact-match validation) whenever your tools have strict API contracts. Reserve LLM-as-a-judge alternatives only for highly open-ended arguments where semantic meaning matters more than exact syntax. By embedding these evaluations into your continuous integration (CI) pipelines, engineering teams can confidently deploy agent updates without risking catastrophic runtime failures.
Core Concepts
Fundamental Evaluation Vocabulary
To build a robust tool-use evaluation framework, you must understand the following core concepts:
- Ground Truth Dataset: A curated collection of input prompts mapped to the exact expected tool calls (tool name and argument payload).
- Tool Call Schema: The JSON Schema defining the expected structure, types, and constraints of a tool's arguments.
- Schema Diffing: The process of structurally comparing the generated tool call arguments against the ground-truth arguments, ignoring trivial formatting differences (like whitespace or key ordering).
- Exact Match (EM) Scoring: A binary metric indicating whether the invoked tool name and arguments match the ground truth exactly.
- Partial Match / Fuzzy Scoring: A continuous metric (0.0 to 1.0) evaluating correctness when some arguments are optional, or when string arguments allow semantic variations.
- Type Coercion: The ability of the evaluation engine to normalize types (e.g., converting
"123"to123if the schema expects an integer) before scoring. - Argument Weighting: Assigning different importance values to different tool arguments (e.g., matching a
user_idis critical, while matching areasonstring is low priority).
How It Works
The Evaluation Lifecycle
The tool-use correctness evaluation workflow operates as a deterministic pipeline that runs parallel to or within your CI/CD suite. It processes agent outputs against a predefined ground-truth dataset to yield a structured correctness report.
Step 1: Execution and Capture
The evaluation run begins by feeding a test prompt from the ground-truth dataset into the agent. The evaluation harness intercepts the agent's execution loop before the tool is actually executed in a real environment. This is achieved by mocking the tool execution layer or using a dry-run agent state. The harness captures the raw tool call payload, which includes the targeted tool name and the generated JSON arguments.
Step 2: Normalization and Type Coercion
Raw JSON payloads generated by LLMs often contain minor formatting anomalies, such as stringified numbers, alternative boolean representations (e.g., "yes" instead of true), or extra whitespace. The evaluation engine normalizes both the generated payload and the ground-truth payload. It parses stringified JSON, orders keys alphabetically, and coerces types based on the target tool's JSON Schema (e.g., converting a string representation of a float to an actual float).
Step 3: Structural and Value Comparison
Once normalized, the engine performs a hierarchical diff of the two payloads:
- Tool Name Validation: If the generated tool name does not match the ground-truth tool name, the score is immediately set to
0.0(unless a multi-tool fallback mapping is explicitly defined). - Required Parameter Check: The engine verifies that all required parameters defined in the schema are present. Missing required parameters result in a failure.
- Value Matching: The engine compares the value of each parameter. For categorical or numeric parameters, it applies exact matching. For text parameters, it may apply string distance metrics (like Levenshtein distance) or semantic embeddings to determine similarity.
Step 4: Score Aggregation
The individual parameter scores are aggregated into a final correctness score. This can be a simple binary score (1.0 for perfect match, 0.0 otherwise) or a weighted average. For example, a critical parameter like transaction_amount might carry a weight of 0.8, while an optional memo field carries a weight of 0.2.
Failure Paths and Edge Cases
- No Tool Called: If the agent returns a text response instead of invoking a tool, the engine records a failure with a score of
0.0and logs a "Missing Tool Call" error. - Multiple Tool Calls: If the agent invokes multiple tools when only one was expected, the engine evaluates the first call or applies a penalty depending on the test configuration.
- Schema Violation: If the generated payload fails basic JSON Schema validation, the evaluation fails immediately before value comparison, preventing runtime errors in the evaluation script itself.
Architecture
The architecture of a tool-use correctness evaluation system consists of five primary components: the Test Runner, the Agent Sandbox, the Mock Tool Registry, the Evaluation Engine, and the Metrics Store.
Execution begins at the Test Runner, which reads test cases from a Ground Truth Database. Each test case contains an input prompt, the expected tool name, and the expected argument payload. The Test Runner passes the input prompt to the Agent Sandbox, which hosts the agent instance under test.
To prevent side effects, the Agent Sandbox is configured with a Mock Tool Registry. Instead of executing actual database writes or API calls, the Mock Tool Registry intercepts the agent's tool-call requests, captures the tool name and argument payload, and returns a pre-configured mock response to allow the agent to complete its execution loop.
Once the agent emits its tool-call payload, the Agent Sandbox forwards this generated payload along with the corresponding ground-truth payload to the Evaluation Engine. The Evaluation Engine contains sub-modules for Schema Validation, Type Coercion, and Diff Scoring. It applies the scoring algorithms to calculate the correctness metrics.
Finally, the Evaluation Engine writes the structured evaluation results (including pass/fail status, parameter-level diffs, and latency) to the Metrics Store. The Test Runner reads these results to determine the overall exit code (0 for success, 1 for failure), which is consumed by the CI/CD pipeline orchestrator to gate deployments.
Ground Truth Dataset Design
Designing a high-quality ground-truth dataset is the foundation of reliable tool-use evaluation. A naive dataset simply maps prompts to static JSON payloads. However, production environments require dynamic validation. Your dataset schema must support:
- Parameter Variability: Defining fields that can accept a range of valid values (e.g., relative dates like "tomorrow" which resolve to different absolute dates depending on the execution timestamp).
- Permissible Alternatives: Specifying alternative tool paths when multiple tools can satisfy the user's intent.
- Strict vs. Loose Matching: Flagging specific parameters that require exact matches (e.g., database IDs) versus those that allow fuzzy matching (e.g., search queries).
Here is a standard schema for a ground-truth test case:
{
"test_id": "TC-042",
"prompt": "Transfer $150 to John Doe's savings account with the memo 'rent'",
"expected_tool": "execute_transfer",
"arguments": {
"amount": 150.00,
"recipient_name": "John Doe",
"account_type": "savings",
"memo": "rent"
},
"rules": {
"amount": { "matching_strategy": "exact" },
"recipient_name": { "matching_strategy": "fuzzy", "threshold": 0.85 },
"account_type": { "matching_strategy": "exact" },
"memo": { "matching_strategy": "ignore_case" }
}
}Schema Diffing Algorithms and Normalization
Direct string comparison of JSON payloads is highly fragile. Minor changes in whitespace, key ordering, or numeric formatting will cause false negatives. A robust evaluation engine must parse and normalize the JSON structures before executing a diff.
The normalization pipeline involves:
- Key Sorting: Recursively sorting all keys in the JSON object to ensure deterministic ordering.
- Type Casting: Utilizing the tool's JSON Schema to cast types. If the schema defines a parameter as a boolean, strings like
"true","yes", or1should be coerced toTrue. - Date/Time Standardization: Parsing datetime strings into ISO 8601 UTC format before comparison. If the agent generates
2026-03-30T15:00:00Zand the ground truth is2026-03-30 15:00:00+00:00, they must resolve to the same timestamp object.
Scoring Methodologies: Exact, Fuzzy, and Semantic
Depending on the nature of the tool arguments, different scoring algorithms must be applied:
| Strategy | Use Case | Algorithm | Pros/Cons |
|---|---|---|---|
| Exact Match | IDs, Enums, Booleans, Numbers | Direct equality (==) |
Fast, deterministic; highly fragile for natural language. |
| Fuzzy Match | Names, Addresses, Short Text | Levenshtein Distance, Jaro-Winkler | Handles typos and minor variations; requires threshold tuning. |
| Semantic Match | Long descriptions, search queries | Vector Embeddings (Cosine Similarity) | Handles synonyms and paraphrasing; requires an embedding model API call. |
For semantic matching, the cosine similarity between the embedding of the generated argument $A_{gen}$ and the ground-truth argument $A_{gt}$ is calculated:
$$\text{Similarity} = \frac{A_{gen} \cdot A_{gt}}{\|A_{gen}\| \|A_{gt}\|}$$
A threshold (typically between 0.82 and 0.90) is set to determine pass/fail.
Running Correctness Checks as a CI Gate
To prevent regressions, tool-use evaluations must be integrated into your CI/CD pipeline (e.g., GitHub Actions, GitLab CI). Running these evaluations on every pull request ensures that prompt modifications or model updates do not degrade tool-calling performance.
A typical CI workflow execution pattern:
- Provision Sandbox: Spin up a lightweight, isolated environment containing the agent code.
- Inject Mock Services: Replace real external APIs with local mock servers or mock Python classes to ensure tests are deterministic, fast, and free of cost.
- Execute Evaluation Suite: Run the evaluation script against the ground-truth dataset.
- Enforce Quality Gates: Define a minimum acceptable correctness score (e.g., 95% pass rate, 100% exact match on critical tools). If the run falls below this threshold, the CI job exits with a non-zero code, blocking the merge.
Code Example
import os
import logging
import json
from typing import Dict, Any, Tuple, Optional
from pydantic import BaseModel, Field, ValidationError
# Configure logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("tool_evaluator")
# Example Tool Schema using Pydantic
class TransferFundsSchema(BaseModel):
recipient_id: str = Field(..., min_length=5)
amount: float = Field(..., gt=0.0)
currency: str = Field(default="USD")
memo: Optional[str] = None
class EvaluationResult(BaseModel):
is_correct: bool
score: float
mismatches: Dict[str, Any]
error_message: Optional[str] = None
class ToolUseEvaluator:
"""
Evaluates the correctness of tool calls generated by an agent
against a ground-truth dataset.
"""
def __init__(self, schema_registry: Dict[str, Any]):
self.schema_registry = schema_registry
def normalize_payload(self, payload: Dict[str, Any], schema_class: Any) -> Dict[str, Any]:
"""
Normalizes and validates the payload against the Pydantic schema
to ensure type coercion and default values are applied.
"""
try:
validated = schema_class(**payload)
# Convert back to dict, sorting keys and standardizing types
return json.loads(validated.model_dump_json())
except ValidationError as e:
logger.warning(f"Payload normalization failed: {e}")
raise
def calculate_score(
self,
generated_tool: str,
generated_args: Dict[str, Any],
expected_tool: str,
expected_args: Dict[str, Any],
rules: Optional[Dict[str, Any]] = None
) -> EvaluationResult:
"""
Compares the generated tool call against the ground truth.
"""
if generated_tool != expected_tool:
return EvaluationResult(
is_correct=False,
score=0.0,
mismatches={"tool_name": {"expected": expected_tool, "actual": generated_tool}},
error_message="Tool name mismatch"
)
schema_class = self.schema_registry.get(expected_tool)
if not schema_class:
return EvaluationResult(
is_correct=False,
score=0.0,
mismatches={},
error_message=f"Schema not found for tool: {expected_tool}"
)
try:
norm_gen = self.normalize_payload(generated_args, schema_class)
norm_exp = self.normalize_payload(expected_args, schema_class)
except ValidationError as e:
return EvaluationResult(
is_correct=False,
score=0.0,
mismatches={"schema_validation": e.errors()},
error_message="Generated arguments failed schema validation"
)
mismatches = {}
total_fields = max(len(norm_exp), 1)
matched_fields = 0
for key, expected_val in norm_exp.items():
gen_val = norm_gen.get(key)
rule = rules.get(key, {}) if rules else {}
strategy = rule.get("matching_strategy", "exact")
if gen_val is None:
mismatches[key] = {"expected": expected_val, "actual": None, "reason": "Missing field"}
continue
if strategy == "exact":
if gen_val == expected_val:
matched_fields += 1
else:
mismatches[key] = {"expected": expected_val, "actual": gen_val, "reason": "Value mismatch"}
elif strategy == "ignore_case" and isinstance(expected_val, str) and isinstance(gen_val, str):
if gen_val.lower() == expected_val.lower():
matched_fields += 1
else:
mismatches[key] = {"expected": expected_val, "actual": gen_val, "reason": "Case-insensitive mismatch"}
else:
# Default fallback to exact
if gen_val == expected_val:
matched_fields += 1
else:
mismatches[key] = {"expected": expected_val, "actual": gen_val, "reason": "Value mismatch"}
score = matched_fields / total_fields
is_correct = score == 1.0
return EvaluationResult(
is_correct=is_correct,
score=score,
mismatches=mismatches
)
# Execution Example
if __name__ == "__main__":
# Ensure environment is configured
api_key = os.environ.get("AGENT_EVAL_API_KEY", "mock_key_for_testing")
logger.info(f"Initializing evaluator with API key: {api_key[:4]}***")
registry = {"transfer_funds": TransferFundsSchema}
evaluator = ToolUseEvaluator(schema_registry=registry)
# Ground Truth
expected_tool = "transfer_funds"
expected_args = {"recipient_id": "USR-99281", "amount": 250.00, "memo": "Rent Payment"}
rules = {"memo": {"matching_strategy": "ignore_case"}}
# Simulated Agent Output (with minor casing difference in memo and stringified amount)
generated_tool = "transfer_funds"
generated_args = {"recipient_id": "USR-99281", "amount": "250.00", "memo": "rent payment"}
result = evaluator.calculate_score(
generated_tool=generated_tool,
generated_args=generated_args,
expected_tool=expected_tool,
expected_args=expected_args,
rules=rules
)
print(result.model_dump_json(indent=2)){
"is_correct": true,
"score": 1.0,
"mismatches": {},
"error_message": null
}