AI Agent Token Economics: Cost Per Run
Source: mortalapps.com- Token economics in agentic systems defines the compounding, non-linear cost structure of multi-turn reasoning loops.
- It solves the problem of unpredictable API billing spikes caused by context accumulation and runaway recursive tool execution.
- Understanding these economics is critical for establishing predictable unit economics and preventing financial denial-of-service in production.
- This guide provides a mathematical framework to model, calculate, and control agent execution costs before scaling deployments.
Why This Matters
When transitioning from simple single-turn LLM applications to autonomous agentic loops, understanding and predicting your ai agent token cost per run becomes a fundamental engineering requirement. Unlike traditional software where API costs scale linearly with request volume, agentic systems exhibit compounding, non-linear cost curves. This behavior stems from the iterative nature of agent architectures like ReAct or Plan-and-Execute, where each step appends the previous history, tool outputs, and reasoning traces back into the context window. If ignored in production, a single runaway agent loop triggered by an edge-case tool failure can consume millions of tokens in minutes, leading to catastrophic billing spikes and service denial.
This approach to token economics was invented to move teams away from reactive cost monitoring toward proactive, deterministic cost modeling. Engineers must be able to calculate the unit economics of a task - such as processing an invoice or resolving a support ticket - before deploying to production. Without this mathematical framework, enterprises risk deploying agents whose operational cost exceeds the human labor cost they were designed to offset. Use this framework when designing multi-turn agents that utilize external tools, maintain long-running state, or employ reflection. For simple, deterministic tasks, a linear pipeline or state machine with fixed LLM calls is a more cost-effective alternative to autonomous agentic loops.
Core Concepts
To accurately model the cost of an agent run, you must understand the key variables that drive token consumption in multi-turn architectures:
- Context Accumulation (Quadratic Growth): In a standard ReAct loop, the input context for step $k$ contains the system prompt, tool definitions, and the entire history of steps $1$ to $k-1$. This causes input token usage to grow quadratically with the number of iterations.
- Tool Definition Overhead: The static token cost of injecting JSON schemas and instructions for available tools into the system prompt. This cost is paid on every single turn, regardless of whether the tools are called.
- Reasoning-to-Action Ratio: The proportion of tokens spent by the model on internal reasoning (e.g., thinking blocks, chain-of-thought) versus the tokens spent generating the final tool call or user response.
- Prompt Cache Hit Rate: The percentage of input tokens that match cached prefixes at the LLM provider level, which typically receive a significant pricing discount.
- State Serialization Cost: The token overhead of saving, restoring, and formatting agent state (e.g., database records, memory buffers) across execution boundaries.
| Architecture Pattern | Cost Growth Rate | Primary Cost Driver | Optimization Target |
|---|---|---|---|
| Single-Turn RAG | Linear | Retrieval Context Size | Vector Chunk Size & Relevance |
| ReAct Loop | Quadratic | Iteration Count & Tool Schemas | Prompt Caching & Loop Budgets |
| Plan-and-Execute | Step-wise Linear | Sub-task Decomposition | Plan Re-use & Parallelization |
How It Works
The lifecycle of an agent run determines how tokens are consumed and billed. Understanding this sequence allows engineers to inject cost-control guards at critical execution boundaries.
Phase 1: Initialization and Tool Registration
When an agent run is initiated, the system compiles the base prompt. This includes the system instructions, user input, and the serialized JSON schemas of all registered tools. This base prompt forms the initial input token payload ($I_0$).
Phase 2: The Execution Loop and Context Accumulation
For each iteration $k$ in the agent loop:
- Inference Call: The compiled context is sent to the LLM. The input token count is $I_k$.
- Generation: The LLM generates a reasoning trace and a tool call. This consumes output tokens ($O_k$).
- Tool Execution: The agent runtime intercepts the tool call, executes the underlying code, and captures the output. The tool output is serialized into text, consuming $T_k$ tokens.
- Context Append: The reasoning trace, tool call, and tool output are appended to the history. The input context for the next step becomes $I_{k+1} = I_k + O_k + T_k$.
Phase 3: Termination and Cost Calculation
The loop terminates when the LLM generates a final answer instead of a tool call, or when a safety budget is exceeded. The total token cost is calculated by multiplying the accumulated input and output tokens by the provider's specific rate per million tokens, factoring in prompt caching discounts for the static system prompt and tool definitions.
Failure Paths and Cost Anomalies
If a tool returns an error, the agent may attempt to self-correct by calling the same tool with modified parameters. If the error persists and no retry limit is enforced, the agent enters an infinite loop. Because the context window accumulates every failed attempt, the cost per step increases linearly (as context grows), making total cumulative cost quadratic until the model's context window is exhausted or the API budget is completely drained.
Architecture
The token tracking and cost control architecture sits between the Agent Executor and the upstream LLM Provider. It acts as a deterministic gateway that monitors, validates, and halts execution based on real-time financial metrics.
Execution begins when the Client sends a task to the Agent Executor. The Agent Executor compiles the initial prompt and routes all LLM requests through an LLM Gateway (or Proxy). The LLM Gateway is responsible for intercepting outgoing payloads and incoming responses. Before forwarding a request to the LLM Provider, the Gateway queries the Budget Controller to verify that the current run has not exceeded its maximum token or dollar budget.
If the budget is valid, the request is sent to the LLM Provider. The provider returns the completion along with usage metadata in the response headers. The LLM Gateway parses these headers, calculates the exact cost based on active pricing tiers (including prompt caching discounts), and writes the transaction to the Token Tracker Database. If the Budget Controller detects that a request will push the run over its limit, it intercepts the call, halts execution, and returns a budget exhaustion error to the Agent Executor, which triggers a graceful fallback routine to return the best available state to the Client.
The Compounding Cost Formula of ReAct Loops
To build a predictive cost model, we must formalize the token consumption of a standard Reasoning and Acting (ReAct) loop. Let $S$ be the size of the static system prompt and tool definitions in tokens. Let $U$ be the size of the user's initial query. Let $O_i$ be the output tokens generated by the LLM at step $i$, and let $T_i$ be the tokens returned by the tool executed at step $i$.
At iteration $k$, the input tokens $I_k$ sent to the LLM are defined by:
$$I_k = S + U + \sum_{i=1}^{k-1} (O_i + T_i)$$
The total cost ($C$) of a run consisting of $N$ iterations is the sum of the input and output costs across all steps:
$$C = \sum_{k=1}^{N} \left( I_k \times R_{in} + O_k \times R_{out} \right)$$
Where $R_{in}$ is the rate per input token and $R_{out}$ is the rate per output token. Substituting $I_k$ into the cost equation reveals the quadratic relationship: each additional step scales the cost of all subsequent steps because the history is re-sent. If tool outputs ($T_i$) are large (e.g., raw database payloads or scraped HTML), the cost escalates rapidly.
Tool Schema Overhead and Minimization
Tool definitions are not free. When using native tool-calling APIs, providers format your JSON schemas into system instructions. A single complex tool schema can easily consume 200 to 500 tokens. If your agent has access to 20 tools, you pay a tax of 4,000 to 10,000 input tokens *on every single turn*.
To mitigate this, implement Dynamic Tool Selection. Instead of registering all tools at the start of the run, use a lightweight semantic router (or a cheaper, fast model) to select a subset of relevant tools (e.g., top 3) based on the current step's objective. This reduces $S$ dynamically, saving thousands of tokens per iteration.
Optimizing Prompt Caching Economics
Modern LLM providers offer prompt caching, which reduces input token costs by up to 50% for prefix matches. To maximize cache hits, structure your context window with static elements at the absolute beginning:
- Static System Instructions: Must be identical across all runs.
- Static Tool Definitions: Keep the schemas and descriptions unchanged.
- User Query: The initial request.
- Dynamic History (Episodic Memory): Append new turns *only* at the end of the prompt.
If you inject dynamic variables (like the current timestamp or user-specific metadata) at the top of your system prompt, you invalidate the cache for the entire prompt, destroying the economic benefits of prompt caching.
Context Eviction and Summarization Tradeoffs
When an agent run approaches the model's context limit or a pre-configured token budget, you must choose between two eviction strategies:
- Sliding Window Eviction: Dropping the oldest turns in the history. This keeps input costs stable but risks losing critical context or causing the agent to repeat previously failed steps.
- Summarization Loops: Invoking a cheaper model to summarize the history of steps $1$ to $k-2$, replacing those turns with a single summary block. This incurs a one-time output token cost for the summary but permanently reduces the input token footprint for all future steps.
Code Example
import os
import logging
import tiktoken
from typing import List, Dict, Any
# Configure logging for production observability
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("TokenEconomicsAgent")
class BudgetExceededError(Exception):
"""Raised when the agent run exceeds the pre-configured financial budget."""
pass
class TokenTracker:
def __init__(self, model_name: str, input_rate_per_m: float, output_rate_per_m: float, max_budget_usd: float):
self.model_name = model_name
self.input_rate = input_rate_per_m / 1_000_000.0
self.output_rate = output_rate_per_m / 1_000_000.0
self.max_budget = max_budget_usd
self.total_input_tokens = 0
self.total_output_tokens = 0
self.total_cost_usd = 0.0
try:
self.encoder = tiktoken.encoding_for_model(model_name)
except KeyError:
logger.warning(f"Model {model_name} not found in tiktoken. Falling back to o200k_base.")
self.encoder = tiktoken.get_encoding("o200k_base")
def count_tokens(self, text: str) -> int:
return len(self.encoder.encode(text))
def track_turn(self, input_text: str, output_text: str):
input_tokens = self.count_tokens(input_text)
output_tokens = self.count_tokens(output_text)
turn_cost = (input_tokens * self.input_rate) + (output_tokens * self.output_rate)
self.total_input_tokens += input_tokens
self.total_output_tokens += output_tokens
self.total_cost_usd += turn_cost
logger.info(f"Turn Cost: ${turn_cost:.6f} | Total Cost: ${self.total_cost_usd:.6f}")
if self.total_cost_usd > self.max_budget:
raise BudgetExceededError(
f"Budget of ${self.max_budget:.4f} exceeded. Current cost: ${self.total_cost_usd:.4f}"
)
# Simulated Agent Execution Loop
def run_agent_task(user_query: str, max_budget: float) -> Dict[str, Any]:
# Fetch pricing from environment or configuration management
input_rate = float(os.environ.get("LLM_INPUT_RATE_PER_M", "2.50"))
output_rate = float(os.environ.get("LLM_OUTPUT_RATE_PER_M", "10.00"))
model = os.environ.get("LLM_MODEL_NAME", "gpt-4o")
tracker = TokenTracker(model, input_rate, output_rate, max_budget)
system_prompt = "You are an agent with access to tools. Solve the user query step-by-step."
history = []
# Simulated multi-turn execution
try:
for step in range(1, 5):
# Compile current context
current_context = f"{system_prompt}
" + "
".join(history) + f"
User: {user_query}"
# Simulate LLM generating reasoning and tool call
simulated_output = f"Step {step}: I need to query the database. Action: query_db(id={step})"
# Track tokens and cost before executing the next step
tracker.track_turn(current_context, simulated_output)
# Append to history
history.append(f"Assistant: {simulated_output}")
history.append(f"Tool Output: Record {step} retrieved successfully.")
return {
"status": "success",
"total_cost": tracker.total_cost_usd,
"input_tokens": tracker.total_input_tokens,
"output_tokens": tracker.total_output_tokens
}
except BudgetExceededError as e:
logger.error(f"Execution halted: {str(e)}")
return {
"status": "budget_exhausted",
"total_cost": tracker.total_cost_usd,
"error": str(e)
}
if __name__ == "__main__":
# Run with a generous budget
result = run_agent_task("Analyze quarterly financial reports.", max_budget=0.05)
print(f"Generous Budget Result: {result}")
# Run with an intentionally low budget to trigger enforcement
fail_result = run_agent_task("Analyze quarterly financial reports.", max_budget=0.001)
print(f"Low Budget Result: {fail_result}")INFO:TokenEconomicsAgent:Turn Cost: $0.000315 | Total Cost: $0.000315
INFO:TokenEconomicsAgent:Turn Cost: $0.000455 | Total Cost: $0.000770
INFO:TokenEconomicsAgent:Turn Cost: $0.000595 | Total Cost: $0.001365
INFO:TokenEconomicsAgent:Turn Cost: $0.000735 | Total Cost: $0.002100
Generous Budget Result: {'status': 'success', 'total_cost': 0.0021, 'input_tokens': 640, 'output_tokens': 50}
INFO:TokenEconomicsAgent:Turn Cost: $0.000315 | Total Cost: $0.000315
INFO:TokenEconomicsAgent:Turn Cost: $0.000455 | Total Cost: $0.000770
INFO:TokenEconomicsAgent:Turn Cost: $0.000595 | Total Cost: $0.001365
ERROR:TokenEconomicsAgent:Execution halted: Budget of $0.0010 exceeded. Current cost: $0.0014
Low Budget Result: {'status': 'budget_exhausted', 'total_cost': 0.001365, 'error': 'Budget of $0.0010 exceeded. Current cost: $0.0014'}