OpenTelemetry GenAI Semantic Conventions for Agent Spans
Source: mortalapps.com- OpenTelemetry GenAI semantic conventions define a standardized telemetry schema for tracking LLM, Tool, Memory, and Orchestration spans in agentic workflows.
- It solves the problem of fragmented, vendor-proprietary tracing by providing a unified, open-standard specification for agent execution paths.
- In production, implementing these conventions ensures cross-platform observability, precise cost attribution, and deterministic latency tracking across distributed agent systems.
- Engineers can trace complex nested loops where an orchestration span coordinates memory retrieval, tool execution, and LLM generation within a single, parent-propagated trace context.
Why This Matters
Implementing the opentelemetry genai semantic conventions is critical for moving agentic systems from experimental scripts to observable, enterprise-grade production software. Historically, tracing LLM applications was plagued by proprietary SDKs and vendor lock-in, which made it impossible to correlate model invocations with the broader microservices ecosystem. When an agent enters an infinite loop, hallucinates tool arguments, or experiences a sudden spike in latency, engineers need a standardized way to diagnose the failure.
Without standardized span types, debugging distributed multi-agent systems becomes an exercise in parsing unstructured logs. The OpenTelemetry GenAI semantic conventions solve this by defining explicit schemas for the four pillars of agent execution: Orchestration, LLM, Tool, and Memory. By structuring telemetry around these four distinct span types, teams can isolate whether a bottleneck is caused by a slow vector database lookup (Memory), a poorly performing model (LLM), a slow external API (Tool), or inefficient state-machine transitions (Orchestration).
This approach is essential for high-throughput production environments where tracing overhead must be minimized, and telemetry data must be ingested by diverse backends like Jaeger, Honeycomb, Datadog, or Dynatrace. Ignoring this standard leads to fragmented observability pipelines, incomplete trace context propagation across agent boundaries, and an inability to calculate accurate cost-per-run metrics. Using this open standard ensures that your observability architecture remains future-proof, vendor-agnostic, and fully integrated with standard W3C trace propagation protocols.
Core Concepts
To build highly observable agentic systems, engineers must master the fundamental telemetry abstractions defined by the OpenTelemetry GenAI specifications:
- GenAI Span: A specialized OpenTelemetry span containing standardized attributes defined by the GenAI semantic conventions, categorizing the work done by generative AI components.
- Orchestration Span: The parent span representing the agent's control loop, state machine, or planning phase (e.g.,
invoke_agent,chat). - LLM Span: A span representing a direct call to a Large Language Model for completion or chat generation, capturing prompt/response tokens and model metadata.
- Tool Span: A span representing the execution of an external function, API, or service invoked by the agent (e.g.,
execute_tool). - Memory Span: A span representing operations on agent state or context, such as vector database queries, episodic memory retrieval, or state persistence.
- Trace Context Propagation: The mechanism of passing tracing metadata (trace ID, span ID) across network and process boundaries using W3C Trace Context headers.
How It Works
The Lifecycle of an Agentic Trace
- Initialization: The client triggers an agent run. The orchestration engine starts a parent span categorized as an orchestration span (e.g.,
gen_ai.systemset to the agent framework name, andgen_ai.operation.nameset toagent_run).
- Context Retrieval (Memory): The agent retrieves conversation history or semantic context. It starts a child Memory span. If the vector database call fails, the span records the exception, sets its status to
ERROR, and closes, allowing the parent orchestration span to decide on a fallback.
- Planning & LLM Generation: The orchestration engine packages the prompt and calls the LLM. It starts a child LLM span. This span records model parameters (temperature, top_p) and, upon completion, records token usage (prompt, completion) and the raw response.
- Tool Execution: The LLM returns a tool-use request. The orchestration engine parses this and initiates a child Tool span (e.g.,
execute_tool). The tool executes. If the tool throws an error, the Tool span captures the stack trace and error message, returning the error context to the LLM in the next turn.
- Resolution: The loop continues until a stop condition is met. The parent orchestration span is closed, calculating the total end-to-end latency, total token cost, and execution path.
Architecture
The architecture of an OpenTelemetry-instrumented agent system consists of five primary components: the Agent Orchestrator, the OpenTelemetry SDK, the Semantic Convention Mapper, the OpenTelemetry Collector, and the Observability Backend.
Execution begins at the Agent Orchestrator when a user request is received. The Orchestrator initializes a root Orchestration Span via the OpenTelemetry SDK. As the orchestrator executes its control loop, it spawns child spans for Memory, LLM, and Tool operations.
The Semantic Convention Mapper is an internal abstraction layer that intercepts these operations and injects the standardized gen_ai.* attributes (such as gen_ai.system, gen_ai.request.model, and gen_ai.usage.input_tokens) into the active span context.
Data flows from the SDK to the local OpenTelemetry Collector over gRPC or HTTP/protobuf using the OpenTelemetry Protocol (OTLP). The Collector buffers, processes, and batches these spans before exporting them to the downstream Observability Backend (e.g., Jaeger, Honeycomb, or Datadog).
When the agent makes outbound calls to external tools or other agents, the SDK injects W3C Trace Context headers (traceparent, tracestate) into the outbound request payload, ensuring that downstream systems can continue the trace as child spans, preserving the complete execution hierarchy.
1. The Four Core
Agent Span Types
The OpenTelemetry GenAI semantic conventions classify agent activities into four distinct span types. Each has specific semantic requirements and attribute namespaces:
| Span Type | Semantic Namespace | Primary Purpose | Key Attributes |
|---|---|---|---|
| Orchestration | gen_ai.orchestration |
Tracks control loops, planning, and agent state transitions. | gen_ai.system, gen_ai.operation.name, gen_ai.agent.name |
| LLM | gen_ai.client / gen_ai.server |
Tracks direct model inference calls (chat, completion, embeddings). | gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |
| Tool | gen_ai.tool |
Tracks execution of external tools, APIs, or sandboxed code. | gen_ai.tool.name, gen_ai.tool.parameters, gen_ai.tool.status |
| Memory | gen_ai.memory |
Tracks read/write operations on short-term, episodic, or long-term memory. | gen_ai.memory.type, gen_ai.memory.operation, gen_ai.memory.key |
2. Attribute Schema Reference
To comply with the OpenTelemetry GenAI semantic conventions, spans must populate specific attributes depending on their type. Below is the precise schema definition for production implementations:
LLM Span Attributes
gen_ai.system: (string) The name of the model provider or framework (e.g.,openai,anthropic,bedrock).gen_ai.request.model: (string) The target model name requested (e.g.,gpt-4o,claude-3-5-sonnet).gen_ai.response.model: (string) The actual model name returned by the provider.gen_ai.request.temperature: (double) The sampling temperature.gen_ai.usage.input_tokens: (int) Number of input tokens processed.gen_ai.usage.output_tokens: (int) Number of output tokens generated.
Tool Span Attributes
gen_ai.tool.name: (string) The fully qualified name of the tool being executed.gen_ai.tool.type: (string) The category of tool (e.g.,web_search,database_query,code_interpreter).gen_ai.tool.parameters: (string) JSON-serialized string of input arguments passed to the tool.
Orchestration Span Attributes
gen_ai.agent.name: (string) The identifier of the agent executing the task.gen_ai.orchestration.strategy: (string) The pattern used (e.g.,react,plan_and_execute,hierarchical).
3. Trace Hierarchy and Nesting Rules
In a complex agent run, spans must be nested deterministically to preserve parent-child relationships. The root span must always be an Orchestration span representing the overall task execution (e.g., invoke_agent).
Within this root span, the orchestrator may execute multiple loops. Each loop iteration can spawn child spans:
invoke_agent(Root Orchestration Span)retrieve_history(Child Memory Span)generate_plan(Child LLM Span)execute_tool(Child Tool Span)database_query(Child DB Span, standard OTel DB convention)evaluate_result(Child LLM Span)
This nesting ensures that latency and cost can be rolled up to the parent span, allowing engineers to calculate the exact cost and latency contribution of each individual tool or memory lookup to the overall agent run.
Code Example
import os
import json
import logging
from opentelemetry import trace
from opentelemetry.trace import StatusCode
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("agent_telemetry")
# Initialize OpenTelemetry SDK
provider = TracerProvider()
processor = BatchSpanProcessor(ConsoleSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("mortalapps.agent.telemetry")
def execute_agent_workflow(user_query: str):
# 1. Orchestration Span (Root)
with tracer.start_as_current_span(
"invoke_agent",
attributes={
"gen_ai.system": "custom_agent_framework",
"gen_ai.operation.name": "agent_run",
"gen_ai.agent.name": "customer_support_agent",
"gen_ai.orchestration.strategy": "react"
}
) as root_span:
try:
# 2. Memory Span (Child)
with tracer.start_as_current_span(
"retrieve_history",
attributes={
"gen_ai.memory.type": "episodic",
"gen_ai.memory.operation": "read",
"gen_ai.memory.key": "user_session_123"
}
):
# Simulate memory lookup
logger.info("Retrieving conversation history...")
history = [{"role": "user", "content": "Hello"}]
# 3. LLM Span (Child)
with tracer.start_as_current_span(
"generate_plan",
attributes={
"gen_ai.system": "openai",
"gen_ai.request.model": "gpt-4o",
"gen_ai.request.temperature": 0.7
}
) as llm_span:
# Simulate LLM API Call
logger.info("Calling LLM for planning...")
llm_span.set_attribute("gen_ai.response.model", "gpt-4o-2024-05-13")
llm_span.set_attribute("gen_ai.usage.input_tokens", 150)
llm_span.set_attribute("gen_ai.usage.output_tokens", 45)
tool_to_call = "search_database"
tool_args = {"query": user_query}
# 4. Tool Span (Child)
with tracer.start_as_current_span(
"execute_tool",
attributes={
"gen_ai.tool.name": tool_to_call,
"gen_ai.tool.type": "database_query",
"gen_ai.tool.parameters": json.dumps(tool_args)
}
) as tool_span:
try:
logger.info(f"Executing tool: {tool_to_call}")
# Simulate tool execution
if not os.environ.get("DB_CONNECTION_STRING"):
raise ConnectionError("Database connection string not configured.")
tool_result = "Found 3 matching records."
tool_span.set_status(StatusCode.OK)
except Exception as tool_err:
tool_span.record_exception(tool_err)
tool_span.set_status(StatusCode.ERROR, str(tool_err))
raise tool_err
root_span.set_status(StatusCode.OK)
return "Workflow completed successfully."
except Exception as err:
root_span.record_exception(err)
root_span.set_status(StatusCode.ERROR, str(err))
logger.error(f"Agent workflow failed: {err}")
return "Workflow failed."
if __name__ == "__main__":
# Set dummy env var for successful run demonstration
os.environ["DB_CONNECTION_STRING"] = "postgresql://user:pass@localhost:5432/db"
execute_agent_workflow("Find recent invoices")INFO:agent_telemetry:Retrieving conversation history...
INFO:agent_telemetry:Calling LLM for planning...
INFO:agent_telemetry:Executing tool: search_database
{Spans exported to console showing nested hierarchy with correct gen_ai attributes}