LLM Observability with OpenTelemetry for Agents: Production Guide

In production agentic systems, traditional application performance monitoring (APM) fails to capture the non-deterministic, multi-turn nature of LLM execution. Implementing robust llm observability opentelemetry standards is critical because agents do not operate in simple request-response cycles; they execute complex loops of reasoning, tool execution, state transitions, and recursive LLM calls. Without standard instrumentation, debugging a failed agent run or identifying why a specific tool call latency spiked becomes an exercise in parsing unstructured logs.

OpenTelemetry (OTel) was invented to provide a vendor-neutral, open-source standard for collecting telemetry data. By extending OTel to generative AI through the emerging gen_ai semantic conventions, engineering teams can capture structured traces, metrics, and logs without being locked into proprietary observability platforms. If you ignore standardized OTel instrumentation in production, your monitoring architecture will fragment. You will end up with custom, brittle logging wrappers that break whenever you switch LLM providers or update agent frameworks.

Using OpenTelemetry for agent observability is the preferred approach over vendor-specific SDKs because it decouples telemetry generation from telemetry ingestion. Whether your backend is Datadog, Honeycomb, Dynatrace, Jaeger, or an internal Prometheus/Grafana stack, the application code remains identical. This decoupling is vital for enterprise compliance, cost attribution, and long-term architectural flexibility, especially when managing multi-agent systems where execution spans cross multiple microservices and organizational boundaries.

To build a production-grade observability pipeline for AI agents, you must understand how OpenTelemetry concepts map to agentic behaviors.

• TracerProvider and Tracer: The TracerProvider is the stateful factory that manages the configuration of your telemetry pipeline, including exporters and processors. A Tracer is obtained from the provider and is used to start and end spans. In an agent application, a single global TracerProvider is initialized at startup, and individual modules use localized Tracers to instrument agent loops.
• Spans and Parent-Child Relationships: A span represents a single unit of work. In agentic systems, spans are nested hierarchically. The root span represents the overall user request or agent task. Child spans represent individual reasoning steps, LLM generation calls, vector database queries, or tool executions.
• **Semantic Conventions (gen_ai.*)**: These are standardized attribute keys defined by the OpenTelemetry community to ensure consistent naming across different LLM providers. They prevent a fragmentation where one system logs model_name and another logs request.model.
• Context Propagation: The mechanism by which trace identifiers (Trace ID, Span ID) are passed across thread boundaries, asynchronous event loops, or network boundaries (e.g., HTTP headers). This ensures that a multi-agent system executing across distinct microservices maintains a single, continuous trace.
• Span Processors and Exporters: A SpanProcessor (such as BatchSpanProcessor) intercepts spans when they end and batches them to avoid blocking the application's execution path. The Exporter (such as OTLPSpanExporter) transmits the batched spans over gRPC or HTTP to an OTel Collector or APM backend.

The lifecycle of an instrumented agent run follows a structured path from initialization to telemetry ingestion. Understanding this internal flow is critical for debugging missing traces or high latency overhead.

Phase 1: Pipeline Initialization

At application startup, the OpenTelemetry SDK is configured. A TracerProvider is instantiated and bound to a BatchSpanProcessor. This processor is configured with an OTLPSpanExporter pointing to an OTel Collector or APM endpoint. The global tracer is registered, allowing any module in the codebase to retrieve it without re-passing configuration objects.

Phase 2: Root Span Creation and Context Binding

When a user or system triggers an agent run, the orchestrator starts a root span. This span is marked with attributes identifying the agent's identity, the session ID, and the high-level task. The SDK binds this span to the current execution context (using thread-local storage in synchronous Python or context variables in asyncio). Any subsequent spans created within this execution context automatically inherit the root span's Trace ID and establish a parent-child relationship.

Phase 3: LLM Call Instrumentation

When the agent decides to query an LLM, the wrapper or SDK interceptor starts a child span. Before the network request is sent, the client-side attributes are populated (e.g., gen_ai.request.model, gen_ai.request.temperature). The request is sent to the LLM provider. Once the response is received, the span is updated with server-side attributes (e.g., gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens). If the LLM call fails due to rate limits or API errors, the span status is set to ERROR, the exception is recorded as a span event, and the span is closed.

Phase 4: Tool Execution and Error Capture

If the LLM response contains a tool call, the agent's execution engine intercepts this and starts a tool-specific child span. The tool's name and arguments are recorded. If the tool executes successfully, the output is captured (subject to redaction policies) and the span ends. If the tool raises an exception, the exception is caught, recorded on the span, the span status is set to ERROR, and the exception is re-raised or handled by the agent's self-correction logic.

Phase 5: Asynchronous Exporting

When a span ends, it is not immediately sent over the network. Instead, the BatchSpanProcessor places it into an in-memory queue. A background worker thread periodically flushes this queue, serializing the spans into the OpenTelemetry Protocol (OTLP) format (typically Protobuf over gRPC or HTTP/JSON) and sending them to the OTel Collector. This ensures that telemetry collection does not add blocking latency to the agent's execution path.

The instrumentation architecture consists of four distinct layers: the Agent Application, the local OpenTelemetry SDK, the OpenTelemetry Collector, and the Observability Backend.

Execution starts within the Agent Application, where the agent orchestrator, LLM clients, and custom tools reside. The application is wrapped by the OpenTelemetry SDK. When the orchestrator initiates a run, the SDK generates a Trace ID and starts the root span. As execution flows through the agent's state machine, the SDK manages the context propagation, ensuring that asynchronous tasks or multi-threaded tool executions carry the correct parent span context.

The SDK's BatchSpanProcessor collects ended spans in an in-memory ring buffer. Periodically, or when the buffer reaches a configured threshold, the processor flushes the spans to the OTLPSpanExporter. The exporter serializes the spans and transmits them via gRPC or HTTP to the OpenTelemetry Collector.

The OpenTelemetry Collector runs as a sidecar or a centralized gateway service. It consists of a receiver (which accepts the OTLP data), processors (which perform batching, memory limiting, attribute filtering, and PII redaction), and exporters (which translate the OTel data into vendor-specific formats). The Collector then pushes the processed telemetry to the final Observability Backend (such as Jaeger for local debugging, or Datadog, Honeycomb, or Dynatrace for production monitoring), where engineers can query, visualize, and alert on the trace data.

The Anatomy of gen_ai Semantic Conventions

To achieve true vendor-neutral observability, your instrumentation must strictly adhere to the OpenTelemetry semantic conventions for generative AI. These conventions define a standardized set of attributes that must be attached to spans representing LLM interactions. Using these standardized keys allows APM platforms to automatically parse your telemetry and generate out-of-the-box dashboards for token usage, cost, and latency.

The core attributes are divided into request and response categories:

• gen_ai.system: Identifies the provider or system being queried (e.g., openai, anthropic, cohere, azure_openai, bedrock).
• gen_ai.request.model: The specific model requested by the client (e.g., gpt-4o, claude-3-5-sonnet).
• gen_ai.response.model: The actual model that generated the response, which may differ from the requested model due to routing or aliasing.
• gen_ai.request.temperature: The temperature setting used for the generation.
• gen_ai.usage.input_tokens: The number of prompt tokens processed.
• gen_ai.usage.output_tokens: The number of completion tokens generated.

When instrumenting an agent, these attributes must be set programmatically on the span before it is closed. For example, when using the OpenAI Python SDK, you extract these values from the Completion response object and write them to the active span using span.set_attribute().

Context Propagation Across Distributed Agent Nodes

In multi-agent systems, workflows are rarely confined to a single process. An orchestrator agent might call a specialized sub-agent running on a separate microservice, or delegate a task to an external tool server via the Model Context Protocol (MCP). To maintain a single, unbroken trace across these boundaries, you must implement context propagation.

OpenTelemetry uses the W3C Trace Context specification as its default propagation format. This specification defines two HTTP headers:

1. traceparent: A structured string containing the version, Trace ID, Parent Span ID, and trace flags (e.g., 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01).
2. tracestate: A set of key-value pairs used to carry vendor-specific filtering and routing information.

When an agent makes an outbound HTTP request to another service, the client-side instrumentation injects the current context into the request headers. On the receiving end, the server-side instrumentation extracts this context and uses it as the parent for its own root span. Here is the conceptual flow of context injection and extraction:

# Client-side Injection
from opentelemetry.propagate import inject
headers = {}
inject(headers)
# headers now contains {'traceparent': '...'}
response = requests.post("http://sub-agent-service/run", headers=headers)

# Server-side Extraction
from opentelemetry.propagate import extract
context = extract(request.headers)
with tracer.start_as_current_span("sub_agent_run", context=context):
    # This span is now correctly linked to the client's trace
    execute_sub_agent_logic()

Instrumenting Non-Deterministic Agent Loops (ReAct, State Machines)

Agent architectures like ReAct (Reasoning and Acting) or state-machine-based pipelines present a unique challenge for tracing. Because the execution path is non-deterministic and can loop multiple times before reaching a terminal state, a flat trace structure becomes unreadable.

To solve this, you must structure your spans to reflect the logical iterations of the loop. The overall agent execution should be represented by a long-lived parent span. Each iteration of the loop should be wrapped in a distinct child span (e.g., agent_loop_iteration). Within each iteration, you create nested child spans for the reasoning phase (LLM call) and the action phase (tool execution).

This hierarchical structuring allows you to easily isolate which specific iteration of a loop caused a failure or introduced unexpected latency. If an agent gets stuck in an infinite loop, the trace will clearly show a repeating pattern of identical child spans, making it simple to identify the root cause of the loop budget exhaustion.

Handling Streaming Responses in OpenTelemetry

Streaming responses (using Server-Sent Events) are standard in production agent systems to minimize perceived latency (Time to First Token). However, streaming complicates telemetry collection because the final token counts and completion details are not available when the initial response starts.

To instrument streaming calls correctly, you must keep the LLM span open while the stream is being consumed. As chunks are read from the stream, you accumulate the generated text. Once the stream is fully consumed, you calculate or extract the final token counts, set the corresponding gen_ai.usage.* attributes, and then end the span.

If the stream is aborted prematurely by the client or interrupted by a network error, you must record this partial execution. Set the span status to ERROR, record the exception, and write the accumulated token count (if available) before closing the span. This prevents "orphaned" spans that remain open indefinitely, which can lead to memory leaks in the SDK.

Redacting PII and Sensitive Data in Spans

By default, capturing the raw prompts and completions in your traces is highly valuable for debugging. However, in enterprise environments, prompts often contain personally identifiable information (PII), proprietary source code, or financial data. Exporting this data to third-party APM vendors can violate compliance frameworks like GDPR, HIPAA, or SOC2.

To mitigate this risk, you should implement a custom SpanProcessor or configure the OpenTelemetry Collector to redact sensitive data before it leaves your network boundary.

Within the application, you can write a custom SpanProcessor that intercepts spans before they are passed to the exporter. This processor inspects the span's attributes (such as gen_ai.prompt or custom tool input attributes) and applies regular expressions or calls a PII detection service to mask sensitive values.

Alternatively, the OpenTelemetry Collector's redaction processor can be configured at the gateway level. This is the preferred architectural pattern as it centralizes compliance rules and ensures that even if application developers accidentally log sensitive data, it is stripped before egressing the corporate network.