> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Built-in Tracing Providers
> Auto-trace LLM calls and tool invocations using Agentex's built-in provider integrations for OpenAI Agents SDK, LiteLLM, and LangGraph.
Agentex provides three built-in tracing providers that automatically create spans for LLM calls and, in some cases, tool invocations. Each provider integrates with a different framework and has different tracing coverage.
Built-in providers give you a baseline — auto-traced LLM calls and, in some cases, tool invocations — but truly useful traces require manual instrumentation by the agent implementer: turn-level spans, custom reasoning steps, structured input/output, and proper parent-child linking. Think of built-in providers as the foundation, not the finished product. See [Span Hierarchy & Best Practices](/docs/v5/tracing/integrations/agentex/span-hierarchy) for how to build on top of them.
## Comparison
| Provider | LLM Tracing | Tool Tracing | Framework |
| ---------------------------------- | :---------: | :----------: | --------------------- |
| `TemporalTracingModelProvider` | Auto | No | OpenAI Agents SDK |
| `adk.providers.litellm` | Auto | No | LiteLLM |
| `create_langgraph_tracing_handler` | Auto | Auto | LangGraph / LangChain |
***
## Providers
### How It Works
For [Temporal-style agents](/docs/v5/agents/agentex/types/agentic-agent) using the OpenAI Agents SDK, LLM calls are auto-traced via `TemporalTracingModelProvider`. For non-Agentex usage of the OpenAI Agents SDK with the SGP tracing SDK directly, see the [OpenAI Agents Integration](/docs/v5/tracing/integrations/openai-agents) page.
The tracing flow is:
1. Your workflow sets `_task_id`, `_trace_id`, and `_parent_span_id` on the workflow instance
2. `ContextWorkflowOutboundInterceptor` propagates these as Temporal headers to activities
3. `ContextActivityInboundInterceptor` reads the headers and sets `ContextVar`s
4. `TemporalTracingResponsesModel` wraps the base model and creates a span for each `get_response()` call
### Auto-Traced Span Structure
Each LLM call produces a single span named `model_get_response`:
```python theme={null}
# Span input:
{
"model": "gpt-4o",
"has_system_instructions": True,
"input_type": "list",
"tools_count": 3,
"handoffs_count": 0,
"has_output_schema": False,
"model_settings": {
"temperature": 0.7,
"max_tokens": 4096,
"reasoning": {"effort": "high"}
}
}
# Span output:
{
"new_items": [
{"type": "message", "content": [{"type": "text", "text": "..."}]},
{"type": "function_call", "name": "search", "arguments": "..."}
],
"final_output": "The agent's text response..."
}
```
The span name is always `model_get_response`. The model name and settings appear in the span's input data.
### Streaming Hooks
`TemporalStreamingHooks` streams tool lifecycle events to the UI in real time:
```python theme={null}
from agentex.lib.core.temporal.plugins.openai_agents import TemporalStreamingHooks
hooks = TemporalStreamingHooks(task_id=params.task.id)
result = await Runner.run(agent, input_list, hooks=hooks)
```
This streams:
* `ToolRequestContent` when a tool starts (name, arguments)
* `ToolResponseContent` when a tool finishes (name, result)
* `TextContent` on agent handoffs
### Tracing Gap: Tool Calls
The OpenAI Agents SDK plugin does **not** create separate spans for tool executions. Tool calls appear inside the model span's `output.new_items` array as `function_call` entries, but they are not their own spans in the trace tree.
If you need tool-level spans, create manual spans around your tool logic. See [Manual Spans](/docs/v5/tracing/integrations/agentex/span-hierarchy#manual-spans-for-custom-logic).
### Available Methods
The `adk.providers.litellm` module exposes four methods, all of which auto-create tracing spans:
| Method | Streaming | Auto-send to UI |
| ---------------------------------- | :-------: | :-------------: |
| `chat_completion` | No | No |
| `chat_completion_stream` | Yes | No |
| `chat_completion_auto_send` | No | Yes |
| `chat_completion_stream_auto_send` | Yes | Yes |
The "auto-send" variants automatically create a `TaskMessage` that appears in the Agentex chat UI.
### Code Example
```python theme={null}
from agentex.lib import adk
response = await adk.providers.litellm.chat_completion_stream_auto_send(
task_id=params.task.id,
llm_config=LLMConfig(
model="gpt-4o-mini",
messages=state.messages,
temperature=0.7,
),
trace_id=params.task.id,
parent_span_id=turn_span.id if turn_span else None,
)
```
### Auto-Traced Span Structure
Each method creates a single span named after the method (e.g., `chat_completion_stream_auto_send`):
* **Input**: The full `LLMConfig`. model name, messages, temperature, and other settings
* **Output**: The complete `Completion` object or `TaskMessage`
* For streaming methods, chunks are aggregated into the final response before being set as span output
### Tracing Gap: Streaming Chunks
Individual streaming chunks are not traced as separate spans. The LLM span captures the final aggregated response. If you need chunk-level visibility, you'll need custom instrumentation.
### How It Works
The LangGraph handler creates tracing spans for both LLM calls and tool calls by hooking into LangChain's callback system:
```python theme={null}
from agentex.lib.adk._modules._langgraph_tracing import create_langgraph_tracing_handler
handler = create_langgraph_tracing_handler(
trace_id=task_id,
parent_span_id=turn_span.id,
)
result = await graph.ainvoke(
input_state,
config={"callbacks": [handler]}
)
```
### Span Naming
| Call Type | Span Name | `__span_type__` |
| --------- | ------------------ | --------------- |
| LLM call | `llm:` | `COMPLETION` |
| Tool call | `tool:` | `CUSTOM` |
### What Gets Traced
* **LLM calls**: `on_chat_model_start()` → `on_llm_end()` captures messages, model settings, output content, and tool calls
* **Tool calls**: `on_tool_start()` → `on_tool_end()` captures tool name, input, and output
This is the only built-in provider that creates separate spans for both LLM calls and tool calls automatically.
### Tracing Gap: Tool I/O Detail
The LangGraph handler captures tool name and string-serialized input/output only. Structured tool arguments and rich return types are coerced to strings. If you need full structured tool I/O, supplement with manual spans.
***
## Choosing a Provider
Use `TemporalTracingModelProvider` for automatic LLM call tracing. This is the default when you use the OpenAI Agents SDK plugin with Temporal workflows. No additional setup is needed for model call spans.
For tool-level tracing, create manual spans around your tool functions using `adk.tracing.span()`. See [Manual Spans](/docs/v5/tracing/integrations/agentex/span-hierarchy#manual-spans-for-custom-logic) for examples.
Use the `adk.providers.litellm` methods. All four methods automatically create tracing spans when you pass `trace_id` and `parent_span_id`.
The `auto_send` variants are convenient for chat-style agents since they handle both tracing and UI message creation in one call.
Use `create_langgraph_tracing_handler` registered as a callback. This gives you the most complete auto-tracing. Both LLM and tool calls get their own spans without any manual instrumentation.
You still need to create turn-level parent spans manually to group operations by conversation turn.