Agentex Tracing Overview

What is Agentex Tracing?

Modern AI agents execute complex, multi-step workflows: reasoning, calling tools, making LLM requests, and coordinating with other agents. Without tracing, these workflows are opaque. When something goes wrong or runs slowly, you’re left guessing. Agentex Tracing gives you deep visibility into your Temporal-style agents by capturing structured spans for every meaningful operation. You get:

One trace per task: trace_id maps directly to your Agentex task_id
A structured span hierarchy: turns, LLM calls, tool invocations, and custom logic all nest via parent_id
Tool and model I/O capture: see exactly what went into and came out of each operation
Profiler-ready data: consistent span naming enables aggregated performance analysis across all your traces

This guide covers tracing specific to Agentex agents. For general tracing concepts (spans, traces, the Traces UI), see Introduction to Tracing.

Architecture

Key Identity Relationships

Concept	ID Field	Typical Value
Trace	`trace_id`	= `task_id` (the Agentex task UUID)
Span	`id`	UUID4, auto-generated
Parent	`parent_id`	`span.id` of the enclosing span
Agent	`data.__agent_name__`	Agent name from manifest
Source	`data.__source__`	`"agentex"` (set by processor)

Core Concepts

trace_id = task_id

Every Agentex task has a unique ID. This same ID becomes the trace_id for all spans associated with that task. This 1:1 mapping means you can look up any task’s full execution history by searching for its task_id in the Traces UI.

Span Hierarchy

Spans form a tree via parent_id relationships. A turn-level span is the parent of the LLM calls and tool invocations that happen during that turn. This hierarchy is what the Traces UI renders as an expandable tree and Gantt chart.

Span Data Model

class Span(BaseModel):
    id: str                       # UUID4 - unique span identifier
    name: str                     # Descriptive operation name
    trace_id: str                 # Groups spans into a trace (= task_id)
    start_time: datetime          # UTC timestamp when span began
    end_time: Optional[datetime]  # UTC timestamp when span ended
    parent_id: Optional[str]      # Parent span ID for hierarchy
    input: Optional[Dict]         # Operation input data
    output: Optional[Dict]        # Operation output data
    data: Optional[Dict]          # Metadata (__span_type__, __source__, etc.)

Quick Start

Initialize Tracing in Your acp.py

Add the tracing initialization at the top level of your acp.py (or workflow.py for Temporal agents). This runs once when the agent process starts and configures both the Agentex and SGP tracing processors so spans appear in the SGP Traces UI.

import os
import logging

from agentex.lib.core.tracing.processors.tracing_processor_manager import (
    add_tracing_processor_config,
)
from agentex.lib.types.tracing import SGPTracingProcessorConfig

logger = logging.getLogger(__name__)

# --- Tracing initialization (top-level, runs once at process start) ---

SGP_API_KEY = os.getenv("SGP_API_KEY")
SGP_ACCOUNT_ID = os.getenv("SGP_ACCOUNT_ID")
SGP_BASE_URL = os.getenv("SGP_CLIENT_BASE_URL")

if SGP_API_KEY and SGP_ACCOUNT_ID:
    add_tracing_processor_config(
        SGPTracingProcessorConfig(
            type="sgp",
            sgp_api_key=SGP_API_KEY,         # Your SGP API key
            sgp_account_id=SGP_ACCOUNT_ID,   # Your SGP account ID
            sgp_base_url=SGP_BASE_URL,       # e.g. "https://api.scale.com"
        )
    )

    # Optional: also initialize the SGP tracing SDK directly
    # (needed if you use scale_gp_beta.lib.tracing.create_span() anywhere)
    try:
        import scale_gp_beta.lib.tracing as tracing
        from scale_gp_beta import SGPClient

        client = SGPClient(
            api_key=SGP_API_KEY,
            account_id=SGP_ACCOUNT_ID,
            base_url=SGP_BASE_URL,
        )
        tracing.init(client=client)
        logger.info("SGP tracing initialized")
    except ImportError:
        logger.warning("scale_gp_beta not installed, skipping SGP SDK tracing init")
else:
    logger.warning("SGP tracing disabled (missing SGP_API_KEY or SGP_ACCOUNT_ID)")

The three required environment variables:

Env Var	Description	Example
`SGP_API_KEY`	Your SGP API key for authentication	`scl_...`
`SGP_ACCOUNT_ID`	Your SGP account identifier	`acc_...`
`SGP_CLIENT_BASE_URL`	The SGP API base URL. Must be set, there is no default.	`https://api.scale.com`

SGP_CLIENT_BASE_URL has no default value. If it is not set, the tracing processor will not know where to send spans. Make sure this is configured in your agent’s environment or secrets.

Create a Turn Span

Wrap each conversation turn in a manual span. This groups all LLM calls and tool invocations for that turn.

from agentex.lib import adk

@acp.on_task_event_send
async def handle_event_send(params: SendEventParams):
    state = ...  # get state
    state.turn_number += 1

    async with adk.tracing.span(
        trace_id=params.task.id,
        name=f"turn:{state.turn_number}",
        input=state.model_dump(),
    ) as span:
        # Everything inside this block nests under this turn span
        pass

Make a Traced LLM Call

Pass trace_id and parent_span_id to your LLM call. The built-in providers auto-create a child span.

    async with adk.tracing.span(
        trace_id=params.task.id,
        name=f"turn:{state.turn_number}",
        input=state.model_dump(),
    ) as span:
        response = await adk.providers.litellm.chat_completion_stream_auto_send(
            task_id=params.task.id,
            llm_config=LLMConfig(model="gpt-4o-mini", messages=state.messages),
            trace_id=params.task.id,
            parent_span_id=span.id if span else None,
        )

        if span:
            span.output = state.model_dump()

View in the Traces UI

Open the SGP Traces page, filter by your agent name, and click on the trace. You should see your turn span with the LLM call nested beneath it.

Trace detail view showing a turn span with a child LLM span

For a complete reference on the SGP Tracing SDK, see Creating Traces and Spans.

The Traces table preview column extracts text from specific preferred keys in your span input and output (e.g., title, query, content). If the preview shows an unhelpful value, add a title key with the value you want displayed.

The Two Tracing Systems

Agentex Native Tracing (`adk.tracing`)

This is the primary system for Temporal agents. Spans are sent to the Agentex backend API.

adk.tracing.span() creates a span with a UUID4 id
On start: calls POST /spans to create the span in the Agentex database
On end: calls PATCH /spans/{id} to update with output and end_time
If inside a Temporal workflow: routes through Temporal activities instead of direct HTTP

SGP Tracing (`scale_gp_beta.lib.tracing`)

This is the SGP platform’s native tracing SDK. You can use it directly or as a dual-write processor alongside Agentex tracing.

The SGP processor intercepts Agentex span events
Converts to SGP span format via scale_gp_beta.lib.tracing.create_span()
Adds metadata: __source__: "agentex", __agent_name__, __agent_id__, __acp_type__
Flushes to the SGP backend via upsert_batch()

Which Should You Use?

For Temporal agents: Use adk.tracing.span(), which handles Temporal activity routing automatically
For standalone scripts: Use scale_gp_beta.lib.tracing.create_span() directly
For dual-write: Configure both processors (Agentex processor is the default; add the SGP processor for SGP UI visibility)

If you only configure the Agentex processor (the default), spans will NOT appear in the SGP Traces UI. Add the SGP processor for dual-write visibility. See the Quick Start above.

Next Steps

Want to understand what’s auto-traced for you? See the Built-in Tracing Providers guide for OpenAI Agents SDK, LiteLLM, and LangGraph coverage and gaps.
For naming conventions, input/output patterns, and profiler-friendly trace structure, see Span Hierarchy & Best Practices.
Building a multi-agent system? See Multi-Agent Tracing to propagate trace context across orchestrator and sub-agents.
For general tracing concepts and the SGP Tracing SDK, see Introduction to Tracing and Creating Traces and Spans.
To understand the Traces UI, see the Traces Table, Profiler, and Trace Detail View pages.
New to Agentex? Start with the Agentex Overview and Agentex Agent Types.

Overview

Getting Started

Infrastructure

Evaluations

Evaluation Dashboards

Tracing

Agents

Agentex Tracing Overview

What is Agentex Tracing?

Architecture

Key Identity Relationships

Core Concepts

trace_id = task_id

Span Hierarchy

Span Data Model

Quick Start

The Two Tracing Systems

Agentex Native Tracing (`adk.tracing`)

SGP Tracing (`scale_gp_beta.lib.tracing`)

Which Should You Use?

Next Steps

​What is Agentex Tracing?

​Architecture

​Key Identity Relationships

​Core Concepts

​trace_id = task_id

​Span Hierarchy

​Span Data Model

​Quick Start

​The Two Tracing Systems

​Agentex Native Tracing (adk.tracing)

​SGP Tracing (scale_gp_beta.lib.tracing)

​Which Should You Use?

​Next Steps

What is Agentex Tracing?

Architecture

Key Identity Relationships

Core Concepts

trace_id = task_id

Span Hierarchy

Span Data Model

Quick Start

The Two Tracing Systems

Agentex Native Tracing (`adk.tracing`)

SGP Tracing (`scale_gp_beta.lib.tracing`)

Which Should You Use?

Next Steps