Skip to main content

​
What is Agentex Tracing?

Modern AI agents execute complex, multi-step workflows: reasoning, calling tools, making LLM requests, and coordinating with other agents. Without tracing, these workflows are opaque. When something goes wrong or runs slowly, you’re left guessing. Agentex Tracing gives you deep visibility into your Temporal-style agents by capturing structured spans for every meaningful operation. You get:
  • One trace per task: trace_id maps directly to your Agentex task_id
  • A structured span hierarchy: turns, LLM calls, tool invocations, and custom logic all nest via parent_id
  • Tool and model I/O capture: see exactly what went into and came out of each operation
  • Profiler-ready data: consistent span naming enables aggregated performance analysis across all your traces
This guide covers tracing specific to Agentex agents. For general tracing concepts (spans, traces, the Traces UI), see Introduction to Tracing.

​
Architecture

​
Key Identity Relationships

ConceptID FieldTypical Value
Tracetrace_id= task_id (the Agentex task UUID)
SpanidUUID4, auto-generated
Parentparent_idspan.id of the enclosing span
Agentdata.__agent_name__Agent name from manifest
Sourcedata.__source__"agentex" (set by processor)

​
Core Concepts

​
trace_id = task_id

Every Agentex task has a unique ID. This same ID becomes the trace_id for all spans associated with that task. This 1:1 mapping means you can look up any task’s full execution history by searching for its task_id in the Traces UI.

​
Span Hierarchy

Spans form a tree via parent_id relationships. A turn-level span is the parent of the LLM calls and tool invocations that happen during that turn. This hierarchy is what the Traces UI renders as an expandable tree and Gantt chart.

​
Span Data Model

class Span(BaseModel):
    id: str                       # UUID4 - unique span identifier
    name: str                     # Descriptive operation name
    trace_id: str                 # Groups spans into a trace (= task_id)
    start_time: datetime          # UTC timestamp when span began
    end_time: Optional[datetime]  # UTC timestamp when span ended
    parent_id: Optional[str]      # Parent span ID for hierarchy
    input: Optional[Dict]         # Operation input data
    output: Optional[Dict]        # Operation output data
    data: Optional[Dict]          # Metadata (__span_type__, __source__, etc.)

​
Quick Start

1

Initialize Tracing in Your acp.py

Add the tracing initialization at the top level of your acp.py (or workflow.py for Temporal agents). This runs once when the agent process starts and configures both the Agentex and SGP tracing processors so spans appear in the SGP Traces UI.
import os
import logging

from agentex.lib.core.tracing.processors.tracing_processor_manager import (
    add_tracing_processor_config,
)
from agentex.lib.types.tracing import SGPTracingProcessorConfig

logger = logging.getLogger(__name__)

# --- Tracing initialization (top-level, runs once at process start) ---

SGP_API_KEY = os.getenv("SGP_API_KEY")
SGP_ACCOUNT_ID = os.getenv("SGP_ACCOUNT_ID")
SGP_BASE_URL = os.getenv("SGP_CLIENT_BASE_URL")

if SGP_API_KEY and SGP_ACCOUNT_ID:
    add_tracing_processor_config(
        SGPTracingProcessorConfig(
            type="sgp",
            sgp_api_key=SGP_API_KEY,         # Your SGP API key
            sgp_account_id=SGP_ACCOUNT_ID,   # Your SGP account ID
            sgp_base_url=SGP_BASE_URL,       # e.g. "https://api.scale.com"
        )
    )

    # Optional: also initialize the SGP tracing SDK directly
    # (needed if you use scale_gp_beta.lib.tracing.create_span() anywhere)
    try:
        import scale_gp_beta.lib.tracing as tracing
        from scale_gp_beta import SGPClient

        client = SGPClient(
            api_key=SGP_API_KEY,
            account_id=SGP_ACCOUNT_ID,
            base_url=SGP_BASE_URL,
        )
        tracing.init(client=client)
        logger.info("SGP tracing initialized")
    except ImportError:
        logger.warning("scale_gp_beta not installed, skipping SGP SDK tracing init")
else:
    logger.warning("SGP tracing disabled (missing SGP_API_KEY or SGP_ACCOUNT_ID)")
The three required environment variables:
Env VarDescriptionExample
SGP_API_KEYYour SGP API key for authenticationscl_...
SGP_ACCOUNT_IDYour SGP account identifieracc_...
SGP_CLIENT_BASE_URLThe SGP API base URL. Must be set, there is no default.https://api.scale.com
SGP_CLIENT_BASE_URL has no default value. If it is not set, the tracing processor will not know where to send spans. Make sure this is configured in your agent’s environment or secrets.
2

Create a Turn Span

Wrap each conversation turn in a manual span. This groups all LLM calls and tool invocations for that turn.
from agentex.lib import adk

@acp.on_task_event_send
async def handle_event_send(params: SendEventParams):
    state = ...  # get state
    state.turn_number += 1

    async with adk.tracing.span(
        trace_id=params.task.id,
        name=f"turn:{state.turn_number}",
        input=state.model_dump(),
    ) as span:
        # Everything inside this block nests under this turn span
        pass
3

Make a Traced LLM Call

Pass trace_id and parent_span_id to your LLM call. The built-in providers auto-create a child span.
    async with adk.tracing.span(
        trace_id=params.task.id,
        name=f"turn:{state.turn_number}",
        input=state.model_dump(),
    ) as span:
        response = await adk.providers.litellm.chat_completion_stream_auto_send(
            task_id=params.task.id,
            llm_config=LLMConfig(model="gpt-4o-mini", messages=state.messages),
            trace_id=params.task.id,
            parent_span_id=span.id if span else None,
        )

        if span:
            span.output = state.model_dump()
4

View in the Traces UI

Open the SGP Traces page, filter by your agent name, and click on the trace. You should see your turn span with the LLM call nested beneath it.
Trace detail view showing a turn span with a child LLM span
For a complete reference on the SGP Tracing SDK, see Creating Traces and Spans.

​
The Two Tracing Systems

​
Agentex Native Tracing (adk.tracing)

This is the primary system for Temporal agents. Spans are sent to the Agentex backend API.
  1. adk.tracing.span() creates a span with a UUID4 id
  2. On start: calls POST /spans to create the span in the Agentex database
  3. On end: calls PATCH /spans/{id} to update with output and end_time
  4. If inside a Temporal workflow: routes through Temporal activities instead of direct HTTP

​
SGP Tracing (scale_gp_beta.lib.tracing)

This is the SGP platform’s native tracing SDK. You can use it directly or as a dual-write processor alongside Agentex tracing.
  1. The SGP processor intercepts Agentex span events
  2. Converts to SGP span format via scale_gp_beta.lib.tracing.create_span()
  3. Adds metadata: __source__: "agentex", __agent_name__, __agent_id__, __acp_type__
  4. Flushes to the SGP backend via upsert_batch()

​
Which Should You Use?

  • For Temporal agents: Use adk.tracing.span(), which handles Temporal activity routing automatically
  • For standalone scripts: Use scale_gp_beta.lib.tracing.create_span() directly
  • For dual-write: Configure both processors (Agentex processor is the default; add the SGP processor for SGP UI visibility)
If you only configure the Agentex processor (the default), spans will NOT appear in the SGP Traces UI. Add the SGP processor for dual-write visibility. See the Quick Start above.

​
Next Steps