> ## Documentation Index > Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt > Use this file to discover all available pages before exploring further. # Agentex Tracing Overview > Instrument Agentex agents with tracing to capture every turn, LLM call, and tool invocation as structured spans in the SGP Traces UI. ## What is Agentex Tracing? Modern AI agents execute complex, multi-step workflows: reasoning, calling tools, making LLM requests, and coordinating with other agents. Without tracing, these workflows are opaque. When something goes wrong or runs slowly, you're left guessing. [Agentex](/docs/v5/agents/agentex/overview) Tracing gives you deep visibility into your Temporal-style agents by capturing structured spans for every meaningful operation. You get: * **One trace per task**: `trace_id` maps directly to your Agentex `task_id` * **A structured span hierarchy**: turns, LLM calls, tool invocations, and custom logic all nest via `parent_id` * **Tool and model I/O capture**: see exactly what went into and came out of each operation * **Profiler-ready data**: consistent span naming enables aggregated performance analysis across all your traces This guide covers tracing specific to Agentex agents. For general tracing concepts (spans, traces, the Traces UI), see [Introduction to Tracing](/docs/v5/tracing/overview). *** ## Architecture ```mermaid theme={null} graph LR subgraph Agent["Agent Code"] MS["adk.tracing.span()"] AL["TemporalTracingModelProvider"] AT["LangGraph Handler"] end subgraph Agentex["Agentex Backend"] API["Span API"] DB[("PostgreSQL")] end subgraph SGP["SGP Platform"] PROC["SGP Tracing Processor"] SGPDB[("SGP Backend")] UI["Traces UI"] end MS --> API AL --> API AT --> API API --> DB API --> PROC PROC --> SGPDB SGPDB --> UI ``` ### Key Identity Relationships | Concept | ID Field | Typical Value | | ------- | --------------------- | ----------------------------------- | | Trace | `trace_id` | = `task_id` (the Agentex task UUID) | | Span | `id` | UUID4, auto-generated | | Parent | `parent_id` | `span.id` of the enclosing span | | Agent | `data.__agent_name__` | Agent name from manifest | | Source | `data.__source__` | `"agentex"` (set by processor) | *** ## Core Concepts ### trace\_id = task\_id Every Agentex task has a unique ID. This same ID becomes the `trace_id` for all spans associated with that task. This 1:1 mapping means you can look up any task's full execution history by searching for its `task_id` in the Traces UI. ### Span Hierarchy Spans form a tree via `parent_id` relationships. A turn-level span is the parent of the LLM calls and tool invocations that happen during that turn. This hierarchy is what the Traces UI renders as an expandable tree and Gantt chart. ### Span Data Model ```python theme={null} class Span(BaseModel): id: str # UUID4 - unique span identifier name: str # Descriptive operation name trace_id: str # Groups spans into a trace (= task_id) start_time: datetime # UTC timestamp when span began end_time: Optional[datetime] # UTC timestamp when span ended parent_id: Optional[str] # Parent span ID for hierarchy input: Optional[Dict] # Operation input data output: Optional[Dict] # Operation output data data: Optional[Dict] # Metadata (__span_type__, __source__, etc.) ``` *** ## Quick Start Add the tracing initialization at the top level of your `acp.py` (or `workflow.py` for Temporal agents). This runs once when the agent process starts and configures both the Agentex and SGP tracing processors so spans appear in the SGP Traces UI. ```python theme={null} import os import logging from agentex.lib.core.tracing.processors.tracing_processor_manager import ( add_tracing_processor_config, ) from agentex.lib.types.tracing import SGPTracingProcessorConfig logger = logging.getLogger(__name__) # --- Tracing initialization (top-level, runs once at process start) --- SGP_API_KEY = os.getenv("SGP_API_KEY") SGP_ACCOUNT_ID = os.getenv("SGP_ACCOUNT_ID") SGP_BASE_URL = os.getenv("SGP_CLIENT_BASE_URL") if SGP_API_KEY and SGP_ACCOUNT_ID: add_tracing_processor_config( SGPTracingProcessorConfig( type="sgp", sgp_api_key=SGP_API_KEY, # Your SGP API key sgp_account_id=SGP_ACCOUNT_ID, # Your SGP account ID sgp_base_url=SGP_BASE_URL, # e.g. "https://api.scale.com" ) ) # Optional: also initialize the SGP tracing SDK directly # (needed if you use scale_gp_beta.lib.tracing.create_span() anywhere) try: import scale_gp_beta.lib.tracing as tracing from scale_gp_beta import SGPClient client = SGPClient( api_key=SGP_API_KEY, account_id=SGP_ACCOUNT_ID, base_url=SGP_BASE_URL, ) tracing.init(client=client) logger.info("SGP tracing initialized") except ImportError: logger.warning("scale_gp_beta not installed, skipping SGP SDK tracing init") else: logger.warning("SGP tracing disabled (missing SGP_API_KEY or SGP_ACCOUNT_ID)") ``` The three required environment variables: | Env Var | Description | Example | | --------------------- | ----------------------------------------------------------- | ----------------------- | | `SGP_API_KEY` | Your SGP API key for authentication | `scl_...` | | `SGP_ACCOUNT_ID` | Your SGP account identifier | `acc_...` | | `SGP_CLIENT_BASE_URL` | The SGP API base URL. **Must be set**, there is no default. | `https://api.scale.com` | `SGP_CLIENT_BASE_URL` has no default value. If it is not set, the tracing processor will not know where to send spans. Make sure this is configured in your agent's environment or secrets. Wrap each conversation turn in a manual span. This groups all LLM calls and tool invocations for that turn. ```python theme={null} from agentex.lib import adk @acp.on_task_event_send async def handle_event_send(params: SendEventParams): state = ... # get state state.turn_number += 1 async with adk.tracing.span( trace_id=params.task.id, name=f"turn:{state.turn_number}", input=state.model_dump(), ) as span: # Everything inside this block nests under this turn span pass ``` Pass `trace_id` and `parent_span_id` to your LLM call. The built-in providers auto-create a child span. ```python theme={null} async with adk.tracing.span( trace_id=params.task.id, name=f"turn:{state.turn_number}", input=state.model_dump(), ) as span: response = await adk.providers.litellm.chat_completion_stream_auto_send( task_id=params.task.id, llm_config=LLMConfig(model="gpt-4o-mini", messages=state.messages), trace_id=params.task.id, parent_span_id=span.id if span else None, ) if span: span.output = state.model_dump() ``` Open the SGP Traces page, filter by your agent name, and click on the trace. You should see your turn span with the LLM call nested beneath it. Trace detail view showing a turn span with a child LLM span For a complete reference on the SGP Tracing SDK, see [Creating Traces and Spans](/docs/v5/tracing/sdk/creating-traces-spans). The [Traces table](/docs/v5/tracing/tracing-ui/traces-table#controlling-the-content-preview) preview column extracts text from specific preferred keys in your span `input` and `output` (e.g., `title`, `query`, `content`). If the preview shows an unhelpful value, add a `title` key with the value you want displayed. *** ## The Two Tracing Systems ### Agentex Native Tracing (`adk.tracing`) This is the primary system for Temporal agents. Spans are sent to the Agentex backend API. 1. `adk.tracing.span()` creates a span with a UUID4 id 2. On start: calls `POST /spans` to create the span in the Agentex database 3. On end: calls `PATCH /spans/{id}` to update with output and end\_time 4. If inside a Temporal workflow: routes through Temporal activities instead of direct HTTP ### SGP Tracing (`scale_gp_beta.lib.tracing`) This is the SGP platform's native tracing SDK. You can use it directly or as a dual-write processor alongside Agentex tracing. 1. The SGP processor intercepts Agentex span events 2. Converts to SGP span format via `scale_gp_beta.lib.tracing.create_span()` 3. Adds metadata: `__source__: "agentex"`, `__agent_name__`, `__agent_id__`, `__acp_type__` 4. Flushes to the SGP backend via `upsert_batch()` ### Which Should You Use? * **For Temporal agents**: Use `adk.tracing.span()`, which handles Temporal activity routing automatically * **For standalone scripts**: Use `scale_gp_beta.lib.tracing.create_span()` directly * **For dual-write**: Configure both processors (Agentex processor is the default; add the SGP processor for SGP UI visibility) If you only configure the Agentex processor (the default), spans will NOT appear in the SGP Traces UI. Add the SGP processor for dual-write visibility. See the Quick Start above. *** ## Next Steps * Want to understand what's auto-traced for you? See the [Built-in Tracing Providers](./built-in-providers) guide for OpenAI Agents SDK, LiteLLM, and LangGraph coverage and gaps. * For naming conventions, input/output patterns, and profiler-friendly trace structure, see [Span Hierarchy & Best Practices](./span-hierarchy). * Building a multi-agent system? See [Multi-Agent Tracing](./multi-agent-tracing) to propagate trace context across orchestrator and sub-agents. * For general tracing concepts and the SGP Tracing SDK, see [Introduction to Tracing](/docs/v5/tracing/overview) and [Creating Traces and Spans](/docs/v5/tracing/sdk/creating-traces-spans). * To understand the Traces UI, see the [Traces Table](/docs/v5/tracing/tracing-ui/traces-table), [Profiler](/docs/v5/tracing/tracing-ui/profiler), and [Trace Detail View](/docs/v5/tracing/tracing-ui/trace-detail-view) pages. * New to Agentex? Start with the [Agentex Overview](/docs/v5/agents/agentex/overview) and [Agentex Agent Types](/docs/v5/agents/agentex/types/agentic-agent).