> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gp.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Agent Tracing

> Propagate trace context across orchestrator and sub-agents to build unified, cross-agent trace trees.

This page builds on the concepts in [Agentex Tracing Overview](./overview) and [Span Hierarchy & Best Practices](./span-hierarchy). Make sure you're familiar with creating spans and setting `parent_id` before diving into multi-agent propagation.

## The Problem

In a [multi-agent system](/docs/v5/agents/agentex/overview), each agent creates its own trace by default (`trace_id = task_id`, and each agent has its own task). The result is fragmented observability: the orchestrator's trace shows only its own spans, and each sub-agent's work lives in a separate, disconnected trace.

```mermaid theme={null}
graph TD
    subgraph Trace1["Trace: orchestrator_task_123"]
        _s1[ ]:::spacer
        _s1 ~~~ O1
        O1["turn:1"]
        O1 --> OL1["llm:gpt-4o"]
        O1 --> OD["dispatch sub-agents"]
    end

    subgraph Trace2["Trace: research_task_456"]
        _s2[ ]:::spacer
        _s2 ~~~ R1
        R1["turn:1"]
        R1 --> RL1["llm:o3"]
        R1 --> RT1["tool:web_search"]
    end

    subgraph Trace3["Trace: analysis_task_789"]
        _s3[ ]:::spacer
        _s3 ~~~ A1
        A1["turn:1"]
        A1 --> AL1["llm:o3"]
    end

    classDef spacer fill:none,stroke:none,color:transparent
    style Trace1 stroke:#e74c3c
    style Trace2 stroke:#3498db
    style Trace3 stroke:#2ecc71
```

You end up with three separate traces instead of one unified view of the entire workflow.

***

## The Solution: Trace ID Propagation

Pass the orchestrator's `trace_id` to every sub-agent so all spans land in a single trace tree. You also thread `parent_span_id` through each dispatch so the hierarchy forms correctly.

```mermaid theme={null}
graph TD
    subgraph Unified["Trace: orchestrator_task_123"]
        _s0[ ]:::spacer
        _s0 ~~~ O1
        O1["turn:1 · orchestrator"]
        O1 --> Sub1["subtask:research"]
        Sub1 --> RL1["llm:o3"]
        Sub1 --> RT1["tool:web_search"]
        O1 --> Sub2["subtask:analysis"]
        Sub2 --> AL1["llm:o3"]
        O1 --> OL1["llm:gpt-4o · synthesis"]
    end
    classDef spacer fill:none,stroke:none,color:transparent
```

Two things make this work:

1. **Same `trace_id` everywhere**: the orchestrator's `task_id` is used as the `trace_id` for all sub-agent spans
2. **Threaded `parent_span_id`**: each dispatch passes the current span's ID so sub-agent work nests correctly

***

## Context Propagation in Temporal

For Temporal-based agents, trace context flows automatically through the interceptor system:

```mermaid theme={null}
graph LR
    subgraph WF["Workflow Side"]
        WI["Workflow Instance"]
        CO["OutboundInterceptor"]
    end

    subgraph Headers["Temporal Transport"]
        H["Activity Headers"]
    end

    subgraph ACT["Activity Side"]
        CI["InboundInterceptor"]
        CV["ContextVars"]
        TM["TracingModel"]
    end

    WI --> CO
    CO --> H
    H --> CI
    CI --> CV
    CV --> TM
    TM --> S["Auto-traced span"]
```

The workflow instance exposes `self._task_id`, `self._trace_id`, and `self._parent_span_id`. The outbound interceptor copies these into Temporal activity headers (`context-task-id`, `context-trace-id`, `context-parent-span-id`). On the activity side, the inbound interceptor reads those headers into `ContextVar`s (`streaming_task_id`, `streaming_trace_id`, `streaming_parent_span_id`) that the tracing model picks up automatically.

### Setting Up Context in Your Workflow

```python theme={null}
from agentex.lib import adk
from temporalio import workflow

@workflow.defn
class MyAgentWorkflow:
    @workflow.run
    async def run(self, params):
        # These are read by ContextWorkflowOutboundInterceptor
        self._task_id = params.task_id
        self._trace_id = params.task_id  # trace_id = task_id

        # Create a turn-level span
        async with adk.tracing.span(
            trace_id=params.task_id, name="turn:1"
        ) as span:
            self._parent_span_id = span.span_id  # Auto-traced LLM calls nest under this

            # Any activity using TemporalTracingModelProvider
            # will automatically create child spans under this parent
            result = await workflow.execute_activity(
                invoke_model_activity, ...
            )
```

<Info>
  `ContextInterceptor` reads `self._task_id`, `self._trace_id`, and `self._parent_span_id` from the workflow instance. Setting these attributes is all you need. The interceptor handles header propagation automatically.
</Info>

***

## Multi-Agent Code Example

### Orchestrator Agent

The orchestrator creates spans for each sub-agent dispatch and passes its `trace_id` and `parent_span_id` through:

```python theme={null}
from agentex.lib import adk

async with adk.tracing.span(
    trace_id=orchestrator_task_id,
    name="turn:1",
    input={"sub_agents": ["research", "analysis", "synthesis"]},
) as orchestration_span:

    for sub_agent_name in sub_agents:
        async with adk.tracing.span(
            trace_id=orchestrator_task_id,
            parent_id=orchestration_span.id,
            name=f"subtask:{sub_agent_name}",
            input={"agent": sub_agent_name},
        ) as dispatch_span:

            # Pass trace context to the sub-agent
            await adk.tasks.create(
                agent_name=sub_agent_name,
                input=task_input,
                trace_id=orchestrator_task_id,
                parent_span_id=dispatch_span.id,
            )
```

### Sub-Agent (Child)

The sub-agent uses the **incoming `trace_id`** from the orchestrator instead of its own `task_id`:

```python theme={null}
@acp.on_task_event_send
async def handle_event_send(params: SendEventParams):
    # Use the propagated trace_id, NOT params.task.id
    trace_id = params.trace_id or params.task.id  # Fallback for standalone use

    async with adk.tracing.span(
        trace_id=trace_id,
        parent_id=params.parent_span_id,
        name=f"turn:{state.turn_number}",
        input=state.model_dump(),
    ) as span:
        response = await adk.providers.litellm.chat_completion_stream_auto_send(
            task_id=params.task.id,
            llm_config=llm_config,
            trace_id=trace_id,
            parent_span_id=span.id if span else None,
        )

        # Message to both parent and child task for visibility
        await adk.messages.create(
            task_id=params.task.id,
            trace_id=trace_id,
            parent_span_id=span.id if span else None,
            content=TextContent(author="agent", content=response_text),
        )
```

***

## Resulting Unified Trace Tree

With trace propagation, the entire multi-agent workflow appears under a single trace:

```mermaid theme={null}
graph TD
    T["Trace: orchestrator_task_123"]

    T --> T1["turn:1 · orchestrator"]
    T1 --> SR["subtask:research"]
    SR --> SRT1["turn:1 · research agent"]
    SRT1 --> SRL1["llm:gpt-4o"]
    SRT1 --> SRTool["tool:web_search"]
    SRT1 --> SRL2["llm:gpt-4o"]

    T1 --> SA["subtask:analysis"]
    SA --> SAT1["turn:1 · analysis agent"]
    SAT1 --> SAL1["llm:o3"]
    SAT1 --> SATool["tool:code_interpreter"]

    T1 --> SS["subtask:synthesis"]
    SS --> SST1["turn:1 · synthesis agent"]
    SST1 --> SSL1["llm:gpt-4o"]
```

***

## Verification

<Steps>
  <Step title="Run your orchestrator agent">
    Start the orchestrator: `agentex agents run --manifest manifest.yaml`
  </Step>

  <Step title="Trigger sub-agent dispatch">
    Send a task or message that causes the orchestrator to dispatch work to sub-agents.
  </Step>

  <Step title="Check spans via Agentex API">
    Query for all spans in the trace:

    ```bash theme={null}
    curl "https://your-agentex-url/spans?trace_id={orchestrator_task_id}"
    ```

    Verify that sub-agent spans share the orchestrator's `trace_id`.
  </Step>

  <Step title="Verify hierarchy">
    Check that each span's `parent_id` references an existing span's `id`. Sub-agent spans should nest under the orchestrator's dispatch spans.
  </Step>

  <Step title="Confirm in the SGP UI">
    Open the [Trace Detail View](/docs/v5/tracing/tracing-ui/trace-detail-view), filter by the orchestrator's `task_id`, and confirm the unified tree shows orchestrator and sub-agent spans together.
  </Step>
</Steps>

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Sub-agent spans appear in a separate trace">
    The sub-agent is using its own `task_id` as `trace_id` instead of the orchestrator's. Check that:

    * The orchestrator passes its `task_id` as `trace_id` when creating the sub-agent task
    * The sub-agent reads and uses the propagated `trace_id` (e.g., `params.trace_id or params.task.id`)
    * All `adk.tracing.span()` calls in the sub-agent use the propagated `trace_id`
  </Accordion>

  <Accordion title="Spans are flat instead of nested">
    Parent-child relationships are missing. Check that:

    * `parent_span_id` is passed through the dispatch (not just `trace_id`)
    * Sub-agent spans set `parent_id` to the received `parent_span_id`
    * Inside the sub-agent, child operations (LLM calls, tools) receive `parent_span_id=span.id` from the enclosing turn span
  </Accordion>

  <Accordion title="Auto-traced LLM spans are orphaned (no parent)">
    The workflow's `self._parent_span_id` was not set before executing the activity. Ensure:

    * `self._parent_span_id = span.id` is set **before** calling `workflow.execute_activity()`
    * The turn span is created and its `id` is captured before any activity execution
    * For Temporal workflows, the `ContextInterceptor` is registered in the worker's interceptor list
  </Accordion>
</AccordionGroup>