Multi-Agent Tracing

This page builds on the concepts in Agentex Tracing Overview and Span Hierarchy & Best Practices. Make sure you’re familiar with creating spans and setting parent_id before diving into multi-agent propagation.

The Problem

In a multi-agent system, each agent creates its own trace by default (trace_id = task_id, and each agent has its own task). The result is fragmented observability: the orchestrator’s trace shows only its own spans, and each sub-agent’s work lives in a separate, disconnected trace. You end up with three separate traces instead of one unified view of the entire workflow.

The Solution: Trace ID Propagation

Pass the orchestrator’s trace_id to every sub-agent so all spans land in a single trace tree. You also thread parent_span_id through each dispatch so the hierarchy forms correctly. Two things make this work:

Same trace_id everywhere: the orchestrator’s task_id is used as the trace_id for all sub-agent spans
Threaded parent_span_id: each dispatch passes the current span’s ID so sub-agent work nests correctly

Context Propagation in Temporal

For Temporal-based agents, trace context flows automatically through the interceptor system: The workflow instance exposes self._task_id, self._trace_id, and self._parent_span_id. The outbound interceptor copies these into Temporal activity headers (context-task-id, context-trace-id, context-parent-span-id). On the activity side, the inbound interceptor reads those headers into ContextVars (streaming_task_id, streaming_trace_id, streaming_parent_span_id) that the tracing model picks up automatically.

Setting Up Context in Your Workflow

from agentex.lib import adk
from temporalio import workflow

@workflow.defn
class MyAgentWorkflow:
    @workflow.run
    async def run(self, params):
        # These are read by ContextWorkflowOutboundInterceptor
        self._task_id = params.task_id
        self._trace_id = params.task_id  # trace_id = task_id

        # Create a turn-level span
        async with adk.tracing.span(
            trace_id=params.task_id, name="turn:1"
        ) as span:
            self._parent_span_id = span.span_id  # Auto-traced LLM calls nest under this

            # Any activity using TemporalTracingModelProvider
            # will automatically create child spans under this parent
            result = await workflow.execute_activity(
                invoke_model_activity, ...
            )

ContextInterceptor reads self._task_id, self._trace_id, and self._parent_span_id from the workflow instance. Setting these attributes is all you need. The interceptor handles header propagation automatically.

Multi-Agent Code Example

Orchestrator Agent

The orchestrator creates spans for each sub-agent dispatch and passes its trace_id and parent_span_id through:

from agentex.lib import adk

async with adk.tracing.span(
    trace_id=orchestrator_task_id,
    name="turn:1",
    input={"sub_agents": ["research", "analysis", "synthesis"]},
) as orchestration_span:

    for sub_agent_name in sub_agents:
        async with adk.tracing.span(
            trace_id=orchestrator_task_id,
            parent_id=orchestration_span.id,
            name=f"subtask:{sub_agent_name}",
            input={"agent": sub_agent_name},
        ) as dispatch_span:

            # Pass trace context to the sub-agent
            await adk.tasks.create(
                agent_name=sub_agent_name,
                input=task_input,
                trace_id=orchestrator_task_id,
                parent_span_id=dispatch_span.id,
            )

Sub-Agent (Child)

The sub-agent uses the incoming trace_id from the orchestrator instead of its own task_id:

@acp.on_task_event_send
async def handle_event_send(params: SendEventParams):
    # Use the propagated trace_id, NOT params.task.id
    trace_id = params.trace_id or params.task.id  # Fallback for standalone use

    async with adk.tracing.span(
        trace_id=trace_id,
        parent_id=params.parent_span_id,
        name=f"turn:{state.turn_number}",
        input=state.model_dump(),
    ) as span:
        response = await adk.providers.litellm.chat_completion_stream_auto_send(
            task_id=params.task.id,
            llm_config=llm_config,
            trace_id=trace_id,
            parent_span_id=span.id if span else None,
        )

        # Message to both parent and child task for visibility
        await adk.messages.create(
            task_id=params.task.id,
            trace_id=trace_id,
            parent_span_id=span.id if span else None,
            content=TextContent(author="agent", content=response_text),
        )

Resulting Unified Trace Tree

With trace propagation, the entire multi-agent workflow appears under a single trace:

Verification

Run your orchestrator agent

Start the orchestrator: agentex agents run --manifest manifest.yaml

Trigger sub-agent dispatch

Send a task or message that causes the orchestrator to dispatch work to sub-agents.

Check spans via Agentex API

Query for all spans in the trace:

curl "https://your-agentex-url/spans?trace_id={orchestrator_task_id}"

Verify that sub-agent spans share the orchestrator’s trace_id.

Verify hierarchy

Check that each span’s parent_id references an existing span’s id. Sub-agent spans should nest under the orchestrator’s dispatch spans.

Confirm in the SGP UI

Open the Trace Detail View, filter by the orchestrator’s task_id, and confirm the unified tree shows orchestrator and sub-agent spans together.

Troubleshooting

Sub-agent spans appear in a separate trace

The sub-agent is using its own task_id as trace_id instead of the orchestrator’s. Check that:

The orchestrator passes its task_id as trace_id when creating the sub-agent task
The sub-agent reads and uses the propagated trace_id (e.g., params.trace_id or params.task.id)
All adk.tracing.span() calls in the sub-agent use the propagated trace_id

Spans are flat instead of nested

Parent-child relationships are missing. Check that:

parent_span_id is passed through the dispatch (not just trace_id)
Sub-agent spans set parent_id to the received parent_span_id
Inside the sub-agent, child operations (LLM calls, tools) receive parent_span_id=span.id from the enclosing turn span

Auto-traced LLM spans are orphaned (no parent)

The workflow’s self._parent_span_id was not set before executing the activity. Ensure:

self._parent_span_id = span.id is set before calling workflow.execute_activity()
The turn span is created and its id is captured before any activity execution
For Temporal workflows, the ContextInterceptor is registered in the worker’s interceptor list

Overview

Getting Started

Infrastructure

Evaluations

Evaluation Dashboards

Tracing

Agents

Multi-Agent Tracing

The Problem

The Solution: Trace ID Propagation

Context Propagation in Temporal

Setting Up Context in Your Workflow

Multi-Agent Code Example

Orchestrator Agent

Sub-Agent (Child)

Resulting Unified Trace Tree

Verification

Troubleshooting

​The Problem

​The Solution: Trace ID Propagation

​Context Propagation in Temporal

​Setting Up Context in Your Workflow

​Multi-Agent Code Example

​Orchestrator Agent

​Sub-Agent (Child)

​Resulting Unified Trace Tree

​Verification

​Troubleshooting

The Problem

The Solution: Trace ID Propagation

Context Propagation in Temporal

Setting Up Context in Your Workflow

Multi-Agent Code Example

Orchestrator Agent

Sub-Agent (Child)

Resulting Unified Trace Tree

Verification

Troubleshooting