The Profiler provides an aggregated overview of where your agent spends its time, helping you pinpoint which spans are consistently causing delays across many executions. Instead of analyzing individual traces one by one, the Profiler shows performance metrics aggregated from all traces matching your filters, making it easy to identify systemic bottlenecks.
The span table is the core of the Profiler. Instead of listing individual traces, it lists every unique span name pertaining to the selected agent (e.g., generate_assistant_message, followup_search_state) and shows performance metrics aggregated from all traces.Key columns include:
Span Name: The name of the operation.
Average Duration: The average time this span takes to complete.
Total Time (%): The percentage of total execution time across all traces consumed by this span. Sorting by this column is the most effective way to find major bottlenecks.
P50 Latency: The median execution time, representing typical performance.
P95 Latency: The 95th percentile latency, crucial for understanding worst-case performance and identifying outliers.
Frequency (Avg. per Trace): How many times this span is called on average in a single execution.
Click any row to view detailed metrics in the Span Inspector.
When you click a row in the span table or a bar in the flame graph, the Inspector panel slides out with a deep dive on that specific span.Detailed Stats: The inspector shows all key metrics for the selected span:
% Execution Time: The percentage of total execution time consumed by this span across all traces.
Average Duration: Mean execution time for this span.
Min: The fastest execution time observed.
P50 (Median): The median execution time, representing typical performance.
P95: The 95th percentile execution time.
P99: The 99th percentile execution time.
Max: The slowest execution time observed.
Latency Distribution: A histogram showing the spread of durations for this span. This visualization helps you understand if latency is consistent or has a long tail of slow outliers.Example Traces: Critical links back to the Traces page for deeper investigation:
View Slowest Traces: Takes you to the Traces page, pre-filtered to show executions where this specific span was in the 95th percentile of slowness.
View Recent Traces: Links to the most recent traces that include this span.
The Problem: A developer notices that the “Deep Research V3” agent’s duration varies wildly. Some traces are fast, but others (like a 117.7s execution) are painfully slow. They don’t know where to start looking.Step 1: Navigate to ProfilerFrom the Traces view, they click the Profiler tab and select the “Deep Research V3” agent to analyze performance over the last 7 days.Step 2: Identify the BottleneckThey sort the Aggregated Span Table by Total Time (%) to find major bottlenecks. They immediately see that final_report_state is at the top, consuming 38% of all agent execution time. They also notice its P95 latency is over 2x its P50 median (85.6s vs 35.5s), confirming it’s a source of inconsistent performance.Step 3: Inspect the SpanThey click the final_report_state row in the span table. The Span Inspector panel slides out, showing a histogram with a long tail. The detailed statistics confirm the outlier problem—some executions are extremely slow while most are reasonable.Step 4: Find Specific ExamplesIn the Span Inspector, they click “View Slowest Traces” to see the actual problem cases.Step 5: DebugThey are taken back to the Traces page, but it’s pre-filtered to show only the 5% of traces where final_report_state was slowest. They click the top trace in this list, which brings them to the detailed waterfall view. Now, instead of guessing, they know to focus their investigation on the final_report_state span and its children to solve the systemic issue.
