OpenTelemetry Integration

​

Prerequisites

• An SGP API key and the ID of the account that will own the traces.
• Your SGP Traces host. The default is sgp-traces.<deployment-url>, where <deployment-url> is the domain your SGP API runs under as api.<deployment-url>. The development deployment, for example, is https://sgp-traces.dev-sgp.scale.com/v1/traces. Deployments can override the host; ask your deployment owner if the default does not resolve.
• A framework or SDK that exports over OTLP/HTTP protobuf. For hand instrumentation in Python:

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

​

Configure the exporter

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://sgp-traces.<deployment-url>/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="x-api-key=${SGP_API_KEY},x-selected-account-id=${SGP_ACCOUNT_ID}"
export OTEL_EXPORTER_OTLP_TRACES_COMPRESSION="gzip" # optional

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

with trace.get_tracer("my-app").start_as_current_span("hello"):
    ...

provider.shutdown()  # flushes pending spans before the process exits

​

Framework notes

• Pydantic AI emits spans only after Agent.instrument_all() or instrument=True, through the global tracer provider, so register the exporter as above. With Logfire, logfire.configure(send_to_logfire=False) builds the exporter from the OTEL_EXPORTER_OTLP_* variables and logfire.instrument_pydantic_ai() turns the spans on.
• OpenLLMetry (Traceloop) ignores the OTEL_EXPORTER_OTLP_* variables. Set TRACELOOP_BASE_URL="https://sgp-traces.<deployment-url>" (it appends /v1/traces) and TRACELOOP_HEADERS="x-api-key=...,x-selected-account-id=...", or pass api_endpoint and headers to Traceloop.init().
• LangSmith and LangChain need pip install "langsmith[otel]", LANGSMITH_TRACING=true to create runs at all, and LANGSMITH_TRACING_MODE=otel to export them over OpenTelemetry (older releases: LANGSMITH_OTEL_ENABLED=true plus LANGSMITH_OTEL_ONLY=true, otherwise runs also go to LangSmith). LangSmith reads only the generic OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS and uses the endpoint verbatim, so include /v1/traces. An already-configured global tracer provider takes precedence.
• Claude Agent SDK and Claude Code keep telemetry opt-in and trace export in beta. Add CLAUDE_CODE_ENABLE_TELEMETRY=1, CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1, OTEL_TRACES_EXPORTER=otlp, and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf to the variables above. Only the built-in claude_code.llm_request spans carry gen_ai.system and gen_ai.request.model, and none carry the semconv input, output, or token keys, so those land as completions with a model and nothing else while interaction and tool spans get no derived fields; the prompt is redacted unless OTEL_LOG_USER_PROMPTS=1. For full input, output, and token mapping, instrument the Agent SDK with otel-instrumentation-claude-agent-sdk instead.

​

Endpoint contract

​

Errors

• 400: the whole request is rejected. Causes: a missing x-selected-account-id (including the legacy x-sgp-account-id in its place), non-protobuf content type, Content-Encoding other than gzip or identity, invalid gzip, empty body, more than 1,000 spans, an invalid span such as an all-zero trace ID or an end time before its start, or a malformed sgp.obs_trace_id or sgp.obs_span_id. Fix the request or producer; do not retry. With an OpenTelemetry Collector, set both send_batch_size: 1000 and send_batch_max_size: 1000; under load the defaults emit 8,192-span batches, which are rejected.
• 401: invalid API key.
• 403: the key is not a member of the account, or its role there is not admin, manager, or editor.
• 413: body over the deployment limit. Lower the export batch size.
• 429: rate limited. The ingress gateway’s limit is one bucket per gateway pod shared by every caller; it answers with a plain-text body and no Retry-After. The service’s own load shedding answers with an OTLP Status and Retry-After. The OTLP specification makes 429, 502, 503, and 504 retryable and exporters from 2026 onward follow it, but older ones differ: the Python OTLP/HTTP exporter up to 1.44.0 retries only 408 and 5xx, so a 429 drops the batch. Keep exporters current, stay within your deployment’s limit, or export through a Collector, which retries 429 with backoff.

​

How spans map to SGP

• Input and output: gen_ai.input.messages and gen_ai.output.messages, then the indexed gen_ai.prompt.N and gen_ai.completion.N attributes, then gen_ai.prompt and gen_ai.completion, then framework-specific keys, then a tool span’s gen_ai.tool.call.arguments and gen_ai.tool.call.result, and finally the legacy gen_ai.content.* events.
• Token usage: gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, or the prompt_tokens and completion_tokens spellings.
• Operation type: gen_ai.operation.name, then traceloop.span.kind, then rerank and vector-store signals, then a tool span (gen_ai.tool.name, gen_ai.tool.call.*) with no model or tokens (custom step), then gen_ai.system or gen_ai.provider.name (completion). A span with derived input or output and none of these is also a completion. The operations fetch_response, guardrail_check, and agent_handoff are custom steps, and so is unknown on a span with gen_ai.system or gen_ai.provider.name but no model, tokens, input, or output.
• Model and provider metadata from the same keys are merged into the span’s attributes.

​

Migrating from the legacy forwarder

​

Viewing traces