LLM Observability

Tracing, metrics and feedback for LLM apps and agents: the span data model, OpenTelemetry GenAI conventions, what to log, and how LangSmith, Langfuse and Phoenix differ.

Contents

LLM observability is the practice of recording what an LLM application actually did on each request (the prompts and context sent, every model, tool and retrieval call, tokens, cost, latency, errors and outcome) as structured traces, then aggregating, searching and scoring them. Classic APM tells you a request was slow or failed; LLM observability also has to tell you why the answer was wrong, because outputs are open-ended and non-deterministic, and a prompt tweak or model upgrade can quietly break cases that used to work (notebook). Synthesis: a wrong answer usually still returns HTTP 200, so error-rate dashboards alone miss it. It is the production half of the eval loop: offline evaluation gates a change, observability shows what real traffic does and feeds failures back into the test set.

The data model: traces, spans, sessions

Every tool in this space uses the same shape as distributed tracing:

ConceptOpenTelemetry GenAILangSmithLangfusePhoenix / OpenInference
One step (LLM call, tool, retrieval)spanrunobservationspan
One request end to endtracetrace (root run)tracetrace
Multi-turn conversationgen_ai.conversation.idthread (thread_id metadata)sessionsession.id
Step typegen_ai.operation.name (chat, execute_tool, invoke_agent, retrieval…)run_type (llm, tool, retriever, chain, prompt, parser, embedding)observation type (generation, tool, agent, retriever, guardrail, evaluator…)openinference.span.kind (LLM, TOOL, AGENT, RETRIEVER, RERANKER, GUARDRAIL…)
Quality signal(events / app-defined)feedback (key + score)scoreannotation / eval

Sources: OTel, LangSmith, Langfuse, Phoenix.

For an agent, the trace mirrors the agent loop: an invoke_agent span containing alternating chat spans and execute_tool spans. The owner’s LangSmith example shows exactly this, with each LLM call and the calculator tool carrying its own latency, tokens and cost (notebook).

flowchart TD
  T["Trace: user turn (conversation.id = c42)"] --> A["invoke_agent support-bot"]
  A --> R["retrieval (query, top-k doc ids, scores)"]
  A --> C1["chat model-x: tokens in/out/cached, TTFT, finish_reason=tool_calls"]
  A --> X["execute_tool lookup_order: args, result, error.type"]
  A --> C2["chat model-x: finish_reason=stop"]
  A --> G["guardrail / evaluator: score"]

Synthesis: diagram assembled from the OTel operation names and the run trees shown by the sources above.

OpenTelemetry GenAI semantic conventions

The OTel GenAI conventions are the vendor-neutral vocabulary; LangSmith, Langfuse and Phoenix all ingest OTLP, so instrumenting once with them avoids lock-in. As of 2026-11 they live in a dedicated repo (open-telemetry/semantic-conventions-genai) and every GenAI-specific attribute is still “Development” stability (only reused core attributes such as error.type and server.address are Stable), so names can still change (source).

Key rules from the spec:

  • Span name {operation} {model}, e.g. chat claude-sonnet-5-5; kind CLIENT. The span covers the whole logical call including automatic retries.
  • Required: gen_ai.operation.name, gen_ai.provider.name (openai, aws.bedrock, gcp.vertex_ai…). The provider is set from the client’s view, so behind a gateway it may not be the true upstream.
  • Recommended request attributes: gen_ai.request.model, max_tokens, temperature, top_p, stop sequences, reasoning.level. Response: response.id, response.model (the resolved snapshot, which can differ from the requested alias), finish_reasons, time_to_first_chunk.
  • Usage: gen_ai.usage.input_tokens, output_tokens, cache_read.input_tokens, cache_write.input_tokens, reasoning.output_tokens, plus text/image/audio splits.
  • Correlation: gen_ai.conversation.id, gen_ai.prompt.name + gen_ai.prompt.version (which template version produced this call), gen_ai.conversation.compacted.
  • Metrics: gen_ai.client.inference.duration (histogram, seconds, buckets 0.01 to 81.92 s), time_to_first_chunk, time_per_output_chunk, and token histograms. Metrics are cheap to keep at 100% even when traces are sampled.
  • Agents and tools: create_agent, invoke_agent, invoke_workflow, plan, execute_tool, memory operations, plus MCP and A2A conventions.

Content capture is off by default. Prompts, outputs, system instructions, tool arguments/results and retrieval queries are treated as sensitive and large. OTEL_INSTRUMENTATION_GENAI_CAPTURE_CONTENT = NO_CONTENT (default) / SPAN_ONLY / EVENT_ONLY / SPAN_AND_EVENT. For production the spec recommends an upload hook that writes content to separate storage (e.g. S3) and records only gen_ai.input.messages_ref-style URIs on the span, giving separate access control and retention for user data (source).

What to log

LayerLogWhy
Identitytrace/span IDs, conversation/session ID, user or tenant ID, app version, environmentslice by customer, release, A/B arm
Prompttemplate name + version (prompt management), variables, system prompt hash, tool definitionsattribute regressions to a prompt change
Model callprovider, requested and resolved model, params (temperature, max_tokens, effort), finish_reasonlength = truncation; content_filter = refusal; model alias drift
Usageinput, output, cache read/write, reasoning tokens; computed costcost per request/feature/tenant; cache hit rate
Latencytotal duration, TTFT, time per output token, queue/retry timestreaming UX, provider degradation
RAGquery, retrieved doc IDs, scores, reranker outputseparate retrieval bugs from generation bugs
Toolsname, arguments, result size, error type, retriesbroken tools, loops, schema errors
Outcomeguardrail hits, eval/judge scores, user feedback (thumbs, edits, retries, abandonment)online quality signal

Synthesis: table combines the OTel attribute set with the notebook’s checklist (trace full prompt, context, tool calls, model and params, output, latency, tokens, cost; watch implicit feedback, sampled judge scores, guardrail hits, and drift in latency, cost and refusal rate) (notebook). For RAG, keep retrieval and generation metrics separate: a bad answer from good context is a generation bug, from bad context a retrieval bug.

Provider-specific usage fields need normalising. Anthropic reports input_tokens as only the tokens after the last cache breakpoint, with cache_creation_input_tokens and cache_read_input_tokens separate (source); OpenAI-style usage counts cached tokens inside prompt tokens (Langfuse’s example: 17,903 prompt tokens of which 17,817 cached become input: 86). Langfuse requires non-overlapping buckets (input excludes cached input) and warns that overlapping buckets double-count cost (source). In streams, Anthropic’s usage in message_delta is cumulative, so take the last value, not the sum (source).

The three main platforms (2026-11)

LangSmithLangfuseArize Phoenix
Vendor / licenceLangChain, proprietary SaaS; self-host only on EnterpriseOpen source (MIT outside ee/), cloud or self-hostArize, Elastic License 2.0 (source-available), local/Docker/Helm or hosted
InstrumentationLangSmith SDK (@traceable), zero-code for LangChain/LangGraph via LANGSMITH_TRACING=true; OTLP at <api>/otelOTel-based SDKs (Python SDK v3+, JS/TS SDK v4+); any OTel instrumentationOpenInference auto-instrumentors over OTLP
Self-host stackEnterprise onlyWeb + Worker, Postgres + ClickHouse + Redis + S3single server process (phoenix serve), Docker, Helm
Cost trackingtotal_cost per runusage x regex-matched model price, with tiers; or ingested costper-span cost = token counts x model price table (built-in, overridable); not retroactive
Evalsdatasets, evaluators, online evaluators, feedbackscores, LLM-as-judge, datasets, prompt managementLLM/code evals, datasets, experiments, playground with span replay
Notable limits25,000 runs per trace; base traces 14-day retention, extended 180-day; free tier 5k traces/monthmust flush() in short-lived processes; only generation/embedding observations carry costsource-available licence (ELv2), not OSI open source

Sources: LangSmith, Langfuse, Phoenix. See also LangSmith.

Synthesis, choosing: LangSmith is the lowest-friction choice for teams already on LangChain/LangGraph. Langfuse fits when data must stay in your VPC or you want an open-source, OTel-first store with cost analytics. Phoenix is strong for local debugging and eval-heavy workflows, especially with LlamaIndex/DSPy. In regulated enterprise deployments, instrument with OTel GenAI attributes and send through an OTel Collector, which lets you switch or dual-ship backends (Langfuse documents teams running it alongside Datadog from the same OTel instrumentation, and filtering which spans reach it in the Collector).

Failure modes and pitfalls

  • PII and secrets in traces. Prompts carry user data; capturing content by default turns the trace store into a sensitive data store. Use the OTel opt-in, redaction hooks, or external storage with its own retention (source).
  • Lost spans. Background batch exporters drop data when a serverless function exits before flush (source). LangSmith’s OTLP endpoint returns 200 before processing, and a span whose parent never arrives is dropped silently (source).
  • Broken trees. Async tasks and thread pools lose OTel context, producing orphan root spans; propagate context explicitly. Synthesis.
  • Sampling bias. Head sampling (e.g. LANGSMITH_TRACING_SAMPLING_RATE=0.1) discards rare failures at the same rate as successes. Synthesis: keep metrics at 100%, and keep all error, guardrail-hit and negative-feedback traces.
  • Convention drift. GenAI attributes are pre-stable; older instrumentations emit gen_ai.system or OpenInference llm.token_count.*. Backends map several dialects, but dashboards keyed on one name break silently (LangSmith).
  • Cardinality. Never put prompts, response IDs or user IDs in span names or metric labels; the spec keeps names low-cardinality (source).
  • Cost miscounting. Wrong price tables, overlapping token buckets, or missing reasoning/cache tokens skew per-feature cost (source).

When to use / when not

  • Turn on tracing from the first prototype; it is the fastest way to debug agent and RAG failures, and traces become the seed of your eval dataset.
  • In production, trace everything or sample with failure-biased retention, compute cost per request, and alert on regressions in p50/p95 latency, cost and refusal rate (notebook). Synthesis: also alert on error rate, length finish-reason rate and cost per tenant.
  • Do not treat observability as evaluation: traces show what happened; you still need scored datasets and LLM-as-judge checks to say whether it was good. Pair with guardrails and cost and latency optimization, which depends on these numbers.

Sources

Suggest changes →