Building Observable LLM Applications: Logging, Tracing, and Debugging in Production

Introduction

Running an LLM application in production without observability is like flying without instruments. Requests come in, responses go out—but when latency spikes, quality drops, or costs surge, you have no systematic way to find the root cause. This guide presents an observability framework for LLM applications built around three pillars: structured logging with correlation IDs, distributed tracing across model calls and tool invocations, and debuggable pipeline design that makes failures inspectable.

Unlike traditional software where a stack trace often points to the problem, LLM applications fail in subtle ways: a retrieval step returns irrelevant passages, a function call times out silently, a prompt template change degrades output quality gradually, or a hallucination passes output validation. These failures require different observability patterns than standard HTTP services.

This guide uses a fictional document-analysis service as a running example. It does not endorse a specific vendor's platform. Adapt the patterns to your stack—whether OpenTelemetry, a commercial APM, or custom logging.

The Observability Gap

Most teams start with basic logging: the user's question, the model's answer, and latency. This has three problems. First, it conflates the model call with the entire request lifecycle. A single request may trigger retrieval, multiple tool calls, a reranker, and several model invocations—each with its own latency, error modes, and quality signals. Second, it misses the structured context that makes failures diagnosable: which prompt template was used, what passages were supplied, which tool schemas were invoked, what the model returned before parsing. Third, it provides no basis for comparing configurations (chunk size, top-k, model version, prompt template) because "the answer looked wrong" is not a comparable signal.

A proper LLM observability system separates the pipeline into independently observable stages: retrieval (what passages were found and ranked?), tool execution (which tools were called with what arguments and results?), generation (what prompt was sent, what completion received, how was it parsed?), and joint behavior (does the final answer reflect the retrieved evidence?).

Core Instrumentation Patterns

Correlation IDs Across the Request Lifecycle

Every user request should receive a stable correlation ID at the application boundary. This ID propagates through every downstream call—retrieval, tool execution, model inference, post-processing—so a single logical request can be reconstructed from fragmented logs.

import uuid
from contextvars import ContextVar
from typing import Optional

correlation_id_var: ContextVar[Optional[str]] = ContextVar("correlation_id", default=None)

def get_correlation_id() -> str:
    cid = correlation_id_var.get()
    if cid is None:
        cid = str(uuid.uuid4())
        correlation_id_var.set(cid)
    return cid

Use this correlation ID in every log entry, trace span, and metric label. When a user reports "the summary was wrong," you can query for that ID and see the complete request flow: the original query, retrieved passages with scores, the prompt sent, the raw completion, the parsed result, and any validation failures.

Structured Logging for LLM Stages

Log each pipeline stage as a structured event with a consistent schema. Extract decision-relevant fields rather than dumping raw prompts into unstructured text.

import json
import logging
from dataclasses import dataclass, asdict
from typing import Any, Optional
from datetime import datetime

@dataclass
class RetrievalLog:
    correlation_id: str
    stage: str = "retrieval"
    query: str = ""
    top_k: int = 0
    retrieved_count: int = 0
    passage_ids: list[str] = None
    scores: list[float] = None
    latency_ms: int = 0
    embedding_model: str = ""
    index_version: str = ""

@dataclass
class ToolCallLog:
    correlation_id: str
    stage: str = "tool_call"
    tool_name: str = ""
    arguments: dict = None
    result_summary: str = ""
    success: bool = True
    error: Optional[str] = None
    latency_ms: int = 0
    attempt: int = 1

@dataclass
class GenerationLog:
    correlation_id: str
    stage: str = "generation"
    model: str = ""
    prompt_template_version: str = ""
    prompt_tokens: int = 0
    completion_tokens: int = 0
    raw_completion: str = ""
    parsed_success: bool = True
    parse_error: Optional[str] = None
    latency_ms: int = 0
    temperature: float = 0.0
    max_tokens: int = 0

def log_event(event: Any) -> None:
    logger = logging.getLogger("llm.pipeline")
    logger.info(json.dumps(asdict(event), default=str))

Key fields per stage:

Retrieval: query text (or hash), top-k requested, passages returned with IDs and scores, embedding model, index version, latency. This lets you detect retrieval drift and correlate retrieval quality with answer quality.

Tool calls: tool name, arguments (sanitized), result summary, success/error, latency, attempt number. Tool calls are where LLM applications most often fail silently: a function returns empty, an API times out, or a schema error is swallowed.

Generation: model identifier, prompt template version, token counts, raw completion (sampled), parse success/failure, latency, sampling parameters. The prompt template version is critical: when you change a prompt, you need to know which version produced which outputs.

Distributed Tracing with OpenTelemetry

Structured logs show what happened; traces show when and in what order, with parent-child relationships mirroring your pipeline.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("llm.application")

def trace_retrieval(query: str, top_k: int):
    with tracer.start_as_current_span("retrieval") as span:
        span.set_attribute("llm.query", query[:500])
        span.set_attribute("llm.top_k", top_k)
        # ... retrieval logic ...
        span.set_attribute("llm.retrieved_count", len(results))
        for i, (pid, score) in enumerate(zip(passage_ids, scores)):
            span.set_attribute(f"llm.passage.{i}.id", pid)
            span.set_attribute(f"llm.passage.{i}.score", score)
        return results

def trace_tool_call(tool_name: str, arguments: dict):
    with tracer.start_as_current_span(f"tool.{tool_name}") as span:
        span.set_attribute("llm.tool.name", tool_name)
        sanitized = {k: ("[REDACTED]" if "key" in k.lower() or "secret" in k.lower() else str(v)[:200])
                     for k, v in arguments.items()}
        for k, v in sanitized.items():
            span.set_attribute(f"llm.tool.arg.{k}", v)
        try:
            result = execute_tool(tool_name, arguments)
            span.set_attribute("llm.tool.success", True)
            span.set_attribute("llm.tool.result_summary", str(result)[:500])
            return result
        except Exception as e:
            span.set_attribute("llm.tool.success", False)
            span.set_attribute("llm.tool.error", str(e))
            raise

def trace_generation(model: str, prompt_template: str, prompt: str):
    with tracer.start_as_current_span("generation") as span:
        span.set_attribute("llm.model", model)
        span.set_attribute("llm.prompt_template", prompt_template)
        span.set_attribute("llm.prompt_tokens", count_tokens(prompt))
        # ... generation logic ...
        span.set_attribute("llm.completion_tokens", count_tokens(completion))
        span.set_attribute("llm.parsed_success", parsed_success)
        return parsed_result

A trace might show:

• Root: document-analysis (2.3s total)
  • retrieval (450ms, 8 passages)
  • rerank (120ms, top-3 selected)
  • tool.lookup_definition (80ms, success)
  • generation (gpt-4o, 1.1s, 450 prompt / 180 completion tokens)
  • validation (15ms, success)

This hierarchy makes latency concentrations and failure locations immediately visible.

Prompt and Completion Sampling

Logging every full prompt and completion is expensive. Implement configurable sampling:

import random
from dataclasses import dataclass

@dataclass
class SamplingConfig:
    enabled: bool = True
    sample_rate: float = 0.1
    always_sample_errors: bool = True
    always_sample_slow: bool = True
    slow_threshold_ms: int = 5000

def should_sample(config: SamplingConfig, latency_ms: int, success: bool) -> bool:
    if not config.enabled:
        return False
    if config.always_sample_errors and not success:
        return True
    if config.always_sample_slow and latency_ms > config.slow_threshold_ms:
        return True
    return random.random() < config.sample_rate

When sampling is disabled, still log structured fields (model, template version, token counts, latency, success) but omit raw prompt/completion text.

Quality Observability: Beyond Latency and Errors

Latency and error rates are necessary but insufficient. You also need to observe quality—whether outputs are actually useful.

Online Quality Signals

Instrument measurable proxies computable on every request without human review:

@dataclass
class QualityLog:
    correlation_id: str
    stage: str = "quality"
    output_format_valid: bool = True
    required_fields_present: bool = True
    citation_format_valid: bool = True
    answer_length_chars: int = 0
    citation_count: int = 0
    refusal_detected: bool = False
    passages_cited: int = 0
    passages_retrieved: int = 0
    citation_coverage: float = 0.0

• Format validity: Does output parse as expected JSON/schema?
• Required fields: Are all contract fields present?
• Citation format: Are citations in expected [doc-id] format?
• Refusal detection: Did the model refuse? (Pattern-match common phrases)
• Citation coverage: What fraction of retrieved passages were cited? Low coverage may indicate retrieval noise or generator ignoring context.

These signals aggregate into dashboards. A sudden drop in citation_format_valid after a prompt change is immediately actionable.

Offline Evaluation Pipeline

Online signals are proxies; true quality requires human or LLM-as-judge evaluation on a labeled set. Build a continuous evaluation pipeline:

@dataclass
class EvaluationJob:
    dataset_version: str
    model_version: str
    prompt_template_version: str
    queries: list[str]
    expected_passages: list[list[str]]
    expected_answers: list[str]

def run_evaluation(job: EvaluationJob) -> dict:
    results = {
        "retrieval_precision_at_k": {},
        "retrieval_recall_at_k": {},
        "faithfulness": [],
        "answer_relevance": [],
        "format_validity_rate": 0.0,
    }
    # Run retrieval evaluation
    # Run generation with gold passages (fixed-context faithfulness)
    # Run end-to-end evaluation
    return results

Schedule on every model version change, prompt template change, or retrieval config change. Compare against baseline. Alert on regressions exceeding thresholds.

Debuggable Pipeline Design

Observability is also designing pipelines so failures are inspectable. Three principles:

1. Explicit Stage Boundaries

Each stage should be a distinct function with a clear input/output contract. Avoid monolithic functions mixing retrieval, tool calling, and generation. Separate stages let you log at boundaries, replay a single stage for debugging, and swap implementations.

2. Deterministic Replay

Store exact inputs to each stage (query, retrieved passages, prompt, model parameters) so any request can be replayed deterministically. Version prompt templates, log exact passage IDs and content hashes, record model parameters.

@dataclass
class ReplayRecord:
    correlation_id: str
    timestamp: datetime
    user_query: str
    retrieval_config: RetrievalConfig
    retrieved_passages: list[PassageRecord]
    prompt_template_version: str
    rendered_prompt: str
    model_config: ModelConfig
    raw_completion: str
    parsed_result: ParsedResult
    validation_result: ValidationResult

With replay records, you can re-run generation with fixed prompts to isolate non-determinism, test new templates against historical passages, re-run validation with updated rules.

3. Failure Context Preservation

When a stage fails, preserve enough context for a human to understand what happened. If a tool call fails, log: tool name and arguments, error response, retry attempt, circuit breaker state, upstream query context. Avoid generic "tool failed" messages.

Operational Practices

Dashboard Design

Build dashboards around questions, not metrics:

1. Healthy? Error rate, latency p50/p95/p99, request volume
2. Quality degrading? Format validity, citation coverage, refusal rate, faithfulness from eval
3. Where is latency? Stacked bar by stage
4. Retrieving well? Precision/recall from eval, passages-per-query, score distribution
5. Tools reliable? Per-tool success rate, latency, error categories
6. What changed? Deployments, prompt versions, model versions, index versions overlaid

Alerting Strategy

Alert on symptoms requiring human intervention:

• Page immediately: Error rate > 5% for 5min, p99 latency > 30s, parse failure > 10%
• Ticket within 1hr: Faithfulness drops > 5pp, citation coverage drops > 10pp, retrieval recall drops > 10pp
• Weekly review: Cost per request trend, token usage by model, new error categories

Incident Response

When an incident occurs, the correlation ID is your entry point:

1. Get correlation ID from user report or alert
2. Query logs for that ID to see full request flow
3. Check trace for latency breakdown and error location
4. Replay failing stage with recorded inputs
5. Compare against eval baseline if quality-related
6. Check recent changes: prompt deployments, model rollouts, index rebuilds

Limitations

• Sampling bias: Sampled prompts may not represent full distribution. Stratify by error type, latency bucket, user tier.
• Cardinality explosion: High-cardinality attributes (query text, passage IDs) can overwhelm tracing backends. Hash or truncate them.
• Evaluation lag: Offline eval runs periodically; a regression may affect users for hours. Consider lightweight online judge for critical paths.
• Privacy: Prompts/completions may contain PII. Sanitize before logging or use separate secure pipeline.
• Model non-determinism: Even with fixed inputs, models can produce different outputs. Set seeds where supported and log the seed.

Conclusion

Observable LLM applications are built on three foundations: correlation IDs stitching fragmented logs into request flows, structured logging at explicit stage boundaries, and distributed traces revealing latency and failure topology. Quality observability requires both online proxies (format validity, citation coverage) and offline evaluation (faithfulness, relevance) run continuously against versioned datasets. Design pipelines with deterministic replay in mind—store exact inputs to each stage so any request can be re-run, inspected, and compared. These practices transform LLM applications from opaque black boxes into systems you can debug, optimize, and trust in production.

For a complementary evaluation framework, see Evaluating Retrieval-Augmented Generation Systems. For pipeline design patterns, see Production-Ready LLM Function Calling Pipelines.

Sources

• OpenTelemetry Documentation - Vendor-neutral observability framework
• OpenTelemetry Semantic Conventions for GenAI - Standard attribute names for LLM spans
• Anthropic: Building effective agents - Workflow vs agent distinction relevant to pipeline design