LLM observability is the engineering discipline of instrumenting large language model applications to understand and improve how they perform: tracing requests through a pipeline, evaluating output quality, attributing cost, and measuring latency. It is the delivery-side counterpart to the security-side visibility that AI observability provides.
- Engineering-side instrumentation of LLM applications for quality and cost
- Core concerns are traces, evaluations, cost attribution, and latency
- Owned by ML platform and AI engineering teams rather than security
- Distinct from AI observability, which asks a governance question instead
Why is LLM observability important?
LLM observability matters because language model applications fail in ways conventional monitoring does not detect. A service can return a 200 response, well inside its latency budget, while the content of that response is wrong, off-policy, or subtly degraded from last week. Uptime tells you nothing about output quality.
That gap widens as applications compose. A retrieval step, a routing decision, several model calls, and a tool invocation each contribute to a final answer, so when quality drops the useful question is which step moved. Tracing is what makes that answerable, and evaluation is what makes quality measurable rather than anecdotal.
Cost is the third pressure, and it is the one that usually secures budget for the tooling. Token spend scales with usage and with prompt design, and without per-request attribution a team cannot tell which feature or which customer is generating it. Gartner expects 40% of enterprise applications to carry task-specific AI agents by the end of 2026, which multiplies both the quality surface and the spend.
What is LLM observability?
LLM observability is the instrumentation and analysis of language model applications for the purpose of improving them. Its subject is application behavior and its audience is the team that builds and operates that application.
Four concerns define the discipline. Tracing follows a single request through every step of a pipeline – retrieval, prompt construction, model call, tool invocation, post-processing – so failure can be localized. Evaluation scores output against reference answers, rubrics, or model-based judges, which turns quality into something trackable across releases. Cost attribution assigns token spend to features, users, or tenants. Latency measurement identifies which step in a chain is slow.
The distinction from AI observability is worth stating plainly, because the terms are used interchangeably and should not be. LLM observability asks whether the application is working well. AI observability asks whether AI activity across the enterprise is known and permitted. Same infrastructure, different question, different owner.
Types of LLM observability
The discipline divides by what is being measured, and the categories map onto different tooling.
Trace-based observability captures the execution path of individual requests, including inputs and outputs at each step. It is the foundation, because everything else needs somewhere to attach. Evaluation-based observability scores output quality, either offline against a fixed dataset before release or online against live traffic afterward. Cost and usage observability tracks token consumption and spend attribution by model. Performance observability covers latency, throughput, and error rates in the conventional sense.
A fifth category has emerged for agents specifically: multi-step trajectory analysis, which evaluates whether an agent's sequence of decisions was sound rather than whether any single response was well formed. That one overlaps with the security framing, since a trajectory is also where unauthorized behavior becomes visible.
LLM observability & Onyx
Onyx is not an LLM evaluation platform, and the two disciplines are complementary rather than competing. Most enterprises will run engineering-side observability tooling for quality and cost alongside AI Observability for security and governance visibility.
Where they meet is the traffic itself. The Onyx AI Gateway routes LLM and agent traffic with centralized policy enforcement and queryable per-request audit, so the same interactions an engineering team traces for quality are also inspected against policy and preserved as an attributable record. Teams that already run a gateway can keep it, since BYO gateway configurations are supported. The engineering team keeps its evaluation workflow; the security team gets a control point rather than a copy of the logs.
