Contents
In How to build a multi-agent system we said: "instrument every handoff before adding agents." The technology that powers that "instrumentation" in production is AI observability. It makes visible what your LLMs and agents are actually doing in production — which tools they call, what they retrieve, where they fail, and how much it costs.
Unlike ordinary app monitoring, AI has a nasty trait: a request can return "200 OK in 50ms" and still confidently lie (hallucinate). In other words, it can be fast and up while the quality is broken. This article walks beginners through the 3 pillars of observability, how it differs from evaluation (evals), the metrics worth watching, and the major tools.
Visualize the "execution tree" of one request
— A trace records inputs, tool calls, retrieval, and outputs as spans
* Tool traits and concepts in this article are quoted from public materials and official docs (as of June 2026). Tool evaluations vary by use case and version — read them as directional.
1. What is AI observability?
AI observability means making the behavior of LLMs and AI agents in production observable from the outside. For each request, you record "which model was called with what prompt, which tools and searches were used, what was returned, and how long and how much it cost" — so that when something breaks, you can trace back to the cause.
The decisive difference from ordinary app monitoring: traditional monitoring checks "is it up, is it fast?" But AI can respond normally and quickly while the content is wrong. Most AI failures are not infrastructure failures but "quality failures" — hallucinations, weak retrieval, unsafe answers, incomplete tasks, poor tool use, and regressions after a prompt change.
So AI needs dedicated observation. Especially in multi-agent systems, failures appear within multi-step causal chains, not at the individual call level. "Which step went wrong, and why" only becomes visible once you capture the full-session trace.
2. The 3 pillars: traces, metrics, logs
Observability is traditionally described in terms of three pillars. The same holds for AI, and the industry standard OpenTelemetry (GenAI conventions) lets you handle all three with a vendor-neutral common schema.
Traces
Record one request's execution path as a tree of spans. You see how LLM calls, tools, retrieval, and reasoning chains flowed. The star of AI observation.
Metrics
Aggregate latency, cost, token count, error rate, and throughput as numbers. Track trends per model/agent.
Logs
Detailed records of individual events — full prompts, error details — the evidence for deep investigation.
OpenTelemetry's GenAI conventions record prompts, model responses, token usage, tool/agent calls, and provider metadata in a standard format. This means you aren't tied to one vendor and can feed AI traces into existing monitoring backends like Datadog or Grafana.
3. How it differs from evaluation (evals)
The thing beginners most often confuse is the difference between "observability" and "evaluation (evals)." They're different things, and they only matter as a set.
🔭 Observability
Shows "what happened": traces, cost, latency, errors. Easy to measure, but on its own it can't tell you "is the answer correct?"
✅ Evaluation (evals)
Measures "is the answer good?": accuracy, groundedness, safety. Explicit evals are required — this is the guardian of quality.
The crux: "cost and latency are easy to measure, but answer quality can't be known without explicit evaluation." That's why 2026's leading tools don't just show traces — they score outputs, alert on quality degradation, and feed insights back into development. Observation and evaluation are two wheels of the same cart.
4. What to watch: key metrics
The indicators to track on a dashboard split broadly into "operational" and "quality".
⚙️ Operational (easy to measure)
- Cost: token billing per request
- Latency: response time (varies widely by input)
- Token usage: catch bloated prompts early
- Error rate / throughput: per model/agent
🎯 Quality (needs evaluation)
- Hallucination: confident but false claims
- Groundedness: most critical for RAG — is it backed by retrieved sources?
- Safety: PII leakage, harmful output
- Task completion / correct tool use
Among quality metrics, in RAG (retrieval-augmented generation) "groundedness (faithfulness)" is the most critical indicator: is the answer actually supported by the retrieved documents, or did the model invent it? Hallucination detection commonly uses LLM-as-a-judge (have an AI score it), semantic similarity, and groundedness scores.
5. Major tools compared
Here are the representative AI observability tools of 2026. Many are moving toward combining tracing and evaluation in one place.
| Tool | Traits | Best for |
|---|---|---|
| LangSmith | Great fit with LangChain/LangGraph. Detailed tracing + eval + monitoring. Low overhead. | LangChain-based production |
| Langfuse | Open source. Self-hostable, so you needn't send data to an external SaaS. | Self-hosting / strict data needs |
| Arize Phoenix | Strong at RAG debugging. Good at visualizing retrieval quality. | RAG investigation/improvement |
| MLflow | Centralizes the whole GenAI lifecycle. | End-to-end dev-to-ops |
| AgentOps | Specialized in monitoring autonomous agents. Multi-step session tracking. | Agent operations |
| OpenTelemetry | The standard. Vendor-neutral; connects to Datadog/Grafana, etc. | Integration with existing monitoring |
Source: various tool comparisons and official info (June 2026). Traits are tendencies; evaluations vary by use case and version.
When unsure, it's safe to start capturing traces in an OpenTelemetry-compliant way. You avoid vendor lock-in and can reselect a tool later. If you use LangChain, LangSmith is an easy entry point; if you want to keep data in-house, Langfuse.
6. How to start, and why it matters for agents
No need to overthink it — start small. What matters is putting observation in place before you ship to production.
Capture traces
Record LLM calls, tools, and retrieval as spans. OpenTelemetry-compliant makes switching later easy.
Visualize operational metrics
Dashboard cost, latency, and tokens. Set alerts on anomalies.
Connect evaluation (evals)
Score production traces for quality and detect degradation. Combine evals with guardrails.
Especially in multi-agent systems, observation isn't "nice to have" — it's essential. Because failures hide in multi-step chains, without a full-session trace you'll never know "where and why it broke." Put observation in before adding agents — that's the rule. It also helps with early detection of security incidents.
Summary
AI observability is the operational foundation that "makes production AI visible." Let's recap.
Key takeaways
- 🔭 Makes production AI's internals visible. Three pillars: traces, metrics, logs.
- ⚠️ 200 OK can still lie. Most AI failures are quality failures, not infrastructure.
- 🔁 Observe + evaluate together. Traces for "what," evals for "is it good."
- 🛠️ Tools: LangSmith/Langfuse/Phoenix/MLflow/AgentOps. The standard is OpenTelemetry.
- 🤖 Essential for agents. Multi-step failures are only visible in a full-session trace.
"Fast and up" isn't enough to trust AI. It's production-grade only when you can see inside and measure quality. Start by capturing traces in an OpenTelemetry-compliant way, then connect evals. For building agents, see here; for safety design, guardrails.
FAQ
Q. How do observability and evaluation (evals) differ?
A. Observability shows "what happened" (traces, cost, latency); evaluation measures "is the answer good." Since a response can be fast and up yet wrong, the basic approach is to use both as a set.
Q. Can't I just use a regular app-monitoring tool?
A. It can measure uptime and speed, but not AI-specific quality like hallucination or groundedness. AI needs dedicated observation (or the OpenTelemetry GenAI conventions) that records prompts, tokens, and tool calls.
Q. Where do I start?
A. It's safe to start capturing traces in an OpenTelemetry-compliant way. You avoid vendor lock-in and can reselect tools like LangSmith or Langfuse later. Then visualize cost and latency, and finally connect evaluation.
Q. Why is it especially important for agents?
A. Agent failures appear not in a single call but within multi-step causal chains. Without a full-session trace you can't pinpoint "which step went wrong and why," making debugging impossible.
