# LLM observability tools compared: a buyer’s guide for AI agents

Eight tools, compared on what matters once an agent is in production: how traces get in, how failures are found, how a change gets tested, and what it costs. Every claim about another vendor links to its documentation, pricing page, or an official announcement. Moda publishes this guide, so it comes last and says where it is the wrong pick.

By Mohammad Al-Rasheed, CTO & Co-Founder at Moda. Published 2026-11-12; vendor claims last verified 2026-11-12.

Canonical: https://moda.dev/llm-observability-tools

## The tools at a glance

| Tool | Best for | How traces get in | Scoring production traffic | Open source and self-hosting | Price to start |
| --- | --- | --- | --- | --- | --- |
| Langfuse | One open-source, self-hostable tool for tracing, prompts, and evals | Python and JS/TS SDKs, OpenTelemetry, 100+ integrations | LLM-as-a-judge and code evaluators you configure, with filters and sampling | MIT outside enterprise directories; free to self-host | Free cloud plan (50k units a month); Core from $29 a month |
| LangSmith | Agents built on LangChain or LangGraph; tracing, evals, and prompts from one vendor | Python and JS/TS SDKs, OpenTelemetry | Online LLM-as-a-judge evaluators at a sampling rate, including multi-turn threads | Closed source; self-hosting is an Enterprise add-on | Free (5k base traces a month, one seat); Plus $39 per seat a month |
| Braintrust | Teams that gate releases on eval scores in CI | SDKs in six languages, OpenTelemetry | Online scoring rules on spans, traces, or sessions, with sampling | Closed source (SDKs are open); data plane in your cloud on Enterprise | Free (1 GB and 10k scores a month); Pro $249 a month |
| Arize Phoenix and AX | OpenTelemetry-native tracing and evals you can run locally or self-host | OpenTelemetry with OpenInference instrumentation | Phoenix: evals you run on traces. AX: continuous online evals and monitors | Phoenix: Elastic License 2.0, free to self-host. AX: closed source | Phoenix free; AX free (25k spans a month), Pro $50 a month |
| Helicone | A gateway with per-request cost, caching, and rate limits; in maintenance mode | Proxy or gateway (a base URL change), or async logging through OpenLLMetry | None built in; stores scores you send | Apache 2.0; self-host with Docker Compose | Free (10k requests a month); Pro $79 a month |
| Datadog Agent Observability | Teams on Datadog that want LLM spans next to APM and infrastructure | Python, Node.js, Java, and Go SDKs; OpenTelemetry | Custom LLM-as-a-judge evaluators on spans, traces, or sessions, with sampling | Closed source; no self-hosted option | Free (40k LLM spans a month); Pro from $160 a month, billed annually |
| OpenLLMetry and Traceloop | Owning vendor-neutral instrumentation you can send to any backend | OpenLLMetry: OpenTelemetry instrumentation libraries | Traceloop monitors run evaluators on spans as they arrive | OpenLLMetry: Apache 2.0. Traceloop: closed, with hybrid and self-hosted options | OpenLLMetry free; Traceloop free (50k spans a month), Enterprise on request |
| Moda | Agents in production: which failures hurt users, and whether your change fixed them | Python and Node.js SDKs, OpenTelemetry, HTTP API | Built-in and custom signals on every trace; failures grouped into problems ranked by users affected | Closed source; self-hosted deployment by arrangement | Invite-only, custom pricing |

Each tool’s section below links the vendor documentation behind its row.

## What LLM observability tools do

An LLM observability tool records what your application sent to a model and what came back, plus the steps around each call: retrieval, tool calls, sub-agents, and the final answer. Each run becomes a trace you can search and open. From there the tools diverge. Some focus on inspecting traces and measuring latency, tokens, and cost. Some are built around evals. Some sit in front of the model provider as a gateway. A few try to tell you which failures matter most.

Traditional application monitoring answers whether a request succeeded and how long it took. You still need that. It misses the failures that matter most for AI agents, where every call returns successfully and the outcome is still wrong: the agent refunds a different order, invents a policy, or tells the user a task is done when it is not. Our guide to [silent agent failures](/silent-agent-failures) covers that class of failure and the evidence needed to catch it.

### Monitoring, observability, and evals

The three words get used interchangeably. They are different jobs:

- **Observability** means you can reconstruct what happened in a run after the fact.
- **Monitoring** means runs are checked continuously against known conditions, with an alert when one is breached.
- **Evals** score outputs against criteria, either offline on a fixed dataset before release or online on production traffic.

Every tool on this page does some of each. Where it puts its weight is the main thing that separates them.

## What to look for in an LLM observability tool

Start from the question you ask most often. A team iterating on prompts mostly asks what the model saw. A team with an agent in production asks which failures are hurting users, and whether the last change helped. The criteria below cover both.

### How traces get in

There are three integration styles.

- **Proxy or gateway.** It sits between your app and the model provider. Changing a base URL is often the whole setup, and caching, routing, and per-request cost come with it. The trade is an extra network hop in the request path, and visibility that stops at the model call: tool calls and your own logic need an SDK as well.
- **SDK.** It instruments your code and can capture every step of an agent, not only the model calls.
- **OpenTelemetry.** The same, through a vendor-neutral standard. The [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) and libraries such as OpenLLMetry let one instrumentation feed more than one backend.

If you might switch tools later, prefer OpenTelemetry. Moving then costs an exporter change instead of a re-instrumentation.

### How it represents an agent run

One agent run is many model calls, tool calls with arguments and results, and often many user turns. Check that the tool shows nested spans, keeps the tool results and not only the model’s text, and groups the turns of one conversation together. Session and thread views vary in depth: some group traces for display, others can score the whole conversation.

### Evals on production traffic

Most of these tools can run an LLM-as-a-judge or code evaluator on live traces. Ask three questions. Who writes the evaluators? Which traces get scored, all of them or a sample? What does scoring every trace cost at your volume? Sampling keeps judge costs down, and it also means a rare failure may show up in only a handful of scored traces.

### Finding failures you did not anticipate

Evaluators find what you thought to check for. A recurring failure nobody predicted shows up only when someone reads traces, or when the tool groups similar failures for you. Some tools cluster traces into topics or patterns. Check how much setup that takes, and whether the groups are ranked by how many users they affect.

### Testing a change before it ships

Datasets and experiments compare prompt or model versions on a fixed set of cases, and several tools run them in CI and fail a pull request on a regression. A test is only as good as its cases. Check how easily a failing production trace becomes a test case, and whether the agent’s tool calls can be replayed without writing mocks.

### Alerts

Check which conditions can trigger an alert. Latency, errors, and cost are standard; quality conditions, such as an evaluator’s failure rate or a drop in scores, vary more by tool. Then check where alerts can go (Slack, PagerDuty, email, webhooks, an issue tracker) and whether they fire per event or after a batch is evaluated.

### Prompt management

Some tools serve prompts to your application at runtime, so a prompt can change without a deploy. Others version prompts from your repository. Runtime serving is quicker to iterate on and puts the vendor’s API on your request path. Repository versioning keeps every prompt change reviewed with the code.

### Cost, latency, and tokens

Most tools compute cost from token counts and a model price table, per call and rolled up per trace. If you need cost per user, customer, or feature, check that you can attach your own metadata and group by it.

### Deployment and data control

Licenses range from MIT and Apache 2.0 to the Elastic License 2.0 and fully closed source. If you plan to self-host, check which features the self-hosted version includes, which need an enterprise license, and what it takes to run. For the hosted version, check regions and compliance reports such as SOC 2 and HIPAA.

### Pricing unit

Vendors price by traces, spans, ingested gigabytes, requests, or seats. One agent run can produce dozens of spans, so a per-span or per-gigabyte price that looks cheap for a chatbot can grow quickly. Model a week of your real traffic before comparing plans.

## The tools, one by one

The seven below recur across the published comparisons of this category, and between them they cover the main approaches: open-source platforms, eval-first tools, a gateway, an APM vendor, and open instrumentation. Each section links the vendor’s documentation behind its claims, checked on November 12, 2026. Four of them changed owners in 2026, and products here change quickly; if something is out of date, [tell us](mailto:founders@modaflows.com?subject=Correction%3A%20LLM%20observability%20tools%20guide) and we correct it.

### Langfuse

Langfuse is an open-source platform for tracing, prompt management, and evals, MIT licensed outside its enterprise directories. Traces come in through Python and JS/TS SDKs, an OpenTelemetry endpoint, and more than 100 integrations. Sessions group the traces of one conversation, and agent graphs draw multi-step workflows. On the eval side it runs LLM-as-a-judge evaluators on production traces with filters and sampling, annotation queues for human review, datasets and experiments, and a GitHub Action that runs experiments in CI. Prompts are versioned and fetched at runtime with client-side caching. Alerts on metrics and scores go to Slack, PagerDuty, webhooks, or GitHub Actions. Langfuse joined ClickHouse in January 2026 and says its licensing has not changed.

**Trade-offs.** Finding failures is work your team sets up: you choose evaluators, filters, and sampling rates before scores appear. Self-hosting is free, but some admin features, including project-level access control, audit logs, data retention policies, and server-side masking, need an enterprise license key. The free cloud plan allows two users and 30 days of data.

**Pricing.** The cloud Hobby plan is free with 50,000 units a month, where a unit is any trace, observation, or score. Core starts at $29 a month. Self-hosting the open-source version is free.

**Best for** teams that want one open-source, self-hostable tool for tracing, prompts, and evals. For a detailed comparison, see [Moda vs Langfuse](/vs/langfuse).

Sources: [Langfuse on GitHub](https://github.com/langfuse/langfuse), [LLM-as-a-judge](https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge), [experiments in CI](https://langfuse.com/docs/evaluation/experiments/experiments-ci-cd), [alerts](https://langfuse.com/docs/observability/features/alerts), [license key features](https://langfuse.com/self-hosting/license-key), [pricing](https://langfuse.com/pricing), [Langfuse joins ClickHouse](https://langfuse.com/blog/joining-clickhouse).

### LangSmith

LangSmith is LangChain’s platform for tracing, evals, prompt engineering, and deployment. It works with any framework and is closest to home for agents built on LangChain or LangGraph. Traces come in through Python and JS/TS SDKs or an OpenTelemetry endpoint that maps GenAI, OpenInference, and OpenLLMetry attributes. Threads group the turns of a conversation by a session or thread ID. Online evaluations run LLM-as-a-judge evaluators on production traces at a sampling rate you set, including multi-turn evaluators for threads. Annotation queues handle human review, and pytest and Vitest/Jest integrations run eval assertions in CI. Alerts on run count, cost, errors, feedback scores, and latency go to Slack, PagerDuty, Dynatrace, or webhooks. Prompts live in a hub with commits and tags, next to a playground.

**Trade-offs.** The platform is closed source, and self-hosting is an add-on to the Enterprise plan. Base traces are kept for 14 days unless you pay for extended retention. Your team writes the evaluators and picks the sampling rate.

**Pricing.** Developer is free with 5,000 base traces a month and one seat. Plus is $39 per seat a month with 10,000 base traces, then pay as you go. Enterprise is custom.

**Best for** teams building on LangChain or LangGraph, and teams that want tracing, evals, and prompt iteration from one vendor.

Sources: [LangSmith docs](https://docs.langchain.com/langsmith/home), [OpenTelemetry tracing](https://docs.langchain.com/langsmith/trace-with-opentelemetry), [threads](https://docs.langchain.com/langsmith/threads), [online evaluations](https://docs.langchain.com/langsmith/online-evaluations), [alerts](https://docs.langchain.com/langsmith/alerts), [self-hosted LangSmith](https://docs.langchain.com/langsmith/self-hosted), [pricing](https://www.langchain.com/pricing).

### Braintrust

Braintrust is built around evals: versioned datasets, experiments you compare run to run, scorers written in code or as an LLM judge, and a GitHub Action that posts eval results on pull requests. Traces come in through SDKs for Python, TypeScript, Go, Java, Ruby, and .NET, or through an OpenTelemetry endpoint that implements the GenAI semantic conventions. A thread view reads a trace as a conversation. Online scoring rules score production spans, whole traces, or groups of traces such as a session, with sampling. Topics classifies traces into recurring themes, and Patterns, in public preview, scans traces on a schedule for recurring problems. Alerts go to Slack, PagerDuty, or webhooks.

**Trade-offs.** Braintrust is not open source; its SDKs are. The service is hosted, and running the data plane in your own cloud requires the Enterprise plan. Alerts are evaluated in batches, so they are not instantaneous. As with Langfuse, someone on your team writes the scorers and sets the sampling rates.

**Pricing.** Starter is free with 1 GB of processed data and 10,000 scores a month, with 14-day retention. Pro is $249 a month. Enterprise is custom.

**Best for** teams whose release process gates on eval scores in CI, with datasets and experiments at the center of the workflow. For a detailed comparison, see [Moda vs Braintrust](/vs/braintrust).

Sources: [Braintrust docs](https://docs.braintrust.dev/), [score production traces](https://docs.braintrust.dev/evaluate/score-online), [run evals in CI](https://docs.braintrust.dev/evaluate/run-in-ci), [OpenTelemetry attributes](https://docs.braintrust.dev/integrations/sdk-integrations/opentelemetry/attributes), [alerts](https://docs.braintrust.dev/observe/alerts), [deployment options](https://docs.braintrust.dev/admin/deployment/index), [pricing](https://www.braintrust.dev/pricing).

### Arize Phoenix and Arize AX

Arize ships two products. Phoenix is a source-available tool for tracing, evals, experiments, and prompt iteration, built on OpenTelemetry and OpenInference instrumentation and free to self-host with no feature limits. Arize AX is the commercial hosted platform. Both accept OpenTelemetry traces, group multi-turn conversations into sessions, and show model calls, retrieval, and tool use inside each trace. Phoenix runs evaluations on production traces, datasets, and experiment results, versions prompts, and has a playground that can replay a traced call. For evals that run continuously with alerting, Phoenix’s docs point to AX: its online evals run LLM, code, or agent judges against production traces, and its monitors notify email, Slack, PagerDuty, Opsgenie, Microsoft Teams, or webhooks. Dynatrace completed its acquisition of Arize on October 1, 2026, and says AX remains available on its own and Phoenix remains open source.

**Trade-offs.** Phoenix’s Elastic License 2.0 is source-available rather than OSI-approved open source, which matters mainly if you plan to offer it as a hosted service. Continuous production monitoring and alerting mean moving to AX. The AX free tier keeps data for 15 days.

**Pricing.** Phoenix is free to self-host. AX Free covers 25,000 spans and 1 GB a month. AX Pro is $50 a month for 50,000 spans and 10 GB. Enterprise is custom.

**Best for** teams that want OpenTelemetry-native tracing and evals they can run locally or self-host, with a hosted upgrade path.

Sources: [Phoenix on GitHub](https://github.com/Arize-ai/phoenix), [Phoenix docs](https://arize.com/docs/phoenix), [self-hosting Phoenix](https://arize.com/docs/phoenix/self-hosting), [AX online evals](https://arize.com/docs/ax/evaluate/run-evals-on-traces), [AX monitors](https://arize.com/docs/ax/observe/production-monitoring), [pricing](https://arize.com/pricing/), [Dynatrace completes acquisition of Arize](https://www.dynatrace.com/news/blog/dynatrace-completes-acquisition-of-arize/).

### Helicone

Helicone is an open-source (Apache 2.0) AI gateway and logging tool. The quickest setup is changing your provider’s base URL to Helicone’s proxy. It also offers an OpenAI-compatible gateway to more than 100 models, and an async mode that logs through OpenLLMetry without sitting in the request path. Sessions group an agent’s requests using headers you set. Helicone tracks cost and latency per request, handles caching, rate limits, and retries through the proxy, manages prompts for the gateway, and sends alerts on cost, errors, latency, and tokens to email or Slack. It does not run evaluations; it stores scores you send.

Mintlify acquired Helicone in March 2026 and said it would run in maintenance mode: security updates, bug fixes, and new models, with no major feature work, and help for customers migrating to another platform. Helicone’s site now says it is part of Experiential Labs, with details to come.

**Trade-offs.** Start with the maintenance status. Beyond that, caching, rate limits, and retries need the proxy, which adds a hop to every model call, and there are no built-in evaluators. The free plan keeps data for 7 days.

**Pricing.** Hobby is free with 10,000 requests a month. Pro is $79 a month and Team is $799 a month, each with usage-based pricing beyond the included requests.

**Best for** teams that mainly want a gateway with cost tracking and caching, and are comfortable with its maintenance status.

Sources: [Helicone on GitHub](https://github.com/Helicone/helicone), [Helicone docs](https://docs.helicone.ai/), [sessions](https://docs.helicone.ai/features/sessions), [scores](https://docs.helicone.ai/features/advanced-usage/scores), [alerts](https://docs.helicone.ai/features/alerts), [pricing](https://www.helicone.ai/pricing), [Mintlify acquires Helicone](https://www.mintlify.com/blog/mintlify-acquires-helicone).

### Datadog Agent Observability (LLM Observability)

Datadog sold this product as LLM Observability; its docs and pricing page now call it Agent Observability. It traces LLM, agent, tool, retrieval, and workflow spans through SDKs for Python, Node.js, Java, and Go (experimental), or through OpenTelemetry using the GenAI semantic conventions, OpenInference, or OpenLLMetry. Sessions group a conversation’s traces. Custom LLM-as-a-judge evaluators run automatically on the spans, traces, or sessions you target, with sampling, through your own model provider account. Annotation queues, datasets, experiments, prompt management, and a playground are included. Its metrics feed Datadog dashboards and monitors, which notify Slack, PagerDuty, email, Microsoft Teams, Jira, ServiceNow, or webhooks. If you already run Datadog APM, LLM spans sit next to your service traces and infrastructure metrics.

**Trade-offs.** Closed source, with no self-hosted option. Traces are kept for 15 days by default. Evaluators call your own model provider account, so judge costs land on your model bill.

**Pricing.** Free up to 40,000 LLM spans a month. Pro starts at $160 a month on annual billing ($240 on demand) for 100,000 LLM spans, then $3.50 per 10,000. Only spans that call a model count, not tool or workflow spans. It can be bought without any other Datadog product.

**Best for** teams already on Datadog that want LLM and agent traces next to the rest of their stack.

Sources: [Agent Observability docs](https://docs.datadoghq.com/llm_observability/), [SDK reference](https://docs.datadoghq.com/llm_observability/instrumentation/sdk/), [OpenTelemetry instrumentation](https://docs.datadoghq.com/llm_observability/instrumentation/otel_instrumentation/), [custom LLM-as-a-judge evaluations](https://docs.datadoghq.com/llm_observability/evaluations/custom_llm_as_a_judge_evaluations/), [monitor notifications](https://docs.datadoghq.com/monitors/notify/), [pricing](https://www.datadoghq.com/pricing/?product=llm-observability).

### OpenLLMetry and Traceloop

OpenLLMetry is not a dashboard. It is an open-source (Apache 2.0) set of OpenTelemetry instrumentations for model providers, vector databases, and agent frameworks. It emits standard spans you can send to Traceloop or to more than 20 other backends, including Datadog, Grafana, Honeycomb, New Relic, Sentry, and an OpenTelemetry Collector. Traceloop, the company behind it, sells a hosted platform on top: monitors that run evaluators on spans as they arrive, experiments on datasets, a prompt registry, and a GitHub app that posts experiment results on pull requests. ServiceNow acquired Traceloop in 2026. Traceloop says OpenLLMetry will remain open source and that it will announce roadmap changes in advance.

**Trade-offs.** OpenLLMetry on its own gives you instrumentation, not analysis; you still need a backend to read the traces. By default it records prompts and completions in span attributes, so review its privacy settings before production. Traceloop’s free tier keeps data for 24 hours, and paid plans are quoted on request.

**Pricing.** OpenLLMetry is free. Traceloop is free up to 50,000 spans a month; Enterprise pricing is on request.

**Best for** teams that want to own their instrumentation and keep the option of sending the same traces to more than one tool.

Sources: [OpenLLMetry on GitHub](https://github.com/traceloop/openllmetry), [Traceloop monitors](https://www.traceloop.com/docs/monitoring/introduction), [trace privacy settings](https://www.traceloop.com/docs/openllmetry/privacy/traces), [pricing](https://www.traceloop.com/pricing), [Traceloop joins ServiceNow](https://www.traceloop.com/blog/traceloop-is-joining-servicenow).

## Where Moda fits, and where it does not

Moda is production monitoring and evals for AI agents. Its job is narrower than most tools on this page: find the failures in production that are hurting users, and test whether a change fixes them.

It reads every production trace end to end and runs built-in signals for frustrated users, failing tools, lazy answers, and hallucinations, meaning replies that told the user something the trace shows is false. You add your own checks in plain language as custom signals. Failures are grouped into problems ranked by how many users they affect, each with the traces behind it and the turn where it went wrong. When your team changes a prompt, model, or tool, Moda replays the change against the traces that failed and compares pass rate, latency, and tokens with the current configuration; tool calls are answered from what those traces recorded, so there are no mocks to write. Moda can also post an evals check on your pull requests. Alerts go to Slack, Linear, or email.

Traces come in through the Moda SDKs for Python and Node.js, the Vercel AI SDK and Claude Agent SDK integrations, the HTTP API, or any OpenTelemetry exporter. Moda reads OpenTelemetry GenAI, OpenLLMetry, and Vercel AI SDK attributes, so an existing OpenLLMetry setup can send traces to Moda without re-instrumenting. Moda is SOC 2 Type I and Type II compliant and HIPAA compliant.

Moda is the wrong pick, or not the only pick, when:

- **You need open source.** Moda is not open source. Self-hosted deployment is available by arrangement with the team.
- **You want to start without talking to anyone.** Access is invite-only and pricing is custom; there is no self-serve free tier.
- **You want prompts served at runtime.** Moda versions prompt files from your repository, with release labels and attribution of production calls to each version. Its SDKs do not fetch prompts.
- **You need a gateway.** Moda does not cache, rate-limit, or retry model calls. Its [Inference beta](/inference), open to a small group of teams, routes an agent’s model calls to the model your team picked and records each call as a trace; it is not a caching gateway.
- **You need infrastructure monitoring or spend budgets.** Moda records token usage from your traces and compares tokens in replays, but it is not an APM tool or a cost dashboard.

Engineers make every change; Moda monitors, finds, and tests. It can run next to a tracing or eval tool you already use. For side-by-side detail, read [Moda vs Langfuse](/vs/langfuse) and [Moda vs Braintrust](/vs/braintrust).

## How to choose

- **You want open source you can self-host:** Langfuse (MIT).
- **You can accept a source-available license to self-host:** Arize Phoenix (Elastic License 2.0).
- **Your agent runs on LangChain or LangGraph:** LangSmith.
- **Evals in CI are your release gate:** Braintrust, which is built around datasets and experiments. Langfuse and LangSmith also run experiments in CI, and Moda can post an evals check on your pull requests.
- **You already run Datadog:** Datadog Agent Observability, which puts LLM spans next to your existing APM traces.
- **You mainly need a gateway for routing, caching, and cost per request:** Helicone is built around one, but read its maintenance status before adopting it for new work.
- **You want to own your instrumentation:** OpenLLMetry, exported to whichever backend you choose.
- **Your agent is in production and you need to know which failures hurt users:** Moda.

You do not have to pick one. With OpenTelemetry instrumentation, an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) can export the same spans to two backends, so trying a second tool costs a configuration change.

## Frequently asked questions

### What is LLM observability?

LLM observability is the practice of recording each model call, tool call, and step of an LLM application as a trace, so you can inspect what happened, measure latency, tokens, and cost, and score the outputs. For AI agents it also means keeping the tool results and every turn of a conversation, because many agent failures are only visible in that context.

### What is the difference between LLM monitoring and LLM observability?

Observability is the ability to reconstruct what happened in a run. Monitoring is the continuous check: rules or evaluators run on traffic as it arrives and alert when a condition is breached. You need observability to debug a failure, and monitoring to learn about it before your users tell you.

### Which LLM observability tools are open source?

Langfuse is MIT licensed outside its enterprise directories. Helicone is Apache 2.0. OpenLLMetry is Apache 2.0, but it is instrumentation rather than a full tool. Arize Phoenix is source-available under the Elastic License 2.0 and free to self-host. LangSmith, Braintrust, Datadog, and Moda are closed source; LangSmith and Braintrust offer self-hosted or bring-your-own-cloud deployments on their enterprise plans, and Moda offers self-hosted deployment by arrangement.

### Do I need an LLM observability tool if I already use Datadog?

Datadog sells one: Agent Observability, formerly LLM Observability, which adds LLM and agent spans next to your APM traces. APM on its own shows that a call to the model succeeded and how long it took, not whether the answer was right. For that you need evaluators or signals that read the content of each trace, from Datadog or another tool.

### Can OpenTelemetry replace an LLM observability tool?

OpenTelemetry standardizes how traces are recorded and exported. It does not store, search, score, or alert on them. You still need a backend, but instrumenting with OpenTelemetry means you can change or add backends later without touching your code.

### Is an AI gateway the same as an observability tool?

No. A gateway sits in the request path to route, cache, and rate-limit model calls, and it logs each call as a side effect. It sees the model calls that pass through it, not your agent’s tool calls or business logic. Some tools, such as Helicone, combine both.

### Is Moda an LLM observability tool?

Moda overlaps with LLM observability tools: it ingests the same traces, including OpenTelemetry and OpenLLMetry spans. Its job is narrower: production monitoring and evals for AI agents. It checks every trace for failures, ranks them by users affected, and replays a change against the traces that failed before it ships.
