# Hallucinations in AI agents: how to detect and prevent them

An agent hallucination usually looks ordinary: a tool call that never ran, an ID off by one character, a number the tool never returned, or “done” after a failed write. Here is how to find each one in a trace and how to stop the next one.

By Mohammad Al-Rasheed, CTO & Co-Founder at Moda. Published 2026-11-12.

Canonical: https://moda.dev/agent-hallucinations

The interactive example on the web page uses constructed booking traces, not customer data. The user asks the agent to cancel booking BK-4471 if the fee is under $50, and the agent always answers “Done. The fee for BK-4471 was $35, so I cancelled it.” Five recorded traces change the verdict: no tool calls at all (claimed tool use with none recorded), both calls made for BK-4417 (wrong arguments), a $75 fee in the tool result (the answer contradicts the result), a failed cancellation (a false claim of success), and matching calls and results (the answer matches the trace, for that narrow check only).

## What is an agent hallucination?

An agent hallucination is an output or action that the agent’s evidence does not support. The evidence is whatever the agent had at that turn: the user’s request, earlier turns, tool results, retrieved documents, and the state of systems it can read.

When a chatbot hallucinates, you get a wrong sentence. An agent’s hallucination can also be an action. It might call a tool with an invented argument, write to the wrong record, or report a finished task that never finished. The run completes, no exception fires, and the answer sounds specific. That specificity is part of the problem.

Hallucinations are one class of agent failures, and often a silent one. [Silent failures in AI agents](/silent-agent-failures) covers the wider class, including wrong-record actions and dropped constraints that involve no invented fact.

For production work, use a definition you can check: a claim or action is hallucinated when the trace contradicts it or contains nothing that supports it. That is narrower than “a reviewer found it surprising,” but a trace can settle it.

## Four types of agent hallucinations

The example at the top of this page keeps one answer fixed and changes the trace behind it. Each trace exposes a different type. A [2025 survey by Lin et al.](https://arxiv.org/abs/2509.18970) organizes agent hallucinations by where they occur in the agent’s workflow. The four below are the ones you can check directly against a recorded trace.

### 1. Tool-call fabrication

The agent says it looked something up, ran code, or sent a message, and the trace has no such call. Sometimes the model names a tool that does not exist, or describes output no tool produced.

This happens in frontier models. Transluce’s investigation of a pre-release OpenAI o3 model found that it [frequently fabricated actions it had taken](https://transluce.org/investigating-o3-truthfulness), including code it claimed to have run, and elaborately justified those claims when users questioned them. The authors caution that their prompts covered a narrow slice of possible inputs.

**What exposes it:** match every action the answer describes to a recorded tool call. A claim of tool use with no matching call is a finding on its own. You do not need to know the correct answer to flag it.

### 2. Wrong tool arguments

The right tool runs with the wrong input: an ID one character off, a date in the server’s time zone, a parameter the model made up, dollars where the API expected cents. Every call succeeds and the schema validates, yet the task fails.

Look-alike records make this more likely. When the context holds both BK-4471 and BK-4417, the model can bind the wrong one. In [TheAgentCompany](https://arxiv.org/abs/2412.14161) benchmark, an agent that could not find the right colleague in a chat tool took a shortcut: it renamed a different user to the intended user’s name. [Xu et al.](https://arxiv.org/abs/2412.04141) split tool hallucination into choosing the wrong tool and misusing the right one. Both show up as arguments that do not match the task.

**What exposes it:** compare arguments with the entities in the user’s request and in earlier tool results. Flag identifiers that appear nowhere in the context, and identifiers that nearly match the requested one.

### 3. Invented facts

The agent states a value its evidence does not contain or directly contradicts: a fee, a policy, a date, a stock level. The worst version contradicts a tool result from the same turn, because a reviewer skimming the answer has no reason to doubt it.

Empty lookups are a common trigger. Asked about a record that does not exist, a model often answers anyway. [ToolBeHonest](https://arxiv.org/abs/2406.20015) tested whether models recognize when their tools cannot solve a task. In that 2024 evaluation, Gemini-1.5-Pro scored 45.3 out of 100 and GPT-4o scored 37.0, and the most common error was misjudging whether the task was solvable at all.

**What exposes it:** compare each material value in the answer with the tool results and documents available at that turn, including which record each value belongs to. A number that appears somewhere in context is not support if it belongs to a different record.

### 4. False claims of success

The agent reports that a task is done when the write failed, is still pending, or never ran. This is the type most likely to cost money, because the user stops checking.

Agent benchmarks avoid trusting the narration. [τ-bench](https://arxiv.org/abs/2406.12045) grades retail and airline agents by comparing the database state at the end of a conversation with the goal state. Graded that way, even gpt-4o completed fewer than half of the tasks in the paper’s evaluation.

**What exposes it:** read the tool status, not the agent’s summary of it. Treat “accepted” and “completed” as different states, and keep the outcome unknown when the trace has no completion evidence.

## Why agents hallucinate

### Guessing is rewarded

[Kalai et al.](https://arxiv.org/abs/2509.04664) argue that language models hallucinate because training and evaluation reward guessing over acknowledging uncertainty. A benchmark graded right-or-wrong gives nothing for “I don’t know,” so a guess always scores at least as well. OpenAI’s [write-up of the paper](https://openai.com/index/why-language-models-hallucinate/) shows the trade-off on SimpleQA. gpt-5-thinking-mini abstained on 52% of questions and answered 26% wrongly. o4-mini abstained on 1%, was slightly more accurate (24% against 22%), and answered 75% wrongly.

Agent prompts can recreate the same incentive. “Always give the customer the fee” asks the model to guess when the lookup fails.

Stronger reasoning is not a reliable cure. [Yin et al.](https://arxiv.org/abs/2510.22977) found that training models for stronger reasoning increased tool hallucination in proportion to the gains in task performance.

### Missing or look-alike evidence

A lookup returns nothing, or returns a near-duplicate, and the model fills the gap with the closest thing in context. Anthropic reports that [resolving arbitrary alphanumeric IDs into meaningful names](https://www.anthropic.com/engineering/writing-tools-for-agents) significantly improved Claude’s precision in retrieval tasks by reducing hallucinations. IDs that differ by one character are hard for people too.

### Tool design

Ambiguous tool names, overlapping tools, optional parameters with unclear defaults, and opaque error codes all leave the model to guess. Large tool sets make selection harder. [LongFuncEval](https://arxiv.org/abs/2505.10570) measured function-calling performance drops of 7% to 85% as the number of available tools grew.

### Errors that get smoothed over

A tool returns an error, a timeout, or a partial result, and the agent writes a confident answer anyway. Sometimes the framework swallows the error before the model sees it. Sometimes the model sees it and summarizes it as success. If you captured the tool result, the contradiction is sitting in the trace.

### Context rot

Long runs accumulate tool output, retries, and abandoned plans. As the input grows, the model uses it less reliably, and the evidence that should anchor an answer competes with everything else in the window. The next section covers it in detail.

## How to prevent context rot in agents

Context rot is the decline in how reliably a model uses its input as that input grows. [Chroma’s 2025 report](https://www.trychroma.com/research/context-rot) evaluated 18 models and found that “models do not use their context uniformly.” Performance grew increasingly unreliable as input length grew, even on simple tasks. On LongMemEval, the same questions scored significantly higher with a focused prompt of about 300 tokens than with the full input of about 113,000 tokens. Earlier, [Liu et al.](https://aclanthology.org/2024.tacl-1.9/) showed that models used information at the start and end of a long input better than information in the middle.

Length hurts even when retrieval is perfect. [Du et al.](https://arxiv.org/abs/2510.05381) measured drops of 13.9% to 85% as inputs grew, including when the irrelevant tokens were replaced with whitespace. For tool use specifically, LongFuncEval found a 7% to 91% drop in retrieving answers from tool responses as the responses got longer, and a 13% to 40% drop as multi-turn conversations got longer.

A bigger context window does not prevent it. Chroma’s full LongMemEval input of about 113,000 tokens is a fraction of the million-token windows offered by models it tested, such as GPT-4.1 and Gemini 2.5 Pro, and focused prompts still scored significantly higher across the model families tested.

### What context rot looks like in a trace

Quality falls late in long runs. The agent asks again for information the user already gave, returns to a plan it abandoned, ignores a constraint from the third turn, or uses a tool result from twenty calls ago instead of the latest one. Each of these can produce the four types above. A wrong argument late in a long run is often an old identifier resurfacing.

### Keep the context small and relevant

Anthropic’s [guide to context engineering](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) frames the goal as the smallest possible set of high-signal tokens. In practice:

- Return compact tool results. Keep the fields the agent needs, paginate large lists, and let the agent fetch more on request. Claude Code caps tool responses at 25,000 tokens by default.
- Clear old tool results once the agent has acted on them.
- Load documents just in time through a search or read tool, instead of preloading everything that might matter.

### Compact without losing constraints

When a run gets long, summarize the earlier turns into a compact state and continue from it. The summary is where constraints go missing, so give it a fixed structure: the user’s goal, explicit constraints such as limits, time zones, scope, and permissions, decisions made, entities in play with their IDs, and open work.

Test a compaction by running it on a long trace and checking that every constraint survives. In a coding agent, the same idea is a compaction command, or a fresh session that starts from a written plan.

### Move state out of the transcript

Keep durable state where the agent can read it on demand: a notes file, a task record, or your application database. Hand self-contained subtasks to sub-agents with clean contexts. Anthropic describes sub-agents that return condensed summaries of 1,000 to 2,000 tokens to the agent that called them.

Enforce the constraints that matter in code. A $50 limit checked by the cancel endpoint does not rot.

### Measure context rot in production

Break your hallucination rate down by run length, tool-call count, or input tokens. If findings cluster in long runs, context is a likely cause, and you have a set of traces to test a trimming or compaction change against before it ships.

## How to detect hallucinations in production traces

Offline evals cover the failures you anticipated. Production traces are where you find the rest, and detection there starts with a record complete enough to judge.

### Capture the evidence

Record the user’s request, prior turns, every tool call with its arguments and full result, errors and retries, the final answer, and the prompt, model, and tool versions. A detector can only compare what you captured. If tool results are truncated, a correct claim can look unsupported. Mark truncation explicitly instead of letting it look like an empty result.

### Check claims against the trace

Most agent hallucinations are a mismatch between two parts of the same trace, so start with deterministic checks:

- **Claimed actions against recorded calls.** Every “I checked,” “I sent,” or “I cancelled” needs a matching tool call.
- **Arguments against the request.** Identifiers, amounts, and dates passed to tools should trace back to the user or to an earlier tool result.
- **Stated values against tool results.** A fee, balance, or status in the answer should match the result it came from, for the same record.
- **Reported outcomes against tool status.** “Done” requires a completed status, not an accepted request or an error.

This is the check behind the example at the top of the page:

```typescript
type LookupCall = { tool: 'get_booking'; bookingId: string; feeCents: number }
type CancelCall = { tool: 'cancel_booking'; bookingId: string; status: string }
type Claim = { bookingId: string; feeCents: number; cancelled: boolean }

function checkClaim(
  request: { bookingId: string },
  claim: Claim,
  calls: Array<LookupCall | CancelCall>,
) {
  const lookup = calls.find((c): c is LookupCall => c.tool === 'get_booking')
  const cancel = calls.find((c): c is CancelCall => c.tool === 'cancel_booking')
  if (!lookup || (claim.cancelled && !cancel)) return 'no-recorded-call'
  if (calls.some((c) => c.bookingId !== request.bookingId)) {
    return 'wrong-arguments'
  }
  const sameBooking = claim.bookingId === lookup.bookingId
  if (!sameBooking || claim.feeCents !== lookup.feeCents) {
    return 'contradicts-result'
  }
  if (claim.cancelled && cancel?.status !== 'cancelled') return 'false-success'
  return 'supported'
}
```

It assumes the answer has already been turned into structured fields. A parser can do that for tightly formatted answers. Free text needs a model, and that extraction step needs the same auditing as any other detector. The function also covers only what the trace recorded. It cannot tell you whether the booking system applied the right fee policy.

### Use judges for what rules cannot decide

Some claims need reading. Did the answer address the question? Is a summary faithful to a long document? Did the agent skip a step it should have taken? An LLM judge can help, with known limits. [Zheng et al.](https://arxiv.org/abs/2306.05685) found that GPT-4 judges agreed with human preferences over 80% of the time on chat responses, and also documented position, verbosity, and self-enhancement biases. That agreement was measured on chat preferences, not on agent actions.

Give the judge the evidence and a narrow question, such as “Is the fee in the answer supported by the get_booking result?” A broad “is this answer good?” is hard to debug. Require the judge to cite the span that supports or contradicts the claim. Calibrate it against human-reviewed traces that include correct refusals and passing controls, and count false alarms as well as catches. For free-text answers with no tool evidence, consistency methods such as [SelfCheckGPT](https://arxiv.org/abs/2303.08896) sample several responses and treat claims the model cannot reproduce as likely inventions.

### Group findings and rank them

A single hallucination is hard to prioritize. The same invented fee across many traces points to a cause: a lookup that fails for one booking type, a tool that returns cents where the prompt expects dollars, a compaction that drops the limit. Group findings by behavior, rank them by users affected, and fix the cause, not each trace.

### Measure the detector

Report precision, the share of flagged traces that were real hallucinations, and recall on a reviewed set, the share of real ones it caught. Track coverage and unknowns too: which traces were evaluated, and which lacked the evidence to decide. The [silent failures guide](/silent-agent-failures#measure-whether-your-detector-is-useful) covers detector measurement in more depth.

## How to prevent hallucinations in agents

No single technique eliminates hallucinations. Anthropic’s own guidance on [reducing hallucinations](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations) says its techniques do not eliminate them entirely. Layer them, and keep detection running after release.

### Ground answers in tool results

- Instruct the agent to state values only from tool results or provided documents, and to say plainly when a lookup returned nothing.
- Give it explicit permission to decline or ask, and make declining cheap: an escalation tool, a not-found response the agent can use. Kalai et al. suggest stating a confidence threshold in the prompt, so a wrong answer costs more than an abstention.
- For long documents, have the model extract supporting quotes first and answer from them, and retract any claim it cannot back with a quote. Both techniques are in Anthropic’s guidance.
- Remove instructions that assume an answer exists, such as “always give the customer a number.”

### Make tool schemas strict and specific

- Use structured outputs or strict function calling where your provider supports them. [Anthropic](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) and [OpenAI](https://developers.openai.com/api/docs/guides/structured-outputs) both offer modes that constrain tool inputs to your JSON schema. That guarantees the shape of the arguments, and nothing about whether the values are right. OpenAI’s documentation tells you to validate any business rules the schema does not express.
- Use enums for closed sets and required fields for anything consequential. Put the unit in the parameter name: `amount_cents`, not `amount`.
- Return identifiers, readable names, and status in every result. Return actionable errors, such as “No booking BK-4417 for this customer. Ask the user to confirm the booking ID,” instead of opaque codes.
- Keep the tool set small and distinct. Merge overlapping tools and namespace related ones.

### Verify before you report

- Before a consequential write, check its arguments against the request and the user’s constraints in application code, not in the prompt.
- After the write, read authoritative state or wait for the completion event before telling the user it is done.
- For factual answers, add a verification pass. [Chain-of-Verification](https://aclanthology.org/2024.findings-acl.212/) drafts an answer, plans verification questions, answers them independently so the answers are not biased by the draft, then revises. It reduced hallucinations across the list-based, question-answering, and long-form tasks the authors tested.

### Turn failures into evals

Every confirmed hallucination is a test case. Keep the smallest trace that reproduces it, add a passing control and an ambiguous case, and run your prompt, tool, or model change against all of them before release. Replays that touch real systems need a sandbox or recorded tool responses.

Then check that the failure did not move somewhere else. A prompt that stops invented fees by refusing every fee question has not solved the task.

### Use judges, carefully

Judges belong in both places: as graders in your evals, and as online checks on production traffic. Keep each one narrow, tied to evidence, and calibrated, and recalibrate when you change the judge model. Where the contract is explicit, prefer a deterministic check.

## How Moda detects hallucinations in production

Moda is infrastructure for catching AI agent mistakes in production and turning them into evals. It reads production traces end to end and detects frustrated users, failing tools, lazy answers, and hallucinations.

Its hallucination signal is deliberately narrow. It flags a reply that told the user, as fact, something the trace shows is false: a status, what exists, or what was done. Each finding quotes the agent’s words beside the evidence that contradicts them, such as a tool result or something the user said, and a second check keeps the finding only when it can quote the part of the trace that shows the reply false. The signal runs once a trace goes quiet, about an hour after its last message. Of the four types above, that covers invented facts and false claims of success whenever the trace shows the opposite.

It does not flag a claim just because the trace cannot confirm it. A missing receipt does not show a statement is false, so an answer that describes a tool call with no recorded call is not a hallucination finding on its own. The trade is fewer findings, each one you can check by reading two quotes.

For the types the built-in signal leaves out, write a custom signal in plain language, for example: “The agent tells the user a booking is cancelled, but no cancel_booking call succeeded for the booking ID the user gave.” Moda searches recent traces for matches so you can confirm the check before it runs on every trace. Moda has no separate context rot signal. Compare your findings across short and long traces to see whether length is a factor.

Findings roll up into problems: recurring failures grouped and ranked by how many users they affect, each with the traces behind it and the turn where it went wrong. They can also drive alerts to Slack, Linear, or email. The dashboard’s Signals page has no Hallucinations tab. From a terminal or a coding agent, `moda detections list --family=hallucination` lists recent findings, and `moda traces findings <trace_id>` shows everything Moda flagged on one trace. The [CLI docs](https://docs.moda.dev/cli/overview) cover setup.

When your team changes a prompt, tool, or model, replay the change against the exact traces that failed and compare pass rate, latency, and tokens. Your team reviews the result and decides what ships.

### Start with one agent

Connect traces through the [Moda SDKs, OpenTelemetry, or HTTP ingestion](https://docs.moda.dev/ingestion/overview). Pick one task where a wrong answer costs something. Read the hallucination findings and the problems they belong to, then add a custom signal for the type your agent is most exposed to.

[Request a Moda workspace](/signup) to try it with your agent. Access is invite-only and requests are reviewed by the Moda team. To walk through a production trace first, [book a demo](https://cal.com/team/moda/demo-meeting?overlayCalendar=true), or [explore Monitor](/monitor).

## Questions engineers ask

### Can you eliminate hallucinations in AI agents?

Not with any single technique. Grounding, strict schemas, and verification make hallucinations rarer. Enforcing consequential rules in code makes them less harmful. Checking production traces makes them visible. OpenAI’s researchers also reject the claim that hallucinations are inevitable, because a model can abstain when it is uncertain. The agent has to be allowed, and rewarded, to do that.

### Does RAG prevent hallucinations?

It reduces answers with no source. It does not stop the agent from using the wrong retrieved record, ignoring the retrieved text, or misreading a long result. Check claims against what was retrieved, and include record IDs in the comparison.

### Do strict tool schemas stop wrong arguments?

They stop malformed arguments: a missing field, a string where a number belongs, a value outside an enum. They do not stop a valid argument that is wrong, such as a real booking ID for the wrong booking. That check needs the request and the context.

### Does a bigger context window prevent context rot?

No. Context rot is about how well a model uses its input, not whether the input fits. In Chroma’s tests, focused prompts of about 300 tokens beat full prompts of about 113,000 tokens, far below the limits of million-token models. Keep contexts small, compact long runs, and enforce important constraints in code.

### Should an LLM judge decide whether an answer is hallucinated?

Use a judge for decisions a rule cannot make, give it the evidence, and calibrate it against human review. Prefer a deterministic comparison where one exists. A judge reading the same misleading context can make the same mistake as the agent.

### What is the difference between a hallucination and a silent failure?

A hallucination is output or action the evidence does not support. A silent failure is any wrong outcome that does not trigger an alert. Many hallucinations are silent failures. Many silent failures, such as a correct action on the wrong day, involve no hallucination. See [silent failures in AI agents](/silent-agent-failures).

## A hallucination review checklist

- Capture tool arguments and full results, not only the final answer.
- Match every action the answer describes to a recorded tool call.
- Compare tool arguments with the entities in the request.
- Compare stated values with the tool results for the same record.
- Treat the tool status as the outcome, not the agent’s summary of it.
- Break findings down by run length to find context rot.
- Calibrate judges against reviewed traces, including passing controls.
- Turn every confirmed hallucination into an eval case and test your change against it.

Pick one trace your agent reported as a success, and check each claim in its final answer against the calls behind it.
