Your agents should never make the same mistake twice.
Infrastructure for catching AI agent mistakes in production and turning them into evals.
Observe
Harness
Validate
Overview
109K
Conversations
Conversations in the last 7 days14.8%
Signal Rate
Conversations with a signal99.2%
Reviewed
Conversations Moda has reviewedNeeds Attention
1
Fabricated data after tool call
6735users affected
8912conversations
2
Order lookup times out under load
4102users affected
5367conversations
3
Benign already-exists conflicts treated as failures
2418users affected
3106conversations
Tracked metrics
Broken tool calls
+12.2 pts compared to 7 days ago
37.5%
Task completion
+8.4 pts compared to 7 days ago
86.3%
Results
Pass
229 of 244 cases
94%
Fail
15 of 244 cases
6%
Change vs old (%)
Latency
−21.1%
Token usage
−21.5%
Success score
+10.5%
−30%
−15%
0
+15%
+30%
For teams running agents in production
Monitor every trace
Moda reads every production trace end to end and detects the failures that never throw an error: frustrated users, failing tools, lazy answers, and hallucinations. Add your own checks in plain language and get alerts in Slack, Linear, or email.
Problem detected
Fabricated data after tool call
6735users affected
Fabricated data after tool call
When a lookup comes back empty, the agent fills in details from a look-alike record instead of saying it found nothing, so users get order numbers and dates that don't exist.
6735
Users affected
244
Linked traces
Linked tracesRelevancy
Quoted a tracking number the order lookup never returned
High
Gave a delivery date copied from another customer's order
High
Confirmed a refund total with no matching invoice
High
Listed opening hours for a store the search didn't find
Medium
Find the mistakes they keep repeating
Failures are grouped into problems ranked by how many users they affect. Each problem shows the traces behind it, the turn where it went wrong, and related problems that share the cause.
High priority
Fabricated data after tool call
6735users affected
High priority
Order lookup times out under load
4102users affected
Low priority
Agent retries a call that succeeded
342users affected
Low priority
Store search returns stale hours
518users affected
Low priority
Refund total skips the shipping fee
207users affected
Medium priority
Delayed response from API
1284users affected
Test every change before it ships
Replay a prompt, model, or tool change against the exact traces that failed and compare pass rate, latency, and tokens. Your team reviews the result and decides what ships.
Proposed change
Run comparisonModel
Opus 5.5
Version
Prompt category
New prompt
You are the support agent for {{company}}.
Answer only from what your tools return.
If a lookup comes back empty, say you could not find it.
Never fill in an order number, date, or amount.
Results
Pass229 of 244 cases
94%
Fail15 of 244 cases
6%
Change vs old (%)
Latency
−21.1%
Token usage
−21.5%
Success score
+10.5%
−30%
−15%
0
+15%
+30%
Tailored to your workflow
Custom signals
Judge
Agent gives time of day and attaches wrong time zone
3.2%
−0.8 pts vs last 7 days
Hit Rate
Hits out of every trace judged1.7K
Hits
Traces the judge matched54.5K
Traces Judged
Traces the judge gave a verdict onRecent hits
Told a customer in Berlin the store opens at 9:00 PT
Gave a Sydney customer the current time in UTC
Booked a 3pm EST callback for a customer in London
Said it was 6pm using the server's time zone
Monitors
Trigger
Agent gives time of day and attaches wrong time zone
Slack alert
Alert
Mention
When the hit rate is above
for an hour
Conditions
Right where you work
