Your agents should never make
the same mistake twice.
Engineer better agents from production data: trace failures, build evals, compare models.
Observe
Harness
Validate
Overview
109K
Conversations
Conversations in the last 7 days14.8%
Signal Rate
Conversations with a signal99.2%
Reviewed
Conversations Moda has reviewedNeeds Attention
1
Fabricated data after tool call
6735users affected
8912conversations
2
Order lookup times out under load
4102users affected
5367conversations
3
Benign already-exists conflicts treated as failures
2418users affected
3106conversations
Tracked metrics
Broken tool calls
+12.2 pts compared to 7 days ago
37.5%
Task completion
+8.4 pts compared to 7 days ago
86.3%
Results
Pass
229 of 244 cases
94%
Fail
15 of 244 cases
6%
Change vs old (%)
Latency
−21.1%
Token usage
−21.5%
Success score
+10.5%
−30%
−15%
0
+15%
+30%
For teams running agents in production
Find recurring failures and how they affect users
Moda reads every production trace, groups recurring failures into problems, and ranks them by users affected.
Problem detected
Fabricated data after tool call
6735users affected
Fabricated data after tool call
When a lookup comes back empty, the agent fills in details from a look-alike record instead of saying it found nothing, so users get order numbers and dates that don't exist.
6735
Users affected
244
Linked traces
Linked tracesRelevancy
Quoted a tracking number the order lookup never returned
High
Gave a delivery date copied from another customer's order
High
Confirmed a refund total with no matching invoice
High
Listed opening hours for a store the search didn't find
Medium
Understand the underlying context and patterns
Open a problem to see the traces behind it and the related problems that share its cause, ranked so you know what to fix first.
High priority
Fabricated data after tool call
6735users affected
High priority
Order lookup times out under load
4102users affected
Low priority
Agent retries a call that succeeded
342users affected
Low priority
Store search returns stale hours
518users affected
Low priority
Refund total skips the shipping fee
207users affected
Medium priority
Delayed response from API
1284users affected
Test changes against production
Replay a prompt or model change against the problem's traces and compare pass rate, latency, and token usage before it ships.
Proposed change
Run comparisonModel
Opus 5.5
Version
Prompt category
New prompt
You are the support agent for {{company}}.
Answer only from what your tools return.
If a lookup comes back empty, say you could not find it.
Never fill in an order number, date, or amount.
Results
Pass229 of 244 cases
94%
Fail15 of 244 cases
6%
Change vs old (%)
Latency
−21.1%
Token usage
−21.5%
Success score
+10.5%
−30%
−15%
0
+15%
+30%
Tailored to your workflow
Custom signals
Judge
Agent gives time of day and attaches wrong time zone
3.2%
−0.8 pts vs last 7 days
Hit Rate
Hits out of every trace judged1.7K
Hits
Traces the judge matched54.5K
Traces Judged
Traces the judge gave a verdict onRecent hits
Told a customer in Berlin the store opens at 9:00 PT
Gave a Sydney customer the current time in UTC
Booked a 3pm EST callback for a customer in London
Said it was 6pm using the server's time zone
Monitors
Trigger
Agent gives time of day and attaches wrong time zone
Slack alert
Alert
Mention
When the hit rate is above
for an hour
Conditions
Right where you work
