Switch from Braintrust

Moda vs Braintrust

For AI agents in production, Moda replaces Braintrust.

Braintrust is a well-built eval and tracing platform, built around versioned datasets, immutable experiments, and scorers, with evals that run in CI on every pull request. It raised an $80M Series B in February 2026, and its newer Topics and Patterns features look for recurring problems in production traces.

Moda at a glance

  • Every trace

    checked by built-in detections for frustration, tool failures, laziness, and hallucinations, with no scorer to write and no sampling rate to set.

    How it works
  • Users affected

    ranks every problem, so the failure that reaches the most people comes first, with the traces behind it.

    How it works
  • SOC 2 and HIPAA

    SOC 2 Type I, SOC 2 Type II, and HIPAA compliance, with self-hosted deployment available.

For teams running agents in production

  • Cardboard
  • Octolane
  • Boardy
  • Pax Historia
  • Maywood
  • Final Round AI
  • Kanu
  • Brief
  • CoreLayer
  • Autumn
  • Codewisp

Replay

A state-of-the-art eval runner for production evals. No mocks to write.

Replay a prompt or model change against the traces that failed, before it ships. Moda turns those production traces into eval cases, plays each one under your current configuration and under the change, and scores both.

  • Your production traces are the test cases

    Each case comes from a real trace. A simulated user plays the scenario your user played, so the agent faces the conversation that actually went wrong, not a prompt someone wrote for a test.

  • Tool calls are simulated for you

    When the replayed agent calls a tool, Moda simulates the tool from the calls recorded in your production traces: the answer comes back in the recorded output format, and fails the way the real tool fails. You write no mocks and no fixtures, and no real tool is called.

  • Both sides judged on every case

    A judge scores every case under Production and under Proposed. The results show the pass rates side by side, and each case opens the full replayed transcript, tool calls included, so you can check a verdict against what the agent did.

In Braintrust, an experiment runs a task you write over a dataset you build, and scorers grade the output. Any tool your agent calls runs inside that task, so it hits the real tool or a mock your team maintains.

One replay, step by step
  1. 1Build an eval set from the production traces that failed
  2. 2A simulated user replays each conversation with your agent
  3. 3Every tool call is simulated from the tool calls recorded in your traces
  4. 4A judge scores Production and Proposed, case by case

Where Braintrust leads

What you give up

Braintrust still leads on three points; if one is a hard requirement, stay on Braintrust.

  • Offline evals gated in CI

    Braintrust's versioned datasets, immutable experiments you compare run to run, and an eval action that posts results on every pull request are a mature workbench. Moda builds eval sets from production traces, with less depth on CI gating.

  • SDK and framework breadth

    Braintrust has TypeScript, Python, Go, Java, Ruby, and C# SDKs, Rust in alpha, and more than 80 listed integrations. Moda has Python and Node SDKs, plus OpenTelemetry and an HTTP API.

  • Public pricing and a free plan

    Braintrust publishes its pricing and has a free Starter plan with no credit card required; Pro is $249 a month. Moda access is invite-only with custom pricing.

Difference 1

Moda checks every trace. Braintrust scores the ones you sample.

What Moda does

Moda reads every production trace end to end and detects the failures that never throw an error: frustrated users, failing tools, lazy answers, and hallucinations. Add your own checks in plain language and get alerts in Slack, Linear, or email.

The detections are built for agent failures. Laziness is an agent that falls short of the effort the task needed. A hallucination is a claim the agent made that its recorded tool results contradict. There is no scorer to write and no sampling rate to choose, and detections typically appear within minutes of ingest.

How Braintrust approaches it

Braintrust's online scoring runs scorers on production traces that match a rule. You write them or pick them (code-based or LLM-as-a-judge), and each rule has a sampling rate. The docs recommend 1-10% for high-volume applications to manage costs. Scores count toward the plan allowance: 10k a month on Starter and 50k on Pro, then billed per 1,000.

Topics classifies traces through three built-in facets: Task, Sentiment (for example FRUSTRATED), and Issues (for example "Tool call failed"). An LLM summarizes each trace through each facet, and a daily pipeline clusters the summaries into topics. Custom facets add domain-specific patterns.

The same job in Braintrust and in Moda

In Braintrust

  1. 1Write or pick a scorer: code-based or LLM-as-a-judge
  2. 2Attach it to a rule with a sampling rate
  3. 3Each score counts toward your plan's allowance

In Moda

  • Built-in detections run on every trace
  • Built for agent failures, hallucinations included
  • No scorer to write, no sampling rate to set

Difference 2

Moda ranks failures by the users they reach. Braintrust ranks them by share of traces.

What Moda does

Failures are grouped into problems ranked by how many users they affect. Each problem shows the traces behind it, the turn where it went wrong, and related problems that share the cause.

New traces are attributed to existing problems within minutes, and a problem you marked fixed reopens if the failure comes back. Problems can be filed in Linear, where status syncs both ways.

How Braintrust approaches it

Patterns, in public preview, has Loop (Braintrust's in-product agent) sample traces on a schedule, investigate candidates, and save findings with evidence, a severity, and a suggested fix. The default schedule is daily at 9:00 AM over the last 30 days of traces.

The patterns list is ordered by severity, from High to Informational, and by overall frequency within each day. Frequency is the share of all traces in the project that a pattern affects. Topics adds a count of traces per topic, regenerated daily.

The same job in Braintrust and in Moda

In Braintrust

  1. 1Patterns investigates on a daily schedule
  2. 2Findings are ordered by severity
  3. 3Frequency is the share of all traces

In Moda

  • Failures are grouped into problems as traces arrive
  • Problems are ranked by users affected
  • A fixed problem reopens if the failure returns

Side by side

Capability by capability

The same rows appear on every Moda comparison page, including the ones where Braintrust is ahead.

Moda vs Braintrust, capability by capability
CapabilityModaBraintrustWhy it matters
Setup effortSDKs or OpenTelemetry: moda init has your coding agent add an SDK, or point an existing OpenTelemetry exporter at Moda. The built-in detections run on traces as they arrive.SDKs or OpenTelemetry for traces. Online scoring rules, scorers, and sampling rates, or the Topics pipeline, are set up per project before results appear.Time to the first useful answer about your agent, not just the first stored trace.
Failure detectionBuilt-in signals on every trace: user emotion and frustration, tool failures, laziness, and hallucinations checked against recorded tool results. Custom signals in plain language.Scorers you write or pick (code-based or LLM-as-a-judge) run on sampled production traces. Topics classifies traces by Task, Sentiment, and Issues facets, plus custom facets.Whether failures that never throw an error are caught on the traces nobody reads, and how much setup that takes.
Grouping and ranking failuresFailures grouped into problems ranked by how many users they affect, with the traces behind each one.Topics clusters facet summaries daily. Patterns (public preview) saves findings ordered by severity and the share of traces affected.Decides what your team works on first.
Testing a change before it shipsReplay a prompt or model change against traces from production and compare pass rates with the current configuration. Tool calls are simulated from the tool calls recorded in those traces, so there are no mocks to write.Playgrounds and experiments on datasets you build, including datasets built from the traces behind a pattern.Shows whether a change helps on the cases that actually failed before users see it.
Offline evals and CIReplays of a change against traces from production, run and reviewed in the dashboard. No CI gate.Versioned datasets, immutable experiments with run-to-run comparisons, and evals on every pull request through a GitHub Action or bt eval.Matters most when your release process gates on eval scores in CI.
Alert destinationsSlack, Linear, and email.Slack, PagerDuty, and webhooks, for log, time-window, and environment alerts. Patterns can also post to Slack.Alerts should land where your team already triages work.
SDKs and frameworksPython and Node SDKs; Vercel AI SDK, Claude Agent SDK, and coding agents; any OpenTelemetry source; HTTP API.TypeScript, Python, Go, Java, Ruby, and C# SDKs, Rust in alpha, and more than 80 listed integrations across providers and frameworks.Less glue code for the languages and frameworks you already run.
OpenTelemetryOTLP/HTTP trace endpoint (JSON or protobuf). Reads OpenTelemetry GenAI, OpenLLMetry, and Vercel AI SDK attributes, and groups spans across OpenTelemetry traces by moda.conversation_id or gen_ai.conversation.id.OTLP endpoint at api.braintrust.dev/otel, with an x-bt-parent header that picks the project. Maps GenAI, OpenInference, and braintrust.* attributes onto its span fields.One instrumentation can feed more than one tool, so trying a second one costs a config change.
Prompt managementPrompt files versioned from your repository, with release labels and per-version attribution of production calls. Not in the request path: the SDKs do not fetch prompts.Versioned prompts with environments, loaded at runtime with loadPrompt() or invoked through Braintrust.Decides whether prompts ship with your code or are served to your app at runtime.
Open source and self-hostingSelf-hosted deployment available; contact the team to set it up. Not open source.Not open source. Hosted SaaS on every plan; on Enterprise, the data plane can run in your cloud (BYOC) or be self-hosted, while Braintrust operates the control plane.Often required when trace data cannot leave your own infrastructure.
Security and complianceSOC 2 Type I, SOC 2 Type II, and HIPAA compliance.SOC 2 Type II and HIPAA BAAs. The SOC 2 report, BAAs, and SAML/OIDC SSO are on Enterprise.Often a procurement gate before any trial starts.
Pricing and free tierInvite-only, with custom pricing. Request access to get a workspace.Public pricing. Free Starter plan with 1 GB of processed data and 10k scores a month; Pro at $249 a month; Enterprise custom.Decides whether you can start today without talking to anyone.

Migration

Switching from Braintrust

What moves depends on how your app sends traces to Braintrust today. Spans from OpenTelemetry instrumentation move by changing where they are exported. Calls traced with Braintrust's own SDK move to the Moda SDK. In both cases, set your session and user IDs on the Moda keys so each agent run stays one trace under the right user, then import the history you already have in Braintrust.

  1. 1.Point your OpenTelemetry exporter at Moda

    If your spans come from OpenTelemetry GenAI instrumentation, OpenLLMetry, or the Vercel AI SDK's telemetry and go to Braintrust's /otel endpoint, Moda reads them as they are. Change the exporter's endpoint to https://moda-ingest.modas.workers.dev/v1/traces, replace its headers with Authorization: Bearer and your Moda API key, and drop the x-bt-parent header. If you send spans through Braintrust's span processor, replace it with a standard OTLP exporter configured the same way. Moda does not read braintrust.* attributes such as braintrust.input or braintrust.metadata, so spans that carry their content only there need GenAI attributes or the Moda SDK.

    Moda ingest accepts uncompressed request bodies only, and the OpenTelemetry spec leaves the default compression to each SDK, so set OTEL_EXPORTER_OTLP_TRACES_COMPRESSION=none. If you export through a Collector, which compresses by default, replace the Braintrust exporter with one for Moda and set compression: none on it.

    Exporter environment variables
    export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://moda-ingest.modas.workers.dev/v1/traces"
    export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Bearer YOUR_MODA_API_KEY"
    export OTEL_EXPORTER_OTLP_TRACES_COMPRESSION="none"
  2. 2.Replace Braintrust SDK tracing with the Moda SDK

    Braintrust's TypeScript and Python SDKs, with initLogger, wrappers such as wrapOpenAI, and the Node.js auto-instrumentation hook, log spans to Braintrust through their own logger. Their OpenTelemetry support is a separate package, so for these calls there is no exporter to repoint.

    Replace those calls with the Moda SDK for Python or Node.js. One init call instruments the OpenAI and Anthropic clients, including streamed responses and tool calls, without changes to your provider code.

  3. 3.Set your session and user IDs on the Moda keys

    Moda groups spans into one trace by moda.conversation_id, or gen_ai.conversation.id, across OpenTelemetry traces, and attributes them to a user by moda.user_id. A span with neither conversation key is grouped by its own OpenTelemetry trace. Set moda.conversation_id to the session or conversation ID your app already tracks and moda.user_id to its user ID. With the Moda SDK, that is moda.conversation_id and moda.user_id in Python. In Node.js, import withContext from moda-ai and wrap each request in withContext(sessionId, userId, async () => { /* your existing agent code */ }). This isolates concurrent requests. Reserve the process-wide Moda.conversationId and Moda.userId setters for scripts handling one session at a time.

    The importer reads the user ID from user_id or userId in a trace's Braintrust metadata. Set moda.user_id to that same value, and a person's traces from before and after the switch land under one Moda user. Session IDs do not carry across the same way: the importer prefixes each imported trace ID with braintrust: and the Braintrust project ID, so a conversation that continues after the switch starts a new Moda trace.

  4. 4.Import your Braintrust history

    In Settings, open Integrations, then Import traces. Choose your Braintrust region, US or EU, and paste a Braintrust API key; Moda stores it encrypted and removes it when the import completes or is cancelled. Then pick a project and a date range of up to 30 days, and run another import for each earlier window. An owner or admin starts the import, and can pause, resume, or cancel it. It runs in the background, so you can leave the page.

    Each Braintrust trace whose root span was created in the range arrives with all of its child spans: parent links, timing, inputs and outputs, models, token counts, and errors, with the original Braintrust record kept on every span. Traces whose metadata carries the same session_id, sessionId, or conversation_id become one Moda trace, and a user_id or userId becomes the Moda user. Datasets, experiments, prompts, and attachments stay in Braintrust.

Full guideMigrating from Braintrust

Questions

Frequently asked questions

Do I need Braintrust anymore?

Not to monitor and test AI agents in production. Moda takes your traces, runs its detections on every one, groups failures into problems ranked by users affected, and replays a prompt or model change against the traces that failed. Keep Braintrust only if you need something it does that Moda does not: a CI gate on curated datasets, an SDK for a language Moda does not cover, or prompts served to your app at runtime.

What do I lose by switching?

Braintrust's dataset and CI eval workflow, its SDK and integration breadth, and public pricing with a free plan. You also leave behind runtime prompt loading and PagerDuty and webhook alerts. Your trace history comes with you: Moda imports Braintrust traces with their spans, sessions, users, and errors.

Can I bring my Braintrust history to Moda?

Yes. In Settings, open Integrations, then Import traces, choose your Braintrust region, paste a Braintrust API key, and pick a project and a date range of up to 30 days. Each trace arrives with all of its spans, traces that share a session ID become one Moda trace, and user IDs carry over. The step-by-step guide is at moda.dev/blog/migrating-from-braintrust.

Do I have to re-instrument my app?

Not if it already emits OpenTelemetry GenAI, OpenLLMetry, or Vercel AI SDK spans: change the exporter's endpoint, API key, and compression. Calls traced with Braintrust's own SDK move to the Moda SDK, which instruments the OpenAI and Anthropic clients with one init call. Either way, set moda.conversation_id to your session ID.

Braintrust has Topics and Patterns. How is Moda different?

Topics classifies traces by task, sentiment, and issue and regenerates its topics daily, and Patterns (public preview) runs scheduled investigations that rank findings by severity and the share of traces affected. Moda runs detections built for agent failures, including hallucinations checked against recorded tool results, and groups them into problems ranked by how many users they affect, with new traces attributed within minutes.

Why not write scorers in Braintrust for frustration, laziness, and hallucinations?

You can. In Braintrust you write and maintain each scorer, choose a sampling rate, and use your plan's score allowance. Moda's detections for these failures run on every trace without that setup, and they read the whole agent run. You can still add your own checks in plain language as custom signals.

How is replay in Moda different from Braintrust experiments?

A Braintrust experiment runs a task you write over a dataset you curate, so any tool your agent calls runs for real or behind a mock you maintain. In Moda you replay a prompt or model change against traces from production: a simulated user plays each conversation, every tool call is simulated from the tool calls recorded in your traces, and a judge compares pass rates with the current configuration. There are no mocks to write, and your team decides what ships.

What does Moda cost?

Moda access is invite-only and pricing is custom. Request access at moda.dev/signup, or email founders@modaflows.com for a quote. Braintrust publishes its pricing and has a free Starter plan.

Can I self-host Moda, or is it open source?

Moda offers self-hosted deployment; contact the team at moda.dev/contact to set it up. Moda is not open source, and neither is Braintrust, which offers BYOC and self-hosted data planes on Enterprise.

What security compliance does Moda have?

SOC 2 Type I, SOC 2 Type II, and HIPAA. Braintrust is SOC 2 Type II compliant and signs HIPAA BAAs on its Enterprise plan.

Replace Braintrust with Moda

Infrastructure for catching AI agent mistakes in production and turning them into evals.

Read the guide: Migrating from Braintrust