# Harness engineering > Harness engineering is the discipline of improving an AI agent by engineering everything around the model — prompts, tools, skills, evals, memory — using evidence from production. The anatomy of the harness is the parts list; the engineering is the loop that turns production traces into verified harness improvements, with the model treated as a fixed component. Canonical: https://moda.dev/harness-engineering Updated: 2026-09-04 The term is showing up in engineering discussions around OpenAI and Anthropic for a concrete reason: the flagship agent products of both labs — Codex, Claude Code — are models inside heavily engineered harnesses, and the people building them keep finding that harness decisions move agent quality as much as model choice does. The same model ships with different tools, context management, and guardrails and becomes a noticeably different product. Once you see that, the question changes from which model is best to who is engineering our harness — and for most teams running an agent in production, the honest answer is no one, or everyone, in scattered prompt edits after each incident. Harness engineering names that scattered work and makes it a practice with a loop, an evidence standard, and a definition of done — it is to agents what site reliability engineering was to infrastructure. This guide covers what the practice consists of, how it differs from the things it is most often confused with — harness anatomy, prompt engineering, and observability — and how the loop runs when it works. ## What harness engineering covers The harness is every part of an agent system that is not model weights: prompts, tools, skills, evals, memory, orchestration. Harness engineering — written out in full as agent harness engineering or AI harness engineering when the bare term could be confused with the Harness.io CI/CD platform or a hardware test harness — is the deliberate improvement of those parts against production evidence. Concretely, the recurring work looks like this: - Maintaining a current map of the harness — which agents exist, with which prompts, tools, and skills — so changes are made against the system as it is, not as someone remembers it. - Attributing production failures to the component responsible: this tool schema invites wrong arguments, this prompt section no longer covers what users ask, this memory lookup retrieves the wrong context. - Tightening tool definitions and routing so the model's cheapest interpretation of a schema is also the correct one. - Building and refreshing eval sets from real traffic, so the gate a change must pass resembles the traffic it will meet. - Packaging recurring competence into skills instead of letting every fix become another paragraph in an ever-growing system prompt. - Verifying each change against replays of production traces before it ships, and reverting when the numbers say to. Source: - Osmani, Agent Harness Engineering: https://addyosmani.com/blog/agent-harness-engineering/ ## OpenAI harness engineering: the agent-first Codex essay OpenAI harness engineering entered the vocabulary through one essay: Harness engineering: leveraging Codex in an agent-first world (February 2026), the lab's first-party account of shipping an internal product of roughly a million lines of code over five months with zero manually written lines. It is the right primary source — read it before any secondhand summary, this page included. The engineers' job was not writing code; it was designing the environment: an AGENTS.md kept as a map rather than a manual, architecture enforced by custom linters whose error messages coach the agent, application logs and metrics made directly legible to the agent, and recurring cleanup agents that pay down drift daily. Its statement of the role change is the cleanest in the literature: humans steer, agents execute. What the essay describes — and what it honestly does not cover — draws the line for everyone else. OpenAI ran the loop on its own product with a dedicated team, bespoke tooling, and Codex itself writing the fixes. Teams who are not OpenAI still need the same loop against their own production traffic, without staffing a lab-scale harness team to run it. That is the practice Moda packages as a platform: map the harness from the repository, detect behavioral failures across production traces, attribute each one to the prompt, tool, skill, eval, or memory component responsible, and verify proposed changes against replayed traffic before they ship. The essay supplies the discipline's proof of concept; the platform is how the loop runs outside the lab. Source: - Lopopolo, Harness engineering: leveraging Codex in an agent-first world (OpenAI): https://openai.com/index/harness-engineering/ ## Anthropic harness engineering: harness design for long-running agents Anthropic harness engineering shows up on the lab's engineering blog from the design side, in two posts worth reading first-hand. Harness design for long-running application development (March 2026) splits long-horizon work across a planner, a generator, and an evaluator that drives the running application the way a user would, with a sprint contract negotiated before any code is written — built on the finding that a model is a poor judge of its own work, so evaluation has to be engineered into the harness as a separate component. Scaling Managed Agents (April 2026) adds the operational corollary: harnesses encode assumptions about what the model cannot do on its own, those assumptions go stale as models improve, and the interfaces around the harness should be designed to outlive any particular harness. Moda is not Anthropic, and this page is not a competitor to those posts — they are design-time guidance from inside the lab that builds Claude, and they set the frame the rest of the industry works in. What they leave open is the production side for teams running agents on top of these models: which of your harness's encoded assumptions is failing this week, which component is responsible, and whether the fix actually held. That is the loop Moda runs as harness engineering infrastructure — production traces in, attribution to the responsible component, verified harness improvements out, each checked against replayed traffic before it ships. Anthropic's posts tell you how to design the harness; the platform tells you, from your own traffic, how to keep it correct. Sources: - Anthropic, Harness design for long-running application development: https://www.anthropic.com/engineering/harness-design-long-running-apps - Anthropic, Scaling Managed Agents: Decoupling the brain from the hands: https://www.anthropic.com/engineering/managed-agents ## How to do harness engineering for coding agents Coding agents are where most teams meet harness engineering first, and the essays that define the practice — OpenAI's Codex account, the martinfowler.com guides-and-sensors framework — are written about coding agents for a reason: the same model ships as a mediocre or an excellent coding agent depending on the harness around it. How to do harness engineering for coding agents is the standard loop from this page, specialized for the artifacts a coding agent actually touches: - Map the coding agent's harness as it exists in the repository: AGENTS.md and system prompts, tool and MCP definitions, SKILL.md files, the linters and CI checks the agent sees, eval suites, memory. The parts list is covered in the agent harness guide; the map has to track the repo as it is, not someone's memory of it. - Observe complete sessions, not isolated tool calls. A coding agent's failure rarely lives in one span — it lives in the arc of a session: the plan, the edits, the test runs, the retries. Capture full traces of production sessions. - Attribute each failure to the component responsible: an AGENTS.md section that no longer matches the codebase, a tool schema that invites wrong arguments, a skill whose steps drifted, a missing lint or test the agent needed as a sensor — or context rot on long sessions, where accumulated diffs and tool output degrade the model mid-task. - Improve as guides and sensors: sharpen the instruction or skill that steers the agent before it acts (a guide), and add the linter, test, or check that catches the failure after it acts (a sensor). Coding agents are unusually harness-friendly here because their outputs are verifiable diffs. - Verify against replayed sessions before shipping: re-run the failing sessions under the changed harness, compare against the current one, and hold the change to the standard two-number gate — targeted failures fixed, held-out sessions unregressed. Two things make coding agents the easiest place to run this loop well. First, the outcome is mechanically checkable — a diff compiles, passes tests, and survives review, or it does not — so verification can be automated to a degree conversational agents cannot match. Second, the harness surfaces are already files in the repository (AGENTS.md, tool configs, SKILL.md, CI), so every improvement lands as a reviewable, revertible commit. The predictable failure mode is the inverse: teams benchmark model choice obsessively and let AGENTS.md rot, when the harness file is the part they control. The published playbooks split by who you are. If you are using a coding agent — Codex, Claude Code, Cursor, or a custom SWE agent — on your own codebase, the martinfowler.com essay's outer-harness framing applies: your job is the guides-and-sensors layer on top of the vendor's built-in harness, tightened every time the agent slips. If you are running a coding agent as a product or an internal platform, the loop above is the job, and it needs production evidence at every step — which sessions fail, which harness file is responsible, whether the fix held. That evidence layer is what Moda provides as a harness engineering platform: production session traces in, attribution to the responsible harness component, and replay-verified improvements out. Sources: - Böckeler, Harness engineering for coding agent users (martinfowler.com): https://martinfowler.com/articles/harness-engineering.html - Moda, What is an agent harness (pillar guide): https://moda.dev/agent-harness - Moda, Context rot and how to prevent it (pillar guide): https://moda.dev/context-rot ## Harness anatomy vs harness engineering The anatomy of an agent harness — which components exist and why — is well covered: LangChain's anatomy guide works backwards from the model to derive the parts (tools and their descriptions, filesystem and sandbox, orchestration, memory, hooks and middleware), and framework harnesses like Deep Agents ship those parts assembled. Knowing the anatomy tells you what a harness is made of. It does not tell you which of your components is failing in production this week, what to change, or whether the change helped. That is the line between anatomy and engineering. Anatomy is the parts list; harness engineering is the discipline that operates on the parts — production traces in, verified harness improvements out. A team can know every component cold and still ship prompt edits by intuition after each incident, which is anatomy without engineering. The distinction is not adversarial: the anatomy writing and the engineering practice need each other. LangChain's own open-problems list for harness building includes agents that analyze their own traces to identify and fix harness-level failure modes — which is precisely the loop this page describes, and the one Moda runs as a product: detect behavioral failures across the trace population, attribute each to the responsible component, propose the diff, and verify it against replayed traffic before it ships. Source: - Trivedy, The Anatomy of an Agent Harness (LangChain): https://www.langchain.com/blog/the-anatomy-of-an-agent-harness ## Harness engineering vs prompt engineering Prompt engineering is one component of the job, and treating it as the whole job is how teams end up with a two-thousand-line system prompt that nobody can safely edit. The difference is scope and evidence. Prompt engineering iterates on one artifact, usually against intuition and a handful of test cases. Harness engineering treats the whole assembly as the design surface and asks, for each observed failure, which component is actually responsible. Many failures that look like prompt problems are not: an agent that picks the wrong tool needs a sharper tool description, not a longer prompt; an agent that forgets what the user said needs a memory fix; an agent that regresses on a change needs an eval that would have caught it. Much of the writing on harness engineering vs prompt engineering frames it as a succession — prompt engineering (iterating on instructions) gave way to context engineering (curating everything the model sees at inference time), which is giving way to harness engineering (engineering the full system around the model, plus the loop that improves it). The succession framing is roughly right about scope: each stage subsumes the previous one, and prompts and context are both harness components. It is wrong if it implies the earlier skills stopped mattering. Prompt and context work remain jobs inside the harness; harness engineering is the discipline that decides, from production evidence, which of those jobs actually needs doing this week. The same goes for the broader label agent engineering: building the agent is the construction job, and harness engineering is the improvement loop that runs once the agent is serving traffic. It is an engineering practice with infrastructure behind it, not a new job title. A useful test: if your improvement process cannot conclude that the prompt is fine and the tool schema is broken, you are doing prompt engineering. The prompt is where every fix lands when it is the only surface you look at. ## Harness engineering vs observability Observability is the input, not the practice. Traces, logs, and dashboards tell you what the agent did; they do not tell you what to change, and they do not verify that a change worked. Agent observability tooling stops at visibility: a trace shows every span green while the user leaves frustrated, because behavioral failures — wrong tool arguments that return 200, context quietly lost, goals silently drifted — are invisible at the span level. Observability is necessary; everything downstream of it is where the improvement happens. Harness engineering is that downstream: detection of behavioral failures across the trace population, attribution to the responsible component, a proposed change, and verification against real traffic before shipping. The output is harness changes from traces, not another dashboard about the traces. The relationship to continual learning is the same from the other direction: continual learning is the research name for systems that improve after deployment without retraining, and harness engineering is what the practice looks like when the improvements land in the harness rather than the weights. Observability feeds it; research named it; the engineering is the job. Source: - Chase, Continual learning for AI agents (LangChain): https://www.langchain.com/blog/continual-learning-for-ai-agents ## Silent agent failure A silent agent failure is a failure that produces no error: every span in the trace succeeds, every tool call returns 200, the response arrives on time — and the user still leaves with the wrong outcome. It is the behavioral failure class traces miss, and the reason the previous section's line between visibility and engineering exists at all. The recurring forms: - Wrong-but-successful tool calls: the arguments satisfied the schema, the API returned 200, and the result never served the task. - Context silently lost: the agent re-asks for information the user already gave, or drops a mid-context constraint without acknowledging it. - Goal drift: the session ends politely and confidently on a task the user never asked for. - Skill and instruction misfires: a SKILL.md stays cold on the task it was written for, or an instruction is followed into a stale step — no exception, wrong behavior. - Misroutes: the wrong model serves the request, the response is valid, and nothing in the trace says the routing was the failure. Silent agent failures are invisible one trace at a time because every infrastructure signal reports healthy — the failure exists in the gap between what the user needed and what the agent did, which no status code carries. They become visible at population scale: cluster production sessions by behavior, and the wrong-but-successful patterns show up as recurring groups with names, rates, and exemplar traces. That detection step is covered in depth in Moda's clustering write-up; the behavioral failure entry in the glossary defines the class. Detection is half the job. Once a silent failure cluster is named, the rest is the loop this page describes: attribute the cluster to the harness component responsible — the chooser section below maps the signatures to prompt, tool, skill, eval, or memory — propose the fix, and verify it against replays before shipping. That is what Moda does with silent agent failures: clusters them across the production trace population and turns them into verified harness improvements, rather than another alert about a trace that already looked green. Sources: - Moda, Behavioral failure (glossary): https://moda.dev/glossary#behavioral-failure - Moda, How clustering finds agent failures (blog): https://moda.dev/blog/clustering ## Should I buy observability or a harness engineering platform? Should I buy observability or a harness engineering platform? If your agent is in production, decide by what you need to be left holding after a bad week. Observability tooling leaves you holding evidence: the trace of every run that went wrong, and a human job of deciding what to change. A harness engineering platform leaves you holding harness changes from traces: a prompt edit, a tightened tool schema, a new skill, an expanded eval set — each attributed to a detected failure pattern and verified against replayed production traffic before it ships. The decision at a glance: | Decision factor | Observability | Harness engineering platform | | --- | --- | --- | | What you buy | Visibility: traces, logs, dashboards of what the agent did | The improvement loop: failure detection, component attribution, replay verification | | Question it answers | What happened on this run? | Which harness component do I change, and did the change work? | | Failure classes it sees | Hard failures: exceptions, timeouts, 4xx/5xx | Behavioral failures too: wrong-but-successful tool calls, lost context, goal drift | | After a bad week you hold | Evidence, and a human job of deciding what to change | Verified harness improvements: prompt, tool, skill, and eval diffs, replay-checked before shipping | | Buy it when | Nobody can see what the agent did | Traces exist, but improvement is still manual and unverified | The two are not substitutes, so the question is really about where your gap is. If nobody can see what the agent did, start with instrumentation — OpenTelemetry-based tracing is a solved problem and largely free. If you already have traces and the improvement work is still manual — someone reading transcripts, guessing at a prompt edit, shipping it unverified — more dashboards will not shorten that path, because the dashboard was never the missing piece. The gap is the middle of the loop: population-scale failure detection, component-level attribution, and replay-based verification. That middle is what a harness engineering platform sells. Buying observability gets you better questions; buying harness engineering gets you verified harness improvements — and it subsumes the ingest layer rather than replacing your existing tracing, since production traces are its raw material either way. Moda sits on exactly one side of this table: it is harness engineering infrastructure — the detection, attribution, and replay-verification layer under the harness you own — not an observability product, and not an LLM router. It never sits in your request path; it turns the traces your existing instrumentation already emits into verified harness improvements. ## The harness engineering loop Run as a practice, harness engineering is a five-step loop. Each step is real engineering work, and most teams currently do steps one and five with a human reading transcripts in between. - Map: build and maintain a versioned model of the harness from the repository — agents, prompts, tools, skills, evals, and how they connect. - Observe: capture complete production traces, not sampled spans. OpenTelemetry-based ingest keeps the instrumentation vendor-neutral. - Attribute: detect behavioral failures and emergent intents across the whole population, and route each to the harness component responsible. - Improve: turn the attribution into a concrete diff — a prompt edit, a tool schema fix, a new skill, an expanded eval set. - Verify: replay the change against real production traffic and compare before shipping; revert if a fixed cluster cost you a different one. This is the loop Moda runs as a product. The harness map syncs from GitHub on every push to the default branch, with citations back to source files. Production traces are analyzed for behavioral failures and use-case clusters, each attributed to a harness component. Proposed changes are checked in a playground that compares production and proposed configurations over replayed traces before anything ships. Source: - Moda docs, Harness overview: https://docs.moda.dev/harness/overview ## Verified harness improvements: the unit of progress The output of harness engineering is not a report. It is a verified harness improvement: a concrete, reviewable change to a prompt, tool, skill, eval, or memory component, proposed from production evidence and checked against replays of real traces before it ships. Verified is the load-bearing word. Most teams already make harness changes; almost none can say whether last month's changes helped, because the change was checked against intuition and a handful of test cases rather than the traffic it was written to fix. Verification closes that gap: replay the affected production traces under the proposed configuration, compare against the current one, and let the numbers decide — including whether fixing one cluster of failures quietly broke a different one. This is also what makes the practice auditable in a way weight updates are not. A verified harness improvement is a diff a reviewer can read, with the evidence attached: the failure pattern that motivated it, the component it changes, and the replay comparison that justified shipping it. It survives a model swap and reverts with a rollback. ## How to evaluate whether a harness change actually fixed the failure To evaluate whether a harness change actually fixed the failure, run a replay protocol — not a complaint counter. Most teams evaluate harness changes by absence of complaints, which measures nothing; the protocol below is what separates harness engineering for agents from prompt tinkering: - Pin the failure cluster first: the set of production traces exhibiting the failure the change targets, identified before the change ships — not reconstructed afterward from memory. - Replay that cluster under the proposed configuration and compare against the current one, trace by trace: did the wrong tool call become the right one, did the lost context survive, did the drifted goal hold. - Run the same comparison on held-out traffic the change was not written for, so a fix for one cluster cannot quietly buy a regression in another. - Make the ship decision from the two numbers together — targeted cluster improved, held-out behavior held — and revert on the same evidence if post-ship traffic disagrees. - Keep watching the cluster after shipping: the failure that motivated the change is now a named pattern, and its recurrence rate is the lasting measure of whether the fix held. The same acceptance rule is emerging in the research literature: Co-Harness, which co-evolves harnesses and model weights, accepts a harness diff only if validation rollouts improve the targeted failure mode without regressing held-out behavior — precisely the two-number gate above, applied by an automated critic. In production the discipline is identical; the difference is that the rollouts are replays of your own traffic. Moda runs this protocol as the verification step of its loop: every proposed improvement ships with the before-and-after comparison on the traces that motivated it, plus the regression check on everything it might have touched. Source: - Chen et al., Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agents (arXiv:2607.22688): https://arxiv.org/abs/2607.22688 ## How to transfer routing policy from frontier to open source models A recurring harness engineering job sits at the model layer: the agent runs well on a frontier model, the team wants some or all of its traffic on an open-source model — for cost, latency, or control — and the routing policy that decides which tasks go to which model has to move without losing task performance. The instinct is to treat this as a model problem, or to shop for a router product. It is neither. It is the standard loop — traces in, attribution in the middle, verified harness changes out — applied to model selection behavior: - Cluster production traces into a task taxonomy, so the routing policy has a denominator: transfer is decided per task, not for the agent as a whole. - Replay each task's real traces through the open-source candidate and compare against the frontier baseline, so the gap is measured on your traffic rather than public benchmarks. - Attribute each regression to the harness component responsible. Many frontier-to-OSS gaps close with a prompt rewrite, a tightened tool schema, or an added skill — not a bigger model. - Move routing task by task: tasks where the open model holds route to it now; tasks where it regresses stay on the frontier model until a verified harness change closes the gap. - Verify every routing change against replayed production traffic before it ships, and watch the affected tasks after — the same evidence standard as any other harness change. This is also the honest answer to standing up a task trained model router without losing performance. The router itself — a rule or classifier trained on your task distribution — is a component you build and own inside your harness. What keeps it from losing performance is the harness engineering underneath it: the task taxonomy that gives it categories, the per-task replay evidence that trains and audits it, and the replay gate that catches the task where the cheaper model quietly regresses before your users do. To be explicit about what Moda is in this picture: Moda is a harness engineering platform, not an LLM router — not a marketplace like OpenRouter, not a gateway like LiteLLM, and not a router SKU. It never sits in your request path and does not resell inference. Moda is the harness infrastructure underneath a model router you build for your own tasks: it produces the verified improvements for agent model routing — the task evidence, the attributed harness fixes, the replay verification — while the routing rule stays in your code, reviewable and revertible like any other harness component. Source: - Moda, What is an LLM router (pillar guide): https://moda.dev/llm-router ## How to build an LLM router on production traces A router built on public benchmarks inherits the benchmark's task distribution instead of yours. Building an LLM router on production traces flips the order: the routing rule is the last artifact of the harness loop, not a product you shop for first. The build runs in five steps: - Cluster production traces into a task taxonomy, so the router's categories come from what your traffic actually contains rather than a benchmark's idea of tasks. - Run per-task model comparisons on replayed traces: for each task cluster, replay real traces through each candidate model and compare against the incumbent, so the routing table is built from evidence on your own traffic. - Write the routing rule as harness code you own — a rule or a classifier trained on your task distribution, mapping task to model — versioned, reviewable, and revertible like any other harness component. - Replay-verify the router before it ships: run the routed configuration against replayed production traffic and compare with the unrouted baseline per task, including regression checks on the tasks whose model did not change. - Keep the loop running after launch: new task clusters appear, models change, and every routing update passes the same replay gate the first version did. This is why building on traces beats buying a generic router. A hosted router or gateway can balance cost and latency, but it cannot know your task taxonomy, cannot replay your traffic, and cannot attribute a per-task regression to the prompt or tool schema that actually caused it. The router that survives contact with production is the one whose categories, evidence, and shipping gate all come from production. Moda's role in this build is the infrastructure, never the router: it produces the task taxonomy from your traces, the per-task model comparisons on replayed traffic, and the verification gate every routing change passes — while the routing rule ships in your code. Moda is not an LLM router, not a marketplace like OpenRouter, not a gateway like LiteLLM, and it never sits in your request path. ## Harness engineering for LLM routers Routers are harness components, and they fail like harness components: silently, per task, and for reasons a request-level dashboard cannot see. A misroute is a behavioral failure — the request succeeded, the wrong model served it, and nothing in the response says so. Harness engineering for LLM routers is the practice of holding the router to the same evidence standard as every other component on this page: traces in, attribution in the middle, verified changes out. The gateway alone does not get you there. A gateway can report which model served each request, at what cost and latency; it cannot tell you the request should have gone elsewhere, because that judgment needs a task taxonomy and per-task quality evidence it does not have. Teams building routers need the harness engineering underneath: task clusters from production traffic to give the router its categories, per-task replay comparisons to say which model holds where, attribution when a route regresses — often a prompt or tool-schema fix lets the cheaper model hold, rather than a model change — and a replay gate on every routing update. That is the division of labor to keep straight: the router is a component teams build and own; harness engineering is the discipline that keeps it correct as traffic and models drift. Moda supplies the discipline's infrastructure — taxonomy, comparisons, attribution, verification — as a harness engineering platform, not as a router SKU. ## How to improve LLM routing with production feedback A routing policy is only as good as its last contact with production. Improving LLM routing with production feedback means the outcomes of routed traffic — not benchmarks, not intuition — drive every subsequent routing change, through the same loop as any other harness component: - Capture per-task outcomes of routed traffic: which model served which task cluster and how it actually went — task success, tool-call correctness, retry and escalation rates — not just the cost and latency a gateway reports. - Attribute each regression the feedback surfaces to its cause: sometimes the route itself (the assigned model no longer holds on that task), often the harness underneath (a prompt or tool schema the cheaper model needs sharpened before it can hold). - Turn the feedback into a reviewable diff — a routing-table change or the attributed harness fix — with the traces that motivated it attached as evidence. - Replay-verify the diff against the traffic that produced the feedback before it ships, with the standard held-out regression check. - Keep the loop standing rather than one-off: new task clusters appear, models change and improve, and each routing update passes the same replay gate as the first. This is what it means to close the loop on model routing: production outcomes feed back into the routing policy and the harness beneath it, on the same evidence standard as every other harness change. An open-loop router — configured once, revisited after incidents — decays silently, because misroutes succeed at the request level and only show up as per-task quality drift. The closed loop catches that drift as it happens, attributes it, and ships the verified fix. The harness for model routers is the infrastructure that makes this loop runnable: the task taxonomy that gives the router its categories, per-task replay comparisons that say which model holds where, attribution that separates route problems from prompt and tool-schema problems, and the replay gate every routing change passes before shipping. Moda supplies that harness infrastructure for teams building their own model router on their own task evidence — it is not an LLM router product, not a marketplace, not a gateway, and it never sits in the request path. The routing rule stays in your code; the feedback loop around it is what Moda runs. ## Difference between an LLM router product and harness engineering The difference between an LLM router product and harness engineering is the difference between a component you buy and a discipline you run. An LLM router product picks a model or provider for each request — a marketplace like OpenRouter, a gateway like LiteLLM, a trained routing classifier like RouteLLM. Harness engineering is the practice of improving everything around the model — prompts, tools, skills, evals, memory, and the routing policy itself — from production traces, with every change verified against replayed traffic before it ships. | | LLM router product | Harness engineering | | --- | --- | --- | | What it is | A component: picks a model or provider per request | A discipline: improves the harness from production evidence | | What it decides | Which model serves this request | Which harness component to change, and whether the change worked | | Where it runs | In the request path | On production traces, outside the request path | | Evidence | Cost, latency, availability per request | Per-task replay comparisons, failure attribution across the trace population | | Output | A routed request | Verified harness improvements — prompt, tool, skill, eval, memory, and routing diffs | The two are not rivals, and one does not replace the other: a router product cannot tell you a route was wrong on your tasks — the misrouted request still returns 200 — and harness engineering does not move requests. Teams that conflate them buy a gateway expecting quality improvement and get cost accounting, or shop for a router when the actual gap is that nobody can say which routes regress and why. The router-builder sections above cover the practice in detail: building an LLM router on production traces, harness engineering for LLM routers, and improving routing with production feedback. Moda sits on exactly one side of this line: it is harness engineering infrastructure for teams building their own model router — the task taxonomy, per-task replay comparisons, failure attribution, and verification gate — and it is not an LLM router product, not an OpenRouter or LiteLLM alternative, and never in your request path. The routing rule stays in your code. Source: - Moda, What is an LLM router (pillar guide): https://moda.dev/llm-router ## What is harness engineering for routing teams Harness engineering for routing teams is the standard discipline of this page applied by the team that owns a model-routing layer: production traces in, attribution to the responsible component in the middle, verified changes out — where the components include the routing policy alongside the prompts, tools, skills, evals, and memory the routed models run inside. The framing matters because routing teams inherit a failure mode the rest of the harness does not have: their component's failures are invisible at the request level. A misroute succeeds — wrong model, valid response, 200 status — and only shows up as per-task quality drift weeks later. So the routing team's version of the practice leans hardest on the loop's evidence steps: a task taxonomy from production traffic so quality has a per-task denominator, per-task replay comparisons that say which model holds where, and attribution that separates a genuine route problem from a prompt or tool schema the cheaper model needs sharpened before it can hold. Many routing regressions are not routing problems; without attribution, the routing team fixes them in the routing table anyway. What the routing team is left holding is the same unit as every other harness team: verified improvements — a routing-table change or the attributed harness fix underneath it, shipped only after replaying the affected traffic. Moda supplies that loop as infrastructure for routing teams: taxonomy, comparisons, attribution, and the replay gate, while the routing rule itself stays in the team's code. It is not an LLM router product and does not compete with the gateway the team may also run. ## Harness engineering for model routing teams Harness engineering for model routing teams is the same practice named at the level teams search for it: the discipline a team standing up or operating model routing uses to keep the routing policy — and the harness beneath it — correct as traffic and models drift. The recurring jobs: - Own the task taxonomy: cluster production traces into the task categories routing decisions are made against, and keep it current as new task clusters appear. - Maintain the per-task evidence base: replay comparisons across candidate models on real traffic, so the routing table is built and audited on your tasks rather than public benchmarks. - Attribute every regression before touching the routing table: route problem, or a prompt, tool, skill, eval, or memory problem the assigned model exposed — the chooser earlier on this page applies to routed traffic unchanged. - Gate every routing change on replays: targeted tasks improved, held-out tasks unregressed — the same two-number standard as any other harness change. - Keep the loop standing: models improve, costs shift, task mixes drift, and each update passes the same gate as the first. The scope line for model routing teams is the one drawn in the sections above: the router is a component the team builds and owns; the LLM router product category — marketplaces, gateways, hosted routers — solves request movement, not quality; and harness engineering is the discipline that keeps the whole routed system correct. Moda's role for these teams is the infrastructure of that discipline — task taxonomy, per-task replay comparisons, failure attribution, verification — as a harness engineering platform. It is not an LLM router, not an OpenRouter or LiteLLM alternative, and it never sits in the request path. ## How to auto curate skills from production sessions Skills are the harness component where recurring competence lives: a SKILL.md file names a task the agent handles repeatedly, describes when to load it, and carries the procedure so the model stops re-deriving it from scratch. Auto-curation is the harness engineering loop applied to that one component — production sessions in, reviewed SKILL.md candidates out — and it runs in five steps: - Cluster production sessions by task, so skill candidates come from the workflows users actually repeat rather than from someone's guess about what deserves a skill. - Identify the recurring competence inside each cluster: the multi-step procedures the agent keeps reconstructing from scratch, and the variants where it goes wrong. - Distill each cluster into a draft SKILL.md — a name, a description that tells the agent when to load it, and the procedure with its checks. - Review the draft like any other harness diff. A generated skill is a proposal with evidence attached, not a deployment. - Gate each candidate on replays of the sessions it was distilled from, compared against a baseline without it, and release only what measurably helps. The word that matters is curate, not generate. An LLM will happily produce a plausible SKILL.md for anything; curation is the evidence standard that decides which skills deserve to exist — recurring in real sessions, distinct from what the prompt already covers, and verified to improve the replays they came from. Moda runs this pipeline as part of the product: production sessions are clustered, recurring workflows are distilled into SKILL.md candidates, and every candidate is gated on baseline-relative replay before it can be released. SKILL.md examples and format questions belong here too. The format is deliberately small — a markdown file with frontmatter (a name plus a description that tells the agent when to load it) and a procedure body — and public SKILL.md examples are easy to find, from Anthropic's open skills repository to the machine-readable skills this site ships for its own integrations. The format is the packaging; the discipline is the curation loop above, which decides what goes in the file and proves it helps. Sources: - Anthropic, open agent skills repository (anthropics/skills): https://github.com/anthropics/skills - Moda, Claude Code integration skill (SKILL.md example): https://moda.dev/skills/claude-code.md ## How to create a skill md file To create a skill md file, make a directory named after the skill and put a SKILL.md file inside it: YAML frontmatter with a name and a description that tells the agent when to load the skill, followed by a markdown body carrying the procedure. That is the whole format. The steps: - Create the folder. The directory name is the skill name — lowercase letters, numbers, and hyphens (pdf-processing, code-review) — and the frontmatter name field must match it. - Write the frontmatter. Two required YAML fields: name, and a description saying what the skill does and when to use it. The description is the trigger — agents load it at startup and match it against the task at hand, so it should carry the keywords a matching task would contain. - Write the body. Plain markdown, no format restrictions: the step-by-step procedure, input and output examples, edge cases. Keep it under roughly 500 lines and move deep reference material into separate files the agent loads on demand. - Add optional resources. scripts/ for executable helpers, references/ for detailed docs, assets/ for templates — loaded progressively, only when the task calls for them. - Put it where your agent looks. Claude Code reads .claude/skills/ in the repository and ~/.claude/skills/ for personal skills; other agents that support the spec, from Gemini CLI to OpenCode and OpenHands, document their own load paths. In harness engineering terms, that file is packaging, and the mechanics above take minutes. The decisions that make the file worth loading — which recurring workflow deserves a skill, what the procedure actually is, whether the skill measurably helps — are the hard part, and they come from production sessions: that is what the auto-curation loop above exists for. Moda ships machine-readable SKILL.md files for its own integrations and generates skill candidates from real traffic rather than from someone's memory of the workflow. Source: - Agent Skills specification (agentskills.io): https://agentskills.io/specification ## What is a skill.md specification? The skill.md specification is a small open format for giving AI agents packaged capabilities: a skill is a directory containing a SKILL.md file, whose YAML frontmatter — a name plus a description of when to use it — tells the agent when to load the skill, and whose markdown body carries the instructions. The public specification lives at agentskills.io. It documents the two required frontmatter fields and the optional ones (license, compatibility, metadata, allowed-tools), the optional scripts/, references/, and assets/ directories, and the progressive-disclosure model: every skill's name and description (about a hundred tokens) load at startup, the full body loads only when the skill is activated, and bundled files load only when the task needs them. Anthropic's open skills repository is the best-known body of public examples, and the format is deliberately agent-agnostic — Claude Code, Gemini CLI, OpenCode, and OpenHands all load the same files. Moda's place in that ecosystem is not the spec; it is what fills the file. The specification defines packaging and says nothing about which skills your agent needs or whether a given SKILL.md actually helps. Moda uses SKILL.md as the packaging for curated production competence: recurring workflows detected in production sessions are distilled into SKILL.md candidates, reviewed like any other harness diff, and gated on replays against a baseline before release — the curation loop described earlier on this page. Source: - Agent Skills, the open skill.md format (agentskills.io): https://agentskills.io/ ## How to improve skill.md from production failures Creating and curating skills is covered earlier on this page; this is the other half of the lifecycle — an existing SKILL.md is live, production shows it failing, and the file needs a repair rather than a replacement. The failure almost always lives in one of three parts of the file, and production traces tell you which: - Wrong trigger: the frontmatter description no longer matches when the skill should load, so the agent skips it on tasks it was written for — or loads it on tasks it was not. The evidence is sessions where the task recurred and the skill stayed cold, or loaded and derailed. - Stale steps: the procedure body references tools, flags, paths, or sequences that have drifted since the skill was written. The evidence is sessions where the skill loaded, the agent followed it faithfully, and the followed step is the one that failed. - Missing checks: the procedure completes but skips a verification the task actually needs, so the agent reports success on work that is wrong. The evidence is wrong-but-successful sessions passing through the skill. - Patch the SKILL.md accordingly — a sharper description, corrected steps, an added check — as a reviewable diff with the failing sessions attached as evidence. - Replay-verify before shipping: re-run the failing sessions under the patched skill against the unpatched baseline, and hold the patch to the same two-number gate as any harness change — targeted failures fixed, held-out sessions unregressed. The discipline matters because the popular alternative — reading a failure anecdote and rewriting the skill by feel — is how skills rot into the same state as the two-thousand-line system prompt. A SKILL.md is a harness component; improving it from production failures is the standard loop applied to one file: traces in, attribution to the trigger, steps, or checks, a patch out, and a replay gate before it ships. Moda runs this repair path alongside its skill curation: detected failures in sessions that touched a skill are attributed to the responsible part of the file, and every proposed patch is gated on replays of the sessions it claims to fix. ## Skill harness improvement Skill harness improvement is the practice of improving the skill layer of an agent harness from production evidence: which skills exist, when they load, what their procedures say, and whether each one measurably earns its place. It is a discipline applied to a harness component you already own — not a product category, not a marketplace of downloadable skills, and not specific to any one agent vendor. The skill layer has become the natural home of this work because skills are where recurring competence is versioned: Lilian Weng's writing on harness engineering for self-improvement frames the harness — workflows, memory, tools, skills — as the optimization target that improves while the model stays fixed, and skills are the component of that target designed to be edited. The full improvement loop on the skill layer runs in both directions covered on this page: curation adds skills that production sessions justify, and repair fixes the trigger, steps, or checks of skills that production failures indict — with every change, in either direction, verified against replayed sessions before it ships. That evidence standard is the difference between skill harness improvement and skill accumulation. A directory of plausible SKILL.md files is not an improved harness; an improved harness is one where each skill's existence, trigger, and procedure are backed by production sessions and replay comparisons. Moda's role is that evidence layer: session clustering to say which skills are justified, failure attribution to say which need repair, and the replay gate that decides what ships. Source: - Weng, Harness Engineering for Self-Improvement (Lil'Log): https://lilianweng.github.io/posts/2026-07-04-harness/ ## How to improve agent tool definitions from traces Tool definitions are the harness component where small wording errors become systematic failures: a vague description invites misselection, a loose schema invites wrong arguments, and both fail silently because the tool call returns 200. Improving agent tool definitions from traces is the harness loop pointed at that component: - Mine the trace population for tool-call failure patterns: wrong tool selected for the task, right tool with wrong arguments, retry loops, and wrong-but-successful calls where the response satisfied the schema but not the task. - Attribute each pattern to the definition, not the model: a description that overlaps a sibling tool's, a parameter whose name suggests the wrong unit, an optional field the task actually requires, an enum missing the value users need. - Tighten the definition so the model's cheapest interpretation is the correct one — sharper descriptions, stricter schemas, disambiguated names — as a reviewable diff with the failing traces attached. - Replay the traces that exhibited the failure under the tightened definition and compare against the current one before shipping, with the usual held-out regression check. This is tool harness improvement: the tool layer of the harness held to the same evidence standard as prompts and skills. It is routinely the highest-leverage repair on the page — a one-line schema fix can retire a failure pattern that a week of prompt edits could not, because the prompt was never the responsible component. Moda detects tool-call failure patterns across the trace population, attributes each to the tool definition responsible, and gates every proposed tightening on replays of the traces that motivated it. ## Which harness layer when agent misbehaves When an agent misbehaves, the harness layer to change — prompt, tool, skill, eval, or memory — is the one the production evidence indicts, and each layer leaves a different signature in the traces. The chooser: - Prompt: the failure cuts across task types, and the traces show an instruction that exists but gets ignored or misread, instructions that contradict each other, or no instruction covering what users actually ask. The repair path is the prompt sections below — improving prompts from production traces, not appending another paragraph by feel. - Tool: the agent picks the wrong tool, calls the right tool with wrong arguments, loops on retries, or gets a wrong-but-successful response that returned 200 without serving the task. The repair path is the tool-definitions section above: tighten the description or schema so the cheapest interpretation is the correct one. - Skill: the failure clusters on one recurring task, where a SKILL.md stayed cold when it should have loaded, loaded and derailed, or was followed faithfully into a stale step. The repair path is the skill-repair section above: fix the trigger, steps, or checks the failing sessions indict. - Eval: a harness change shipped and regressed behavior nothing gated — the failure is not in any one component but in the absence of the check that would have caught it. The repair is an expanded eval set built from the traces that regressed. - Memory: the agent re-asks for information the user already gave, contradicts an earlier turn, or retrieves the wrong context into the window. The repair lives in the memory layer — what gets written, what gets retrieved, and when. The chooser matters because the default failure mode is skipping it: every fix lands in the system prompt because the prompt is the only surface anyone edits, and the actual culprit — a loose tool schema, a stale skill step, a missing eval — survives to fail again. Attribution comes before editing: read the failing traces, identify the layer whose signature they carry, and only then open the file. The five layers themselves are the parts list covered in the agent harness guide. At production volume nobody runs this chooser by hand, which is the attribution step of the loop Moda automates: behavioral failures detected across the whole trace population, each one routed to the prompt, tool, skill, eval, or memory component responsible, with the evidence attached — so the layer decision is made from the traces rather than from whoever edited the prompt last. Source: - Moda, What is an agent harness (pillar guide): https://moda.dev/agent-harness ## How to improve prompts from production traces To improve prompts from production traces, you mine the trace population for failures the prompt actually owns, patch the responsible section as a reviewable diff, and replay the failing traces under the patched prompt before it ships: - Mine the trace population for prompt-owned failure patterns: an instruction that exists but gets ignored or misread mid-context, sections that contradict each other, coverage gaps where no instruction addresses what users actually ask, and stale sections describing behavior the system no longer has. - Confirm the prompt is the responsible layer before editing it. Many failures that look like prompt problems are tool schema, skill, or memory problems — the chooser above decides — and a prompt edit for a tool failure fixes nothing while making the prompt longer. - Attribute each confirmed failure to the specific prompt section responsible, not to the prompt as a whole: which instruction, which example, which constraint the failing traces implicate. - Patch as a minimal reviewable diff — sharpen, relocate, or delete the indicted section rather than appending a new paragraph — with the failing traces attached as evidence. - Replay the traces that exhibited the failure under the patched prompt and compare against the current one before shipping, with the standard two-number gate: targeted failures fixed, held-out traffic unregressed. This differs from the prompt optimization loop as usually practiced, where a prompt is iterated against a static eval set or rewritten by an optimizer until a score moves. That loop tests the prompt against the cases someone remembered to write; production traces supply the denominator it is missing — which failures actually recur, at what rate, and whether the failing behavior was ever the prompt's fault at all. Optimizing a prompt against an eval set that lacks the failing traffic polishes the wrong surface. This is the prompt-layer sibling of the tool and skill loops covered above — improving agent tool definitions from traces, and improving SKILL.md from production failures — the same discipline pointed at a different file. Moda runs it as part of the platform: prompt-owned failure patterns detected across the trace population, attribution down to the responsible prompt section, and every proposed edit gated on replays of the traces that motivated it. ## Prompt harness improvement Prompt harness improvement is the practice of improving the prompt layer of an agent harness — system prompts, AGENTS.md, per-task instructions — from production evidence, with every change replay-verified before it ships. It is a discipline on a harness component you own, and the prompt layer's counterpart to the tool harness improvement and skill harness improvement sections above: the same loop, pointed at the instructions. The improvement runs in both directions. Production traffic justifies additions — a recurring intent no instruction covers earns a new section, or more often a new skill so the prompt stays small. Production failures justify repairs and cuts — an ignored instruction gets sharpened or moved out of the mid-context dead zone, contradicting sections get reconciled, and paragraphs that no longer change behavior get deleted. Deletion is the half that never happens without evidence: nobody removes a prompt paragraph on intuition, because nobody can say what it was protecting. Replay comparison makes the cut safe. The scope line matters here: prompt engineering is a component inside harness engineering, not the whole practice, and prompt harness improvement is not a return to prompt-only iteration. It is the prompt layer held to the same evidence standard as every other layer on this page — failures attributed to the specific section responsible, minimal diffs rather than accretion, and a replay gate before shipping. That standard is what keeps the prompt from becoming the two-thousand-line file where every fix lands by default. Moda supplies the evidence layer: prompt-owned failures detected across production traces, attribution to the responsible section, and the replay verification that decides what ships. ## Who needs harness engineering, and when The practice earns its cost at a specific point: when an agent is in production, users depend on it, and the volume of traces exceeds what anyone reads by hand. Before that point, the founder reads every transcript and the harness fits in their head; formalizing the loop is premature. After it, the failure mode is predictable: improvement decisions get made from the loudest complaint rather than the largest pattern, every fix lands in the system prompt because it is the only visible surface, and nobody can say whether last month's changes helped. Teams usually arrive at harness engineering after noticing that they benchmark models rigorously and change their harness by vibes — while the harness is the part they actually control. The buy-vs-build question is about the middle of the loop. Instrumentation is a solved problem and shipping is your deploy pipeline. Population-scale failure detection, component-level attribution, and replay-based verification are the parts that are genuinely hard to build in-house — that is the part Moda sells, and the demo is the fastest way to see it against your own kind of traffic. ## Frequently asked questions ### What is harness engineering? Harness engineering is the discipline of improving an AI agent by engineering everything around the model — prompts, tools, skills, evals, memory — using evidence from production. The model is treated as a fixed component; the harness is the system you design, measure, and iterate, and the output is verified harness improvements produced from production traces. The working loop is: map the harness, observe production, attribute failures to the responsible component, improve it, and verify the change against real traffic. ### What is harness engineering for agents? The same discipline, qualified for its subject: harness engineering for agents is the improvement of a deployed AI agent by engineering the components around its model — prompts, tools, skills, evals, memory — from production evidence, with every change verified against replayed traffic before it ships. The qualifier matters mostly for disambiguation: unqualified, harness engineering can collide with the Harness.io CI/CD platform and with hardware test harnesses. For agents, the practice is the loop this page describes — map the harness, observe production, attribute failures to the responsible component, improve it, verify the change — and the output is verified harness improvements rather than dashboards or reports. ### How to do harness engineering for coding agents? Run the harness engineering loop on the artifacts a coding agent actually touches. Map the harness as it exists in the repository — AGENTS.md and system prompts, tool and MCP definitions, SKILL.md files, linters and CI checks, evals, memory. Observe complete production sessions rather than isolated tool calls. Attribute each failure to the component responsible: a stale AGENTS.md section, a loose tool schema, a drifted skill, a missing test sensor, or context rot on long sessions. Improve it as a guide (steering the agent before it acts) or a sensor (catching the failure after), and verify by replaying the failing sessions under the changed harness — targeted failures fixed, held-out sessions unregressed — before it ships. Coding agents are the friendliest case for the discipline: outputs are verifiable diffs and the harness surfaces are files in the repo, so every improvement is a reviewable, revertible commit. Moda runs this loop as a harness engineering platform — production session traces in, attribution to the responsible component, replay-verified improvements out. ### Is harness engineering the same as the anatomy of an agent harness? No — anatomy is the parts list, engineering is the practice. Anatomy guides (LangChain's is the best known) explain which components a harness needs and why: tools, orchestration, memory, filesystem, hooks. Harness engineering is what happens after the harness exists and is serving production traffic: detecting which component is failing, changing it, and verifying the change against replayed traces. Knowing the anatomy without running that loop is how teams end up shipping prompt edits by intuition after every incident. ### Harness engineering vs prompt engineering: what is the difference? Prompt engineering iterates on one artifact, usually by intuition. Harness engineering treats the whole assembly — prompts, tools, skills, evals, memory — as the design surface and attributes each production failure to the component actually responsible. Many failures that look like prompt problems are tool schema, memory, or eval problems; a practice that can only edit the prompt fixes them all in the wrong place. The same relationship holds one level up for context engineering: curating what the model sees is one harness component, and harness engineering is the loop that decides, from production evidence, which component to change. ### What is agent harness engineering? The same discipline under its fully qualified name — useful because the bare phrase harness engineering collides with the Harness.io CI/CD platform and with hardware test harnesses. Agent harness engineering is the practice of improving an AI agent by engineering its harness — prompts, tools, skills, evals, memory — from production evidence. Addy Osmani's essay of that title captures the working posture: the scaffolding around the model is a real artifact, and it tightens every time the agent slips. The tightening loop is what a harness engineering platform automates — failures detected across production traces, attributed to the responsible component, and improvements verified against replayed traffic before they ship. ### What is OpenAI harness engineering? The methodology from OpenAI's essay Harness engineering: leveraging Codex in an agent-first world (February 2026), the lab's first-party account of the discipline: a team shipped an internal product of roughly a million lines of code over five months with zero manually written lines, by moving the engineers' work from writing code to designing the environment — AGENTS.md as a map rather than a manual, mechanical architecture enforcement through custom linters, application logs and metrics made legible to the agent, and recurring cleanup agents that pay down drift. Read the original; it is the canonical statement. What it does not cover is how teams who are not OpenAI run that loop — that is what a harness engineering platform like Moda provides, turning your production traces into verified harness improvements. ### What is Anthropic harness engineering? The harness-design framing from Anthropic's engineering blog: harness design for long-running work, and interfaces built to outlive any single harness. Harness design for long-running application development (March 2026) splits long-horizon tasks across planner, generator, and evaluator agents with a sprint contract negotiated before code is written, on the finding that a model cannot reliably judge its own output — evaluation has to be a separate engineered component. Scaling Managed Agents (April 2026) argues that harnesses encode assumptions about what the model cannot do, that those assumptions go stale as models improve, and that the surrounding interfaces should stay stable while harnesses change. Both are design-time guidance from inside the lab that builds Claude; the production-side complement — detecting which of your harness's assumptions is failing now, and verifying the fix — is what Moda runs from your traces as harness infrastructure, not as a competing lab. ### What is AI harness engineering? The same discipline with the AI qualifier attached, useful when the bare phrase could be read as the Harness.io CI/CD platform or a hardware test harness. AI harness engineering is the practice of improving an AI agent by engineering the harness around its model — prompts, tools, skills, evals, memory — from production evidence, with every change replay-verified before it ships. The labs anchored the term (OpenAI's Codex essay, Anthropic's harness-design posts); the practice for everyone else is the loop on this page, and the platform that runs it against your own production traces is what Moda provides. ### How is harness engineering different from observability? Observability tells you what the agent did; harness engineering decides what to change and verifies that it worked. Traces and dashboards are the input. The practice is everything downstream: detecting behavioral failures traces miss (wrong-but-successful tool calls, context loss, goal drift), attributing them to a harness component, and replaying proposed changes against production traffic before shipping. ### What is a silent agent failure? A silent agent failure is an agent failure that produces no error signal: every span succeeds, every tool call returns 200, the response arrives — and the outcome is still wrong for the user. The recurring forms are wrong-but-successful tool calls, silently lost context, goal drift, skill and instruction misfires, and misroutes. They are invisible one trace at a time because the failure lives in the gap between what the user needed and what the agent did, which no status code carries; they become visible when production sessions are clustered by behavior at population scale. Moda detects silent agent failures by clustering production failures across the whole trace population, attributes each cluster to the prompt, tool, skill, eval, or memory component responsible, and turns them into verified harness improvements gated on replayed traffic. ### What are agent failure modes? Agent failure modes split into two families. Hard failures throw signals infrastructure already catches: exceptions, timeouts, 4xx/5xx responses, rate limits. Behavioral failures — the family this page cares about — succeed at the infrastructure level and fail the user: wrong-but-successful tool calls, context loss, goal drift, reasoning loops, skill misfires, misroutes. The behavioral family is the silent agent failure class defined in the previous answer, catalogued in Moda's glossary under behavioral failure (/glossary#behavioral-failure), with the population-scale detection method covered in the clustering write-up (/blog/clustering). Harness engineering is the discipline that consumes that taxonomy: each detected failure mode is attributed to the harness component responsible and fixed as a verified, replay-gated improvement. ### Should I buy observability or a harness engineering platform? Decide by where your gap is. If you cannot see what your agent did, buy observability — instrumentation is a solved problem and OpenTelemetry keeps it vendor-neutral. If you already have traces and improvement is still manual — reading transcripts, guessing at prompt edits, shipping them unverified — buy a harness engineering platform: it turns those traces into verified harness improvements, which is the part that is genuinely hard to build in-house. More dashboards do not close that gap; harness changes from traces do. The two are complements, not substitutes: observability answers what happened on this run, a harness engineering platform answers which harness component to change and whether the change worked, including on the behavioral failures that succeed at the infrastructure level. Moda is on the harness engineering side of that line — harness infrastructure, not an observability product and not an LLM router — and it consumes the traces your existing instrumentation already emits rather than replacing it. ### What are verified harness improvements? Verified harness improvements are concrete, reviewable changes to a prompt, tool schema, skill, eval set, or memory component that were proposed from production evidence and checked by replaying real production traces under the proposed configuration before shipping. The verification is the point: the change is compared against current behavior on the traffic it was written to fix, including whether it regressed anything else, and it reverts with a rollback if the numbers say to. ### How do you get harness changes from traces? Through the harness engineering loop. Capture complete production traces, not sampled spans; detect behavioral failures and emergent intents across the whole population; attribute each finding to the harness component responsible — this prompt section, this tool schema, this memory lookup; turn the attribution into a concrete diff; and verify the diff against replayed production traffic before it ships. Traces are the raw material at every step, but the deliverable is the harness change, not the trace. ### How do you evaluate whether a harness change actually fixed the failure? With a replay protocol, not with the absence of complaints. Pin the failure cluster — the production traces exhibiting the failure — before the change ships; replay that cluster under the proposed configuration and compare trace by trace against the current one; run the same comparison on held-out traffic the change was not written for; and make the ship-or-revert decision from both numbers together: targeted failures fixed, held-out behavior held. After shipping, keep watching the cluster's recurrence rate — that is the lasting answer. This two-number gate is the same acceptance rule the Co-Harness research applies with an automated critic, and it is the verification step Moda runs on every proposed harness improvement. ### How do you transfer routing policy from frontier to open source models? Treat it as harness engineering, not a model swap. Cluster production traces into a task taxonomy so the routing policy has a per-task denominator; replay each task's real traces through the open-source candidate and compare against the frontier baseline; attribute regressions to the harness component responsible — many gaps close with a prompt rewrite, a tightened tool schema, or an added skill rather than a bigger model; then move routing task by task, keeping regressed tasks on the frontier model until a verified harness change closes the gap. Every routing change is verified against replayed production traffic before it ships, which is what holding task performance means in practice. ### Can you build a task trained model router without losing performance? Yes — if the router is trained and audited on your own task distribution rather than public benchmarks, and the harness work underneath it is done. The router — a rule or classifier that sends each task to the model that serves it best — is a component you build and own inside your harness. What prevents performance loss is the harness engineering under it: a task taxonomy from production traffic, per-task replay comparisons showing where a cheaper or open model holds and where it regresses, attributed harness fixes that close the gaps, and a replay gate every routing change passes before shipping. Moda supplies that infrastructure; the router stays yours. ### What are verified improvements for agent model routing? Routing changes held to the same evidence standard as any other harness change: proposed from per-task replay comparisons on your own production traces, attributed to what actually moved the numbers — sometimes the model choice, often a prompt or tool-schema fix that lets a smaller model hold — and checked by replaying the change against real traffic before it ships. A verified routing improvement is a reviewable diff with the evidence attached: the task cluster it affects, the before-and-after comparison, and the regression check on everything it might have broken. ### How do you build an LLM router on production traces? Run the harness loop and let the router fall out of it. Cluster production traces into a task taxonomy so the router's categories match your traffic; run per-task model comparisons on replayed traces to build the routing table from evidence rather than benchmarks; write the routing rule as harness code you own — a rule or a classifier trained on your task distribution; replay-verify the routed configuration against the unrouted baseline before it ships, including regression checks on unchanged tasks; and keep every later routing update behind the same replay gate. Moda supplies the taxonomy, the per-task comparisons, and the verification; the router itself stays in your code. ### What is harness engineering for LLM routers? Treating the router as a harness component with the same evidence standard as prompts, tools, and skills. A misroute is a behavioral failure — the request succeeds on the wrong model and nothing in the response says so — which is why gateway cost and latency stats are not enough. Harness engineering for LLM routers means giving the router a task taxonomy from production traffic, per-task replay comparisons that say which model holds where, attribution when a route regresses (often a prompt or tool-schema fix rather than a model change), and a replay gate on every routing update. The router is a component you build; harness engineering is what keeps it correct as traffic and models drift. ### How do you improve AI agent model routing? With verified harness improvements, not by buying a router product. Routing quality improves the same way any harness component improves: detect the misrouted task clusters in production traces, attribute each regression to what actually caused it — sometimes the model assignment, often a prompt or tool schema that a cheaper model needs sharpened — propose the routing or harness change, and verify it against replayed production traffic before shipping. A router product can move requests; it cannot tell you which routes are wrong on your tasks or prove a change helped. That evidence loop is the improvement mechanism, and it is what Moda provides under a router you own. ### How do you improve LLM routing with production feedback? Let routed traffic grade the routing policy, then run the harness loop on what it finds. Capture per-task outcomes of routed traffic — task success, tool-call correctness, retry and escalation rates, not just cost and latency; attribute each regression to the route (the model no longer holds on that task) or to the harness underneath (a prompt or tool schema the cheaper model needs sharpened); turn the finding into a reviewable diff, routing-table change or harness fix; and replay-verify it against the traffic that produced the feedback before shipping. The loop stays standing: new tasks appear, models change, and every update passes the same replay gate. ### How do you close the loop on model routing? By making production outcomes the input to the next routing change. An open-loop router is configured once and revisited after incidents; it decays silently because misroutes succeed at the request level and only appear as per-task quality drift. Closing the loop means per-task outcome capture on routed traffic, attribution of each regression to the route or to the harness component responsible, and a replay-verified diff before anything ships — the same evidence standard as every other harness change. Moda runs that loop as harness infrastructure under a router you build and own. ### What is a harness for model routers? The infrastructure a routing policy needs around it to stay correct: a task taxonomy from production traffic that gives the router its categories, per-task replay comparisons that say which model holds where, attribution that separates route problems from prompt and tool-schema problems, and a replay gate every routing update passes before shipping. The router itself is a small rule or classifier; the harness around it is what keeps it honest as traffic and models drift. Moda provides that harness for teams standing up routers on their own task evidence — it is not an LLM router, not a marketplace like OpenRouter, and not a gateway like LiteLLM. ### Is Moda an LLM router or an OpenRouter alternative? No. Moda is a harness engineering platform — it is not an LLM router, not a model marketplace like OpenRouter, not a gateway like LiteLLM, and it does not resell inference or sit in your request path. Moda is the infrastructure for teams building a model router for their own tasks: it supplies the task taxonomy from production traces, the per-task model comparisons on replayed traffic, and the verification behind every routing change, while the routing rule itself lives in your harness as code you own. ### What is the difference between an LLM router product and harness engineering? An LLM router product is a component that picks a model or provider for each request — a marketplace like OpenRouter, a gateway like LiteLLM, a trained classifier like RouteLLM — and it runs in the request path on cost, latency, and availability signals. Harness engineering is a discipline, not a component: it improves the system around the model — prompts, tools, skills, evals, memory, and the routing policy itself — from production traces, with every change verified against replayed traffic before it ships. The router moves requests; harness engineering decides what to change and proves the change worked. They are complements, not rivals: a router product cannot tell you a route was wrong on your tasks, because a misroute still returns 200. Moda is on the harness engineering side of the line — infrastructure for teams building their own router, never a router product. ### What is harness engineering for routing teams? The standard harness engineering loop applied by the team that owns a model-routing layer: production traces in, attribution to the responsible component, verified changes out — where the components include the routing policy alongside the prompts, tools, skills, evals, and memory the routed models run inside. Routing teams lean hardest on the evidence steps, because misroutes are invisible at the request level: a task taxonomy gives quality a per-task denominator, per-task replay comparisons say which model holds where, and attribution separates genuine route problems from the prompt or tool schema a cheaper model needs sharpened. Every routing change ships through the same replay gate as any other harness change. Moda supplies that loop as infrastructure; the routing rule stays in the team's code. ### What is harness engineering for model routing teams? The jobs a team standing up or operating model routing runs to keep the routing policy and the harness beneath it correct: own the task taxonomy clustered from production traces, maintain per-task replay comparisons across candidate models, attribute every regression before touching the routing table — route problem or harness problem — gate every routing change on replays with the two-number standard (targeted tasks improved, held-out tasks unregressed), and keep the loop standing as models and traffic drift. The LLM router product category — marketplaces, gateways, hosted routers — solves request movement, not quality; harness engineering for model routing teams is the quality discipline. Moda provides its infrastructure as a harness engineering platform — not an LLM router, not an OpenRouter or LiteLLM alternative, and never in the request path. ### How do you improve agent tool definitions from traces? Mine the trace population for tool-call failure patterns — misselection, wrong arguments, retry loops, and wrong-but-successful calls that returned 200 without serving the task; attribute each pattern to the definition responsible (an overlapping description, a misleading parameter name, a loose schema, a missing enum value) rather than to the model; tighten the definition so the cheapest interpretation is the correct one; and replay the failing traces under the tightened definition against the current one before shipping, with a held-out regression check. Tool definition fixes are routinely the highest-leverage harness repairs, because failures that look like reasoning problems are often schema problems. ### What is tool harness improvement? Improving the tool layer of the agent harness — tool descriptions, parameter schemas, names, and routing between overlapping tools — from production evidence, at the same standard as any other harness change: failures detected across traces, attributed to the specific definition responsible, fixed as a reviewable diff, and verified against replays before shipping. It is a discipline on a component you own, not a product category. Moda supplies the detection, attribution, and replay verification; the tool definitions stay in your harness code. ### Which harness layer do you change when an agent misbehaves? The layer whose signature the failing traces carry. Prompt: the failure spans task types and the traces show an ignored, contradictory, or missing instruction. Tool: wrong tool selected, wrong arguments, retry loops, or wrong-but-successful calls that returned 200 without serving the task. Skill: the failure clusters on one recurring task where a SKILL.md stayed cold, misfired, or carried a stale step. Eval: a change shipped and regressed behavior no gate caught — the missing component is the check itself. Memory: the agent re-asks for known information, contradicts an earlier turn, or retrieves the wrong context. Attribute before editing; the default mistake is routing every fix into the system prompt because it is the only surface anyone edits. Moda automates this attribution across the trace population, routing each detected failure to the responsible layer with the evidence attached. ### How do you improve prompts from production traces? Mine the trace population for failures the prompt actually owns — ignored instructions, contradicting sections, coverage gaps, stale sections; confirm the prompt is the responsible layer rather than a tool schema, skill, or memory problem; attribute each failure to the specific prompt section responsible; patch it as a minimal reviewable diff — sharpen, relocate, or delete rather than append — with the failing traces attached; and replay those traces under the patched prompt against the current one before shipping, holding the change to the two-number gate: targeted failures fixed, held-out traffic unregressed. This beats optimizing against a static eval set because production traces supply the denominator — which failures recur, at what rate, and whether the prompt was ever the culprit. Moda runs this loop as part of the platform. ### What is prompt harness improvement? Improving the prompt layer of an agent harness — system prompts, AGENTS.md, per-task instructions — from production evidence: adding coverage that recurring traffic justifies, sharpening or relocating instructions that production failures indict, and deleting sections that no longer change behavior, with every change gated on replayed traces before it ships. It is the prompt-layer counterpart of tool harness improvement and skill harness improvement — the same discipline pointed at the instructions — and it is not a return to prompt-only iteration: prompt engineering stays one component inside harness engineering, held to the same evidence standard as every other layer. Moda supplies that evidence layer: prompt-owned failures detected across production traces, attribution to the responsible section, and the replay gate that decides what ships. ### How do you auto curate skills from production sessions? Cluster production sessions by task, identify the recurring multi-step workflows the agent keeps re-deriving inside each cluster, distill each one into a draft SKILL.md with a name, a load-time description, and the procedure, then review it like any harness diff and gate it on replays of the source sessions against a baseline without it. Curation is the evidence standard: only skills that recur in real sessions and measurably improve their replays get released. Moda runs this pipeline as part of the product. ### Where do SKILL.md examples and the SKILL.md format fit into harness engineering? Under skill packaging. The SKILL.md format is small — markdown with a frontmatter name and description that tell the agent when to load the skill, followed by the procedure — and good SKILL.md examples are public, from Anthropic's open skills repository to the machine-readable skills this site ships at /skills/claude-code.md and /skills/moda-cli.md. In harness engineering the format is the easy part; the discipline is curating which skills to write from production sessions and verifying against replays that each one helps. ### How do I create a skill md file? Make a directory named after the skill (lowercase letters, numbers, hyphens) and put a SKILL.md file in it. The file starts with YAML frontmatter carrying two required fields — name, which must match the folder, and description, which says what the skill does and when to use it — followed by a plain markdown body with the procedure, examples, and edge cases. Optional scripts/, references/, and assets/ directories hold material the agent loads on demand. Then place the folder where your agent looks: Claude Code reads .claude/skills/ in the repository and ~/.claude/skills/ for personal skills. The mechanics take minutes; deciding which workflow deserves the file, and verifying the skill helps, is harness engineering — which is why Moda curates skill candidates from production sessions instead of writing them from memory. ### What is a skill.md specification? A small open format for packaging agent capabilities, published at agentskills.io: a skill is a directory containing a SKILL.md file with YAML frontmatter (a required name and description, plus optional license, compatibility, metadata, and allowed-tools fields) and a markdown instruction body, with optional scripts/, references/, and assets/ directories. Agents load skills progressively — name and description at startup, the body on activation, bundled files as needed. Anthropic's open skills repository supplies the best-known public examples, and multiple coding agents load the same files. The spec defines the packaging only; Moda uses it as the container for curated production competence, with the curation and replay verification supplying what the format deliberately leaves out. ### What's the difference between a skill, an MCP tool, and a system prompt? They are three different harness components with different loading semantics. A system prompt is always-on instructions: everything in it occupies context on every request, so it should carry only what is truly global. A skill — a SKILL.md file — is an on-demand packaged procedure: the agent sees only its name and description until a task matches, then loads the full procedure, which gives you recurring competence without permanent context cost. An MCP tool is an external capability: a function or API the agent can call through the Model Context Protocol, doing work outside the model rather than instructing it. Harness engineering treats the three as separate design surfaces. The common failure mode is dumping every fix into the system prompt because it is the only surface anyone edits — competence that recurs belongs in skills, capabilities belong in tools, and the prompt stays small enough to reason about. ### How do you improve skill.md from production failures? Attribute each failure to the part of the file responsible, patch that part, and replay-verify before shipping. Production failures indict one of three parts of a SKILL.md: the trigger (the frontmatter description no longer matches when the skill should load, so it stays cold or misfires), the steps (the procedure references tools, flags, or sequences that have drifted), or the checks (the procedure completes without the verification the task needs, producing wrong-but-successful sessions). Patch the indicted part as a reviewable diff with the failing sessions attached, then re-run those sessions under the patched skill against the unpatched baseline — targeted failures fixed, held-out sessions unregressed — before it ships. This is the repair half of the skill lifecycle; creating and curating new skills from sessions is covered separately on this page. ### What is skill harness improvement? Improving the skill layer of an agent harness from production evidence: adding skills that recurring sessions justify, repairing the trigger, steps, or checks of skills that production failures indict, and retiring skills that no longer earn their place — with every change gated on replayed sessions before it ships. It is a practice on a harness component you own, not a SKU and not a marketplace of downloadable skills. The framing follows the harness-engineering literature — Lilian Weng's harness engineering for self-improvement treats the harness as the optimization target that improves while the model stays fixed — and skills are the component of that target built to be edited. Moda supplies the evidence layer: session clustering, failure attribution to the responsible part of each SKILL.md, and the replay gate. ### Is harness engineering the same as continual learning? They describe the same loop from different altitudes. Continual learning is the research term for systems that keep improving after deployment without forgetting; harness engineering is the practice, for teams whose improvements land in prompts, tools, skills, evals, and memory rather than in model weights. If you are shipping the loop rather than publishing about it, harness engineering is the name for the work. ### Why does harness engineering come up around OpenAI and Anthropic? Because both labs' agent products are models inside heavily engineered harnesses, and both have now written that up first-hand: OpenAI's harness engineering post describes shipping a product with zero manually written code by engineering the environment instead, and Anthropic's harness design posts describe splitting long-running work across planner, generator, and evaluator components. The consistent finding is that harness decisions — tool design, context management, skills, evaluation — move agent quality as much as model choice. If the labs with the best models invest this heavily in the layer around them, the layer is where teams building on those models should look first. ### How does Moda fit in? Moda is a harness engineering platform. It maps your harness from the repository on every push, with citations back to source files; analyzes production traces for behavioral failures and emergent use cases; attributes each finding to the prompt, tool, skill, eval, or memory component responsible; and verifies proposed improvements against replayed production traffic before they ship. ## Keep reading - [What is an agent harness (pillar)](https://moda.dev/agent-harness) - [Agent harness (glossary)](https://moda.dev/glossary#agent-harness) - [What is an LLM router (pillar)](https://moda.dev/llm-router) - [Continual learning (pillar)](https://moda.dev/continual-learning) - [Self-improving agents (pillar)](https://moda.dev/self-improving-agents) - [Behavioral failure (glossary)](https://moda.dev/glossary#behavioral-failure) - [Moda vs LangSmith](https://moda.dev/vs/langsmith) - [Moda vs Raindrop](https://moda.dev/vs/raindrop) ## Moda skills for AI agents Moda is a harness engineering platform: it turns production traces into verified improvements for the agent harness — prompts, tools, skills, evals, memory. Machine-readable artifacts for agents working with Moda: - Claude Code integration skill: https://moda.dev/skills/claude-code.md - Moda CLI skill: https://moda.dev/skills/moda-cli.md - Node.js SDK integration skill: https://moda.dev/skills/sdk-node.md - Python SDK integration skill: https://moda.dev/skills/sdk-python.md - Agent skills index: https://moda.dev/.well-known/agent-skills/index.json - LLM reference: https://moda.dev/llms.txt ## See it on your traffic Book a demo: https://cal.com/team/moda/demo-meeting?overlayCalendar=true