Pillar guide
Harness engineering
The discipline of improving AI agents by engineering the harness around the model — what it covers, how it differs from prompt engineering and observability, and what the working loop looks like in production.
Definition
What is harness engineering?
Harness engineering is the discipline of improving an AI agent by engineering everything around the model — prompts, tools, skills, evals, and memory — using evidence from production. The model is treated as a fixed component; the harness is the system you design, measure, and iterate. It is to agents what site reliability engineering was to infrastructure: the practice that turns keeping the thing working from heroics into a discipline.
The term is showing up in engineering discussions around OpenAI and Anthropic for a concrete reason: the flagship agent products of both labs — Codex, Claude Code — are models inside heavily engineered harnesses, and the people building them keep finding that harness decisions move agent quality as much as model choice does. The same model ships with different tools, context management, and guardrails and becomes a noticeably different product. Once you see that, the question changes from which model is best to who is engineering our harness — and for most teams running an agent in production, the honest answer is no one, or everyone, in scattered prompt edits after each incident.
Harness engineering names that scattered work and makes it a practice with a loop, an evidence standard, and a definition of done. This guide covers what the practice consists of, how it differs from the two things it is most often confused with — prompt engineering and observability — and how the loop runs when it works.
Updated
The practice
What harness engineering covers
The harness is every part of an agent system that is not model weights: prompts, tools, skills, evals, memory, orchestration. Harness engineering is the deliberate improvement of those parts against production evidence. Concretely, the recurring work looks like this:
- Maintaining a current map of the harness — which agents exist, with which prompts, tools, and skills — so changes are made against the system as it is, not as someone remembers it.
- Attributing production failures to the component responsible: this tool schema invites wrong arguments, this prompt section no longer covers what users ask, this memory lookup retrieves the wrong context.
- Tightening tool definitions and routing so the model's cheapest interpretation of a schema is also the correct one.
- Building and refreshing eval sets from real traffic, so the gate a change must pass resembles the traffic it will meet.
- Packaging recurring competence into skills instead of letting every fix become another paragraph in an ever-growing system prompt.
- Verifying each change against replays of production conversations before it ships, and reverting when the numbers say to.
Contrast
Harness engineering vs prompt engineering
Prompt engineering is one component of the job, and treating it as the whole job is how teams end up with a two-thousand-line system prompt that nobody can safely edit.
The difference is scope and evidence. Prompt engineering iterates on one artifact, usually against intuition and a handful of test cases. Harness engineering treats the whole assembly as the design surface and asks, for each observed failure, which component is actually responsible. Many failures that look like prompt problems are not: an agent that picks the wrong tool needs a sharper tool description, not a longer prompt; an agent that forgets what the user said needs a memory fix; an agent that regresses on a change needs an eval that would have caught it.
A useful test: if your improvement process cannot conclude that the prompt is fine and the tool schema is broken, you are doing prompt engineering. The prompt is where every fix lands when it is the only surface you look at.
Contrast
Harness engineering vs observability
Observability is the input, not the practice. Traces, logs, and dashboards tell you what the agent did; they do not tell you what to change, and they do not verify that a change worked.
Agent observability tooling stops at visibility: a trace shows every span green while the user leaves frustrated, because behavioral failures — wrong tool arguments that return 200, context quietly lost, goals silently drifted — are invisible at the span level. Observability is necessary; everything downstream of it is where the improvement happens. Harness engineering is that downstream: detection of behavioral failures across the conversation population, attribution to the responsible component, a proposed change, and verification against real traffic before shipping.
The relationship to continual learning is the same from the other direction: continual learning is the research name for systems that improve after deployment without retraining, and harness engineering is what the practice looks like when the improvements land in the harness rather than the weights. Observability feeds it; research named it; the engineering is the job.
The loop
The harness engineering loop
Run as a practice, harness engineering is a five-step loop. Each step is real engineering work, and most teams currently do steps one and five with a human reading transcripts in between.
- Map: build and maintain a versioned model of the harness from the repository — agents, prompts, tools, skills, evals, and how they connect.
- Observe: capture complete production conversations, not sampled spans. OpenTelemetry-based ingest keeps the instrumentation vendor-neutral.
- Attribute: detect behavioral failures and emergent intents across the whole population, and route each to the harness component responsible.
- Improve: turn the attribution into a concrete diff — a prompt edit, a tool schema fix, a new skill, an expanded eval set.
- Verify: replay the change against real production traffic and compare before shipping; revert if a fixed cluster cost you a different one.
This is the loop Moda runs as a product. The harness map syncs from GitHub on every push to the default branch, with citations back to source files. Production conversations are analyzed for behavioral failures and use-case clusters, each attributed to a harness component. Proposed changes are checked in a playground that compares production and proposed configurations over replayed conversations before anything ships.
Adoption
Who needs harness engineering, and when
The practice earns its cost at a specific point: when an agent is in production, users depend on it, and the volume of conversations exceeds what anyone reads by hand.
Before that point, the founder reads every transcript and the harness fits in their head; formalizing the loop is premature. After it, the failure mode is predictable: improvement decisions get made from the loudest complaint rather than the largest pattern, every fix lands in the system prompt because it is the only visible surface, and nobody can say whether last month's changes helped. Teams usually arrive at harness engineering after noticing that they benchmark models rigorously and change their harness by vibes — while the harness is the part they actually control.
The buy-vs-build question is about the middle of the loop. Instrumentation is a solved problem and shipping is your deploy pipeline. Population-scale failure detection, component-level attribution, and replay-based verification are the parts that are genuinely hard to build in-house — that is the part Moda sells, and the demo is the fastest way to see it against your own kind of traffic.
Frequently asked
Questions
What is harness engineering?
Harness engineering is the discipline of improving an AI agent by engineering everything around the model — prompts, tools, skills, evals, memory — using evidence from production. The model is treated as a fixed component; the harness is the system you design, measure, and iterate. The working loop is: map the harness, observe production, attribute failures to the responsible component, improve it, and verify the change against real traffic.
How is harness engineering different from prompt engineering?
Prompt engineering iterates on one artifact, usually by intuition. Harness engineering treats the whole assembly — prompts, tools, skills, evals, memory — as the design surface and attributes each production failure to the component actually responsible. Many failures that look like prompt problems are tool schema, memory, or eval problems; a practice that can only edit the prompt fixes them all in the wrong place.
How is harness engineering different from observability?
Observability tells you what the agent did; harness engineering decides what to change and verifies that it worked. Traces and dashboards are the input. The practice is everything downstream: detecting behavioral failures traces miss (wrong-but-successful tool calls, context loss, goal drift), attributing them to a harness component, and replaying proposed changes against production traffic before shipping.
Is harness engineering the same as continual learning?
They describe the same loop from different altitudes. Continual learning is the research term for systems that keep improving after deployment without forgetting; harness engineering is the practice, for teams whose improvements land in prompts, tools, skills, evals, and memory rather than in model weights. If you are shipping the loop rather than publishing about it, harness engineering is the name for the work.
Why does harness engineering come up around OpenAI and Anthropic?
Because both labs' agent products are models inside heavily engineered harnesses, and their engineers consistently report that harness decisions — tool design, context management, skills — move agent quality as much as model choice. That framing has spread: if the labs with the best models invest this heavily in the layer around them, the layer is where teams building on those models should look first.
How does Moda fit in?
Moda is a harness engineering platform. It maps your harness from the repository on every push, with citations back to source files; analyzes production conversations for behavioral failures and emergent use cases; attributes each finding to the prompt, tool, skill, eval, or memory component responsible; and verifies proposed improvements against replayed production traffic before they ship.
See harness engineering on your traffic.
Moda turns production conversations into the production signal these loops need: intent clusters, behavioral failure exemplars, frustration root causes.