Pillar guide
What is an agent harness?
A plain-English definition of the agent harness in AI systems: the parts it is made of, why the harness rather than the model is usually what fails in production, and how teams map and improve theirs.
Definition
The short answer
An agent harness is everything wrapped around a language model to turn it into a working AI agent: the system prompts that frame it, the tools it can call, the skills and workflows that structure its work, the evals that gate changes, and the memory it carries between turns. The model supplies capability; the harness decides how that capability behaves in production.
The word harness borrows from test engineering, where a test harness is the fixture that holds a component in place so you can drive it and observe it. An agent harness holds a model the same way. The model itself is frozen and general-purpose; everything that makes it behave like your agent — knowing your product, calling your APIs, following your escalation policy — lives in the code and configuration around it. Two teams can deploy the identical model and get agents that behave nothing alike, because the harness is where the behavior actually comes from.
That makes the harness the highest-leverage surface in an agent system, and also the least examined one. Teams benchmark models for weeks and then assemble the prompts, tools, and memory around the winner in an afternoon. This guide defines each part of the harness, explains why production failures usually trace back to it rather than to the model, and covers how to map and improve one systematically.
Updated
Anatomy
What an agent harness is made of
Every production agent harness is some combination of six kinds of parts. Frameworks name them differently; the responsibilities are stable.
- Prompts: the system prompt and its fragments — persona, policy, formatting rules, injected context. The densest concentration of behavior per byte anywhere in the system.
- Tools: the functions the model can call, defined by schemas. A tool's name, description, and argument types are instructions to the model as much as they are an API contract.
- Skills: packaged procedures the agent can load for a class of task — instructions, examples, and sometimes scripts. Skills are how a harness accumulates competence without every prompt growing forever.
- Evals: the test cases that gate changes. An eval set built from real traffic is the difference between knowing a harness change is safe and hoping it is.
- Memory: what the agent carries across turns and sessions — conversation state, user facts, retrieval indices. Memory decides what the model sees, which decides what it does.
- Orchestration: the loop that binds the rest together — turn structure, retries, sub-agent routing, guardrails on input and output.
None of these parts are model weights. That is the defining property of the harness: every component is code or configuration a human can read, diff, review, and revert. When the agent improves, the improvement is visible in version control.
The claim
The harness, not the model, is usually what fails
When a production agent gives a wrong answer, the reflex is to blame the model. The evidence usually points elsewhere: the model did exactly what the harness told it to do, and the harness told it the wrong thing.
The failure signatures are consistent across teams. A tool description is ambiguous, so the model passes subtly wrong arguments — every call returns 200, and the result is quietly wrong. The system prompt was written for last quarter's users and does not cover what people ask now. Memory retrieves the wrong context, and the model reasons flawlessly from a false premise. An eval suite full of hand-written cases passes while real traffic regresses. None of these are model failures, and no amount of model upgrading fixes them.
This is also why the same model scores differently under different harnesses on agentic benchmarks: harness design — which tools are exposed, how errors are surfaced, how context is managed — is part of what is being measured. A model swap changes an agent less than most teams expect, and a harness change usually changes it more.
Disambiguation
Agent harness, LLM harness, eval harness — which one do you mean?
The word harness is overloaded in AI, and the meanings are related but not interchangeable. If you searched for one of these, here is the map.
- Agent harness (this page): the production scaffolding around a model — prompts, tools, skills, evals, memory, orchestration — that turns it into a deployed AI agent.
- LLM harness / eval harness: a benchmarking fixture that runs a model against standardized tasks under controlled settings. EleutherAI's lm-evaluation-harness is the canonical example and is where many people first meet the word.
- Test harness (general software): the older term both of the above borrow from — the fixture that mounts a component so it can be exercised and observed.
The senses connect: an eval harness holds a model still so you can measure it; an agent harness holds a model in production so it can do work. And a well-run agent harness contains an eval harness inside it — the eval component exists to measure the rest. (If you were looking for wiring harnesses, this page is about AI systems, not automotive electrical assemblies.)
Entropy
Why harnesses drift in production
A harness is aligned with reality on the day it ships and drifts from that point forward, because the things it encodes assumptions about keep moving.
- Users change: new intents show up in traffic that no prompt, tool, or skill was designed to handle, and the agent improvises badly.
- The product changes: an API adds a required field, a workflow gains a step, and a tool schema written six months ago silently stops matching reality.
- The model changes: a provider upgrade shifts instruction-following in ways that break prompt assumptions that used to hold.
- The harness itself accretes: every incident adds a paragraph to the system prompt, and nobody deletes anything, because nobody can prove what is load-bearing.
Drift is invisible in dashboards that track latency and error rates, because a drifted harness returns successful responses to the wrong effect. It shows up first in behavior: repeated retries, users rephrasing, quiet abandonment. Catching it requires reading production conversations at population scale — which is exactly the work nobody has time to do by hand.
Practice
Mapping and improving a harness
You cannot improve a harness you cannot see. Most teams have no artifact that answers the basic question: what agents do we run, with what prompts, tools, and skills, and how do they relate?
The map has to come from the repository, because that is where the harness lives. Moda builds it there: connect the GitHub App, and every push to the default branch triggers an analysis that captures the agents in the codebase, the artifacts they use (prompts, tools, skills, evals, model configurations), and the relationships between them, with citations back to source files. The result is a versioned topology of the harness that stays current as the code changes.
Improvement then has a defined shape: production traces come in, failures and emergent intents are detected and attributed to the specific harness component responsible — this prompt, this tool schema, this missing skill — and each proposed change is verified against replays of real traffic before it ships. That loop, run continuously, is what this site calls harness engineering; the companion guide covers it as a practice.
Frequently asked
Questions
What is an agent harness?
An agent harness is everything wrapped around a language model to turn it into a working AI agent: system prompts, tool definitions, skills, evals, memory, and the orchestration loop that binds them. The model supplies raw capability; the harness determines how the agent actually behaves — which is why two agents on the identical model can behave nothing alike.
What is a harness in AI, generally?
The term borrows from test engineering, where a harness is the fixture that holds a component so it can be driven and observed. In AI it has two common senses: an eval harness (like EleutherAI's lm-evaluation-harness) holds a model still so it can be benchmarked, and an agent harness holds a model in production so it can do useful work. Both are scaffolding around a model, built for different purposes.
Is an agent harness the same as an agent framework?
No. A framework (LangChain, CrewAI, the Claude Agent SDK) is the toolkit you might build a harness with. The harness is your specific assembly: your prompts, your tool schemas, your skills, your eval set, your memory design. Two teams on the same framework have completely different harnesses, and the differences are where their agents' behavior diverges.
Is the harness the same as agent scaffolding?
Effectively yes — scaffolding is a common synonym for the same layer. Harness has become the more precise term because it carries the test-engineering connotation of holding a component so it can be observed and exercised, which is exactly the relationship this layer has to the model.
Why do agent failures trace to the harness instead of the model?
Because the harness encodes all the assumptions specific to your product and users — what tools exist, what the prompt covers, what memory retrieves — and those assumptions drift as users, product, and models change. Ambiguous tool descriptions, stale prompts, and wrong retrieved context all produce failures with perfectly healthy API calls. Model upgrades fix none of them.
How does Moda map an agent harness?
From the repository. With the Moda GitHub App connected, every push to the default branch triggers an analysis that captures the repo's agents, their artifacts (prompts, tools, skills, evals, model configurations), and the relationships between them, with citations back to source files. The dashboard shows the harness as an interactive topology graph, versioned on every sync — and production signals are attributed back to the specific harness component that needs to change.
See agent harness on your traffic.
Moda turns production conversations into the production signal these loops need: intent clusters, behavioral failure exemplars, frustration root causes.