The harness engineering glossary

Agent harness

Also known as: Agent harness layer, Agent scaffolding

Everything that wraps the model call to produce agent behavior: system prompts, tool definitions and routing, workflow orchestration, retrieval, memory, evals, and guardrails.

The agent harness is the layer between the user and the model — what makes an agent an agent rather than a chat completion. It includes the system prompt that frames the agent, tool schemas and routing logic, multi-step workflow definitions, retrieval indices, memory state and how it is summoned per turn, evals that gate ships, and guardrails on input and output. Most production agent improvement happens on the harness layer, not on the model weights, because harness edits are fast to ship, easy to inspect, easy to reverse, and apply across whichever model you mount underneath. Moda is self-improvement for AI agents on the harness layer: every behavioral failure and frustration event is attributed to a specific harness component (prompt, tool, workflow, context, memory, eval, or model), and the resulting learnings live in a latent space outside the model weights so they apply across models and adapt per user.

RelatedContinual learning, Self-improving agent, Behavioral failure, Moda

Read the guide: What is an agent harness →

Agent laziness

A behavioral failure where the agent declines, hedges, or gives a generic answer to a task it is capable of completing.

Agent laziness is a class of behavioral failure in AI agents where the model returns 'I cannot help with that', refuses an in-scope request, or substitutes a high-level non-answer when the user asked for specifics. It usually traces back to overly cautious system prompts, safety filters tuned too aggressively, or instruction-following regressions after a model upgrade. Moda detects laziness by comparing the user's stated goal to the agent's trajectory and flagging mismatches at the conversation level.

RelatedBehavioral failure, Goal drift, Frustration root cause

AI agent observability

Also known as: Agent observability, Agentic observability

In a harness engineering workflow, the input layer: the conversation-level record of what production agents did. The record only pays off when it becomes a specific harness change.

Harness engineering treats AI agent observability as raw material, not a destination. The term covers recording and understanding what production agents are doing at the conversation level — which intents users bring, where trajectories break, how users experience the agent — rather than only the call- or token-level view that traditional APM and LLM tracing record. But watching that record does not fix an agent; the fix ships in the harness — a revised prompt, a tightened tool schema, a new skill, an updated eval, a memory change. Moda is not an observability platform: it consumes the traces this layer produces and turns them into verified harness improvements, each attributed to the harness component that caused the failure.

RelatedAgent harness, LLM observability, Behavioral failure, Intent discovery

Behavioral failure

A failure mode that occurs at the conversation level, not at the API call level — invisible to traces alone.

A behavioral failure is when an AI agent does the wrong thing while every individual API call returns 200 OK. Examples include calling the right tool with subtly wrong arguments (tool misuse), forgetting earlier turns (context loss), refusing in-scope tasks (agent laziness), fabricating facts (hallucinations), retrying the same failed action (reasoning loops), and silently abandoning the user's objective (goal drift). Detecting behavioral failures requires comparing trajectories and outcomes, not just status codes.

RelatedTool misuse, Context loss, Agent laziness, Hallucination, Reasoning loops, Goal drift

Cluster hierarchy

Also known as: Cluster Hierarchy V2, Hierarchical clustering

Three-level segment clustering — Category → Subcategory → Cluster — built with HDBSCAN over Qwen3 embeddings.

Moda's cluster hierarchy groups conversation segments into a three-level taxonomy: broad categories, mid-level subcategories, and tight clusters. The hierarchy is built with HDBSCAN over 4096-D Qwen3-Embedding-8B embeddings, with UMAP projection for visualization. New segments are assigned online via kNN so the taxonomy stays current without rerunning the full clustering job. Cluster labels are generated with TF-IDF plus Claude Haiku 4.5.

RelatedIntent discovery, Segment, Conversation analytics

Context loss

A behavioral failure where the agent forgets earlier turns, contradicts itself, or asks the user to re-supply known information.

Context loss happens when an AI agent loses track of information from earlier in the conversation — forgetting user preferences, repeating questions already answered, or contradicting prior statements. It is common in long sessions, multi-step workflows, and conversations that exceed the model's effective attention window. Context loss is a behavioral failure: every individual call may succeed, but the conversation as a whole degrades.

RelatedBehavioral failure, Frustration root cause

Continual learning

Also known as: Continuous learning, Learning loop, Harness-layer learning

Harness-layer improvement from production signal: updating prompts, tools, workflows, retrieval, memory, and evals so the learnings live outside the model weights and apply across whichever model the harness mounts.

The agent harness is where continual learning (sometimes called continuous learning) actually happens in production: revised system prompts, retightened tool schemas, restructured workflows, expanded retrieval indices, new skills, updated eval sets, and per-user memory. The term describes closing the loop between what an agent does in production and how the agent is built. Model fine-tunes are an option but rarely the first step — they are heavier, harder to inspect, and bound to a specific model. Keeping learnings outside the model weights makes them portable across models, inspectable, reversible, per-user adaptable, and continuously updatable without retraining. Moda is not a continual learning system; it is a harness engineering platform that does the work this loop needs — turning production traces into verified improvements to the harness.

RelatedAgent harness, Self-improving agent, Intent discovery, Behavioral failure, Moda

Read the guide: Continual learning →

Conversation analytics

Also known as: LLM conversation analytics, Conversation intelligence

Analytics that treat full agent conversations — not individual calls — as the unit of analysis.

Conversation analytics for AI agents measures population-level behavior across every interaction: what users are trying to do, how often they succeed, where they get stuck, and which intents correlate with churn or escalation. Unlike per-call telemetry, conversation analytics requires segmenting, clustering, and labeling natural-language interactions. Moda automates this with hierarchical clustering of conversation segments.

RelatedIntent discovery, AI agent observability, Cluster hierarchy

Frustration root cause

Also known as: Frustration detection

For every detected frustration event, the trigger turn, leading trajectory, affected user goal, and counterfactual for what the agent should have done.

Frustration root cause goes beyond sentiment scoring. When Moda detects a frustration event in a conversation, we attach four things: the trigger turn that flipped sentiment, the trajectory leading up to it, the user goal that was at risk, and a counterfactual — what the agent should have done instead. The counterfactual is generated by a rationalizing language model (RLM) pass over the full conversation. The result is a debuggable signal, not a number on a chart.

RelatedBehavioral failure, AI agent observability

Goal drift

A behavioral failure where the agent silently abandons the user's original objective and solves a different problem.

Goal drift is when an AI agent quietly changes the problem it is solving partway through a conversation. The user asks for X, the agent helps with related-but-different Y, and the original objective is never met. Goal drift is detected by comparing initial user intent to final outcome at the conversation level — something traces of individual calls cannot see.

RelatedBehavioral failure, Intent discovery

Hallucination

When the agent fabricates facts, invents tool calls, or claims to have performed actions it did not perform.

In an AI agent context, a hallucination is any output that is presented as factual but is not grounded in real data, real tool calls, or the conversation history. Common forms include inventing API parameters, citing non-existent tools, fabricating customer records, and claiming completion of actions the agent never took. Moda treats hallucinations as a category of behavioral failure and surfaces them when the agent's claimed actions diverge from its actual tool calls.

RelatedBehavioral failure, Tool misuse

Intent discovery

Also known as: Intent clustering, Intent taxonomy

Hierarchical clustering of production conversations into a 3-level taxonomy with no manual tagging.

Intent discovery is the process of figuring out what users are actually trying to do with your agent, without anyone hand-labeling conversations. Moda's intent discovery embeds every segment with Qwen3-Embedding-8B (4096-D), clusters them with HDBSCAN, projects with UMAP, and labels clusters with TF-IDF plus Claude Haiku 4.5. The output is a Category → Subcategory → Cluster taxonomy of real user goals.

RelatedCluster hierarchy, Segment, Conversation analytics

LLM observability

Call- and trace-level monitoring of LLM applications — prompts, completions, tokens, latency, errors. It sits upstream of the harness layer: it records what happened; harness engineering decides what to change.

In a harness engineering workflow, LLM observability is the layer that produces the raw record: prompts, completions, token usage, latency, retries, and errors for every call. Tools like LangSmith and Langfuse own this layer, and it is essential for debugging individual requests. A record is not an improvement, though — the fix lands in the harness: prompts, tool definitions, skills, evals, memory. Moda is not an LLM observability tool; it sits downstream, ingesting the traces this layer emits (or raw telemetry over OTLP) and converting them into verified harness changes. Teams typically keep their tracing tool and add Moda on top of the same trace stream.

RelatedAgent harness, AI agent observability, LLM tracing

LLM tracing

Recording the step-by-step execution of an LLM application — every prompt, tool call, and response — for debugging.

LLM tracing captures the chronological steps of a single LLM-powered request: the prompts sent, the tool calls made, the intermediate reasoning, and the final response. Tracing is essential for debugging individual interactions but only sees one conversation at a time. To answer 'what are users doing across thousands of conversations' or 'which behavioral failure is most common,' tracing data has to be aggregated and clustered — the job of agent analytics.

RelatedLLM observability, AI agent observability

MCP server

Also known as: Model Context Protocol server

A server that exposes tools and resources to AI coding assistants via the Model Context Protocol.

An MCP (Model Context Protocol) server is a process that exposes a set of tools and resources to an MCP-aware client like Claude Code, Cursor, or Windsurf. Moda ships an MCP server for AI agent debugging: from inside your IDE you can pull failing conversations, inspect tool call failures, and ask Moda's data agent natural-language questions over your production conversation data.

RelatedAI agent observability, Moda

Moda

Moda is a harness engineering platform for teams with an AI agent already in production. It turns production traces into verified improvements for the agent harness: prompts, tools, skills, evals, memory.

Moda is a harness engineering platform. Teams with an agent already in production stream traces in through an OpenTelemetry-native pipeline; Moda clusters conversations into a hierarchy of user intents, detects behavioral failures — tool misuse, context loss, agent laziness, hallucinations, reasoning loops, goal drift — and attributes each one to the harness component responsible. The output is not a dashboard to watch: it is a verified change to a specific prompt, tool definition, skill, eval, or memory, and every frustration event ships with a root cause and an agent counterfactual. Moda is backed by Y Combinator and provider-agnostic across Anthropic, OpenAI, Google, OpenRouter, and any OTEL-compatible LLM stack.

RelatedAgent harness, Continual learning, Intent discovery, Behavioral failure

OpenLLMetry

An open-source SDK that emits OpenTelemetry-compatible traces for LLM and agent applications.

OpenLLMetry is an open-source instrumentation library that emits OpenTelemetry traces for LLM applications across major providers (OpenAI, Anthropic, Google, Cohere, Mistral, AWS Bedrock, Azure OpenAI) and agent frameworks (LangChain, LlamaIndex, Mastra). Moda ingests OpenLLMetry traces directly through its OTLP HTTP endpoint, so teams with existing instrumentation can keep their current telemetry path. New integrations can use the first-party Moda SDK instead.

RelatedOpenTelemetry, Zero-config ingest

OpenTelemetry

Also known as: OTEL, OTLP

The industry-standard open-source framework for emitting and collecting telemetry — traces, metrics, and logs.

OpenTelemetry (OTEL) is a CNCF project that defines a vendor-neutral wire format (OTLP) and SDKs for traces, metrics, and logs. Moda ingests AI agent telemetry via OTLP/HTTP, which means any application already instrumented with OpenTelemetry — directly or through OpenLLMetry — can stream production conversations into Moda without bespoke integration work.

RelatedOpenLLMetry, Zero-config ingest

Reasoning loops

A behavioral failure where the agent retries the same failed action, oscillates between answers, or fails to converge.

Reasoning loops occur when an AI agent gets stuck — repeating the same tool call after it has already failed, flipping between two competing answers, or running a chain-of-thought that never terminates. The user experience is delay, wasted tokens, and eventual abandonment. Moda detects reasoning loops by analyzing per-turn action sequences and surfacing cycles.

RelatedBehavioral failure, Tool misuse

Segment

A topic-coherent slice of a conversation, identified by cosine drift between consecutive turn-pair embeddings.

A segment is the unit of clustering in Moda. We embed turn pairs with Qwen3-Embedding-8B and walk the conversation looking for cosine drift between consecutive embeddings; when drift crosses threshold, we cut a segment boundary. Clustering segments — instead of whole conversations — keeps clusters topically tight and lets a single multi-topic conversation contribute to multiple intent clusters.

RelatedCluster hierarchy, Intent discovery

Self-improving agent

Also known as: Self-improving AI agent

An AI agent that updates its harness — prompts, tools, workflows, context, memory, evals — from production signal, in a latent space outside the model weights so improvements apply across whichever model the harness mounts and can adapt to each user.

A self-improving agent does not stop changing after deployment. Each production interaction becomes data: which intents are underserved, which tools misfire, which prompts trigger frustration, which corrections the user actually accepts. That signal feeds back into the harness — prompts, tool definitions, workflows, retrieval, memory state, evals, and routing — so the next version of the agent is materially better than the last. Keeping the learnings outside the model weights makes them portable across models, inspectable, reversible, and per-user adaptable. Moda provides the discovery and debug half of the loop and attributes every signal to a specific harness component, so the team knows exactly what to change next.

RelatedAgent harness, Continual learning, Intent discovery, Behavioral failure, Moda

Read the guide: Self-improving agents →

Tool call failure

An explicit failure of a tool call — timeout, error response, or invalid arguments — surfaced with error subtypes and trends.

A tool call failure is the surface-level cousin of tool misuse: the tool itself errored, timed out, or returned an invalid response. Moda automatically flags these across conversations, groups them by error subtype, plots hit-rate trends, and lets you drill into the affected conversations. Unlike tool misuse, tool call failures are visible to standard tracing — but they are not always visible across thousands of conversations at once.

RelatedTool misuse, Behavioral failure

Tool misuse

A behavioral failure where the agent calls the right tool with wrong arguments — or the wrong tool — while still returning a plausible response.

Tool misuse covers all the ways an agent can break a workflow without anything raising an error: passing subtly wrong arguments, choosing the wrong tool from a similar set, or producing a confident response that is not actually grounded in the tool's output. Detecting tool misuse requires understanding tool semantics — what the tool was supposed to do — not just whether it returned 200.

RelatedBehavioral failure, Tool call failure, Hallucination

Zero-config ingest

Three lines of SDK code plus OpenTelemetry-native intake — works with any LLM provider, no lock-in.

Zero-config ingest is Moda's onboarding promise: run moda init, let your coding agent integrate the Moda SDK, add your API key, and ship. Conversations stream into your workspace within seconds once telemetry is flowing. Existing OpenTelemetry or OpenLLMetry instrumentation can send traces over OTLP HTTP, and custom harnesses can POST raw JSON to /v1/ingest or /v1/ingest/multi. Hierarchical intent maps are generated during the weekly clustering run after at least 50 conversation segments have accumulated.

RelatedOpenTelemetry, OpenLLMetry