Pillar guide
Continual learning
A working definition of continual learning, the two schools of thought behind it, what catastrophic forgetting actually is, and how the loop runs for AI agents in production.
Definition
What is continual learning?
Continual learning is the ability of a machine learning system to keep acquiring new knowledge and skills after it has been deployed, incorporating new data over time without losing what it already learned. That last constraint is the hard part: naive updates overwrite prior knowledge, a failure known as catastrophic forgetting.
The term comes out of the neural-network literature, where it describes training a network on a sequence of tasks and measuring how much of task one survives task five. That research question is still open, and it is still the reason the word exists. But over the last two years the phrase has been picked up by people shipping AI agents, who mean something adjacent but not identical: a system that gets measurably better at its job the longer it runs in production. Both usages are legitimate. They differ mainly in where the learning is stored.
This guide covers both. It starts with the general definition and the classic problem — catastrophic forgetting — because that is what the term means in the literature and what most people are actually asking about. It then covers the split that matters in practice: whether a system learns by changing its model weights or by changing everything around the model. Neither answer is universally right, and the honest version of this argument includes the cases where retraining wins.
We build in this space. Moda is a continual learning layer for AI agents, and it sits firmly on one side of that split. Where our position is doing the talking rather than the evidence, we say so.
Updated
Foundations
What is catastrophic forgetting?
Catastrophic forgetting is what happens when you train a neural network on new data and it gets worse at things it used to do well. It is not a subtle degradation. A network fine-tuned on a new task can drop from strong performance on its original task to near-chance in a single training run.
The mechanism is straightforward once you look at it. A network's knowledge is distributed across its weights — there is no cell holding "how to classify a cat" that gradient descent can route around. When you train on new examples, the optimizer moves weights toward whatever reduces loss on those examples. It has no term in its objective for preserving performance on data it can no longer see. The weights that encoded the old task get overwritten because nothing was defending them.
The literature frames this as a stability-plasticity trade-off: a system rigid enough to retain old knowledge is usually too rigid to absorb new knowledge, and vice versa. Decades of research have attacked it from three broad directions — replaying old data alongside new (rehearsal), penalizing changes to weights identified as important for prior tasks (regularization, of which Elastic Weight Consolidation is the best-known example), and allocating separate parameters per task so the tasks cannot collide (architectural methods). Each works within limits. None has made the problem go away.
This matters for the rest of this page for one reason: catastrophic forgetting is a property of learning in weights. A system that stores what it learned somewhere other than the weights does not have this failure mode. It has different ones, covered below — but not this one.
Sourcesvan de Ven et al., Continual Learning and Catastrophic Forgetting (arXiv:2403.05175) · Wang et al., A Comprehensive Survey of Continual Learning (arXiv:2302.00487)
The split
The two schools: learning in weights vs. learning in the harness
Ask two people what continual learning means and you will get two answers, depending on which of these they work on. The disagreement is not really about definitions. It is about where a learned thing should live.
The weight-based school treats continual learning as a training problem. The system learns by updating parameters — supervised fine-tuning on new examples, reinforcement learning against a reward signal, or newer approaches like sparse memory layers and test-time training. This is the school that owns the term historically, and it is where the interesting recent results are. Cameron Wolfe's survey of RL-based continual learning for LLMs makes a specific and useful claim here: across several papers, RL forgets substantially less than supervised fine-tuning on the same data. One study he reviews reports under a 1% average accuracy drop for RL against nearly 30% for SFT. The proposed mechanism is that on-policy data keeps the updated model close to the base model in KL divergence, so there is simply less distribution shift to forget across.
Andreessen Horowitz's essay "Why We Need Continual Learning" argues the strong version of this position. Aubakirova and Bornstein lay out a spectrum by where compression happens: at the context end, frozen weights plus retrieval and agent scaffolding; in the middle, attachable modules and adapters; at the weights end, genuine parametric learning after deployment. Their argument is that the context end is nearing its ceiling — "a bigger filing cabinet is still a filing cabinet" — and that the unlock is letting models do after deployment what made them powerful during training. We disagree on the timeline, but it is the clearest published version of the weight-side case.
The harness-layer school treats continual learning as a systems problem. The model stays frozen. What changes is everything around it: system prompts, tool definitions and schemas, workflow structure, retrieval indices, memory, skills, and eval sets. LangChain's Harrison Chase splits this into three layers — model, harness, and context — where the harness is the code, instructions, and tools shared by every instance of the agent, and context is the additional instructions and skills that configure it per user or per organization. Traces are the raw material for all three.
The practical difference is what a "learning" is made of. In the weight school it is a parameter delta: fast at inference, but opaque, tied to one model, and expensive to produce or revert. In the harness school it is a diff a human can read — a paragraph added to a system prompt, a tool argument made required, a retrieval index extended to cover a topic it was missing. That diff survives a model swap, can be reviewed before it ships, and reverts with a rollback rather than a checkpoint promotion.
SourcesWolfe, Continual Learning with RL for LLMs · Aubakirova & Bornstein, Why We Need Continual Learning (a16z) · Chase, Continual learning for AI agents (LangChain)
Honest tradeoffs
When weight updates are the right tool
Harness-layer learning is where most production teams should start, and this page argues that. It is not where every problem ends. There are cases where no amount of prompt engineering substitutes for moving the weights, and pretending otherwise wastes a quarter.
- The behavior is a capability, not an instruction. If the model cannot do the thing — a specialized reasoning format, a domain notation it has never seen, a language it handles poorly — telling it to try harder does not help. Train.
- The pattern is stable and high-volume. Once a behavior is well-defined, well-evaluated, and hit on most requests, folding it into weights buys latency and token cost that a long system prompt spends every call.
- Context cost dominates. A harness learning is paid for in tokens on every request. At sufficient scale and stability, distilling it into the model is straightforwardly cheaper.
- The task is narrow and the deployment is closed. A single model serving one well-scoped task, with no plan to swap providers, loses the portability argument against fine-tuning — and the fine-tune is likely a smaller, cheaper model.
- You want the smaller model. Distilling a large model's behavior into a small one is a training problem by construction. No harness edit shrinks a model.
The reverse also holds: harness edits are the wrong tool when the signal they would be built from is noise. Rewriting a prompt off three angry conversations is not continual learning, it is anecdote with extra steps. The bar for a harness change should be the same as the bar for a training run — enough evidence to believe the pattern is real, and an eval that catches it if you are wrong.
In practice the two schools compose rather than compete. Harness edits are the fast loop, shipping in hours against this week's production signal. Weight updates are the slow loop, consolidating patterns that have proven stable over months. A team running only the slow loop is blind between training runs. A team running only the fast loop eventually accumulates a system prompt nobody can reason about.
In production
Continual learning for AI agents in production
For a deployed agent, continual learning is a loop with five steps, and each one is a real engineering problem. Most teams that say they have continual learning have steps one and five and a person doing two through four by reading transcripts on Friday afternoon.
- Observe. Capture complete production conversations — every turn, tool call, argument, and result — not sampled logs. OpenTelemetry is the sane default here; Moda ingests OTLP directly so instrumentation is not vendor-specific.
- Detect. Find what went wrong behaviorally and what users were actually trying to do. This is the step that does not exist in most stacks, because it requires reading conversations at population scale rather than inspecting them one at a time.
- Generate. Turn detected failures into a concrete proposed change to a specific harness component — this prompt, this tool schema, this retrieval index, this skill.
- Validate. Run the proposed change against an eval set built from real production traffic, and check that fixing one cluster did not regress another.
- Ship. Deploy the harness change, tag it, and watch the cluster it was meant to fix. Revert if the numbers say to.
The detection step deserves detail, because the failures that matter are invisible to tracing. Every span in a broken conversation can return 200 OK. Moda detects against a behavioral failure taxonomy: tool misuse (right tool, subtly wrong arguments), context loss (the agent forgets or contradicts earlier turns), reasoning loops (the same failing action retried, or oscillation between answers), goal drift (the agent quietly solves a different problem than the one asked), hallucinations (claims not grounded in any tool result), and agent laziness (refusing or hedging on a task it can do). None of these are error states. All of them cost users.
Running alongside detection is intent clustering across the entire conversation population — not a sample. Conversations are segmented into topic-coherent slices, embedded, and clustered into a three-level Category → Subcategory → Cluster taxonomy with no manual labeling. That taxonomy is what makes a failure count interpretable: "tool misuse is up" is not actionable, "tool misuse is up 4x inside the deal-creation cluster, which is 11% of traffic" is a ticket. It also surfaces emergent intents — things users started asking for that nobody designed the agent to handle.
The output of the loop is a harness diff attributed to a layer: prompt, tool, workflow, context, memory, eval, or model. That last one is deliberate. When the evidence says the answer is a model change or a fine-tune, the honest thing is to route it there rather than pretend a prompt edit will do.
Evidence
What this looks like on real traffic
The argument above is only worth as much as the production evidence behind it. Octolane is an AI CRM — sales teams run their whole pipeline by talking to it — and by the time they came to us they had far more customer sessions than any human could read.
In the first few weeks, Moda analyzed 43,000 chat sessions and Octolane shipped five product fixes off the result. None of those fixes were a fine-tune. All of them were changes to the harness and the product around it.
The clearest example is one that would never have reached a support ticket. Users were creating deals successfully — no errors, no failed calls, no complaints. But across sessions the same hesitation showed up: after creating a deal, users were unsure what had actually been captured. Nothing was broken. The workflow was quietly costing people confidence. That pattern was only visible because every session pointed the same direction at once. Octolane added a confirmation step so users could review what was captured before completing the workflow.
The operating change mattered more than any single fix. Octolane's product, engineering, and GTM teams now review the week's patterns together every Friday, looking at the same clusters instead of arguing from whichever transcript each of them happened to read. That is what the loop looks like when it works — not a model that mysteriously improves, but a team that finds out what is wrong faster than users churn over it.
SourceHow Octolane turned 43,000 AI chat sessions into 5 shipped product fixes
Practice
How to implement continual learning
This is vendor-neutral until the last step. A team can build most of this loop themselves, and understanding what it takes is the best way to decide which parts are worth buying.
- Instrument with OpenTelemetry, and capture whole conversations. The common mistake is logging model calls without the thread that connects them. Behavioral failures are defined at the conversation level; a per-call log cannot represent context loss or goal drift because neither exists in a single span. Use OTLP so the instrumentation outlives whatever you point it at.
- Build eval sets from production, not from imagination. Hand-written evals encode what you thought users would do. Pull failing conversations out of production, cluster them, and turn the clusters into cases. An eval suite that stops resembling live traffic will pass a change that regresses real users.
- Put learnings in explicit, versioned artifacts. Whether it is a skills file, a memory store, a retrieval index, or a prompt fragment, the learned thing should be an object in version control with an author and a diff. If you cannot answer "why does the agent do this?" by reading something, nobody can debug it.
- Attribute failures to a layer before you fix them. The same symptom — the agent gave a wrong answer — routes to a prompt fix, a tool schema fix, a retrieval fix, or a model change depending on cause. Teams that skip attribution fix symptoms in whichever layer they happen to be looking at.
- Close the loop with a revert path. Every harness change should be revertible in the time it takes to redeploy. If it is not, you will hesitate to ship, and the loop stops being continual.
Where Moda fits is the middle three steps. Instrumentation is OTel and shipping is your deploy pipeline; neither needs us. Detection, generation, and validation are the parts that are genuinely hard to build: population-scale intent clustering, a behavioral failure taxonomy that works across domains, and the attribution layer that maps an observed failure to the harness component responsible for it. We ingest OTLP traces, run that analysis over every conversation rather than a sample, and hand back proposed harness improvements with the evidence attached.
If you want to go deeper on the research side, ContinualAI is the best entry point — a non-profit that maintains a continual learning course, an open-source library (Avalanche), and a community that has been working on this since well before it became an agent-infrastructure topic.
Measurement
Metrics that tell you the loop is real
A continual learning loop that cannot be measured is a story people tell about their roadmap. These are the numbers that move when it works — and none of them are average sentiment or session length.
- Intent coverage: the share of production conversations that fall inside an intent cluster the agent was designed to handle. Rising coverage means the harness is catching up to what users actually want.
- Emergent intent latency: how long between a new intent appearing in traffic and the harness covering it. This is the single best proxy for loop speed.
- Behavioral failure rate: incidents per 100 conversations, broken out by taxonomy class. Aggregate rates hide that fixing tool misuse can raise context loss.
- Time-to-fix: median time from detecting a failure to shipping a harness change against it.
- Regression rate: the share of harness changes that improved one cluster while degrading another. This is operational catastrophic forgetting — the metric most teams do not track until it bites them.
Risks
Failure modes to plan for
Every continual learning loop fails in roughly the same five ways. Knowing them in advance is worth a quarter of debugging.
- Reaching for a fine-tune first. The default assumption when an agent regresses is that the model is wrong. It usually is not — the harness has drifted out of alignment with how users now phrase things.
- Operational catastrophic forgetting. The weight-level problem has a harness-level twin: a prompt edit that fixes intent A regresses intent B, because the eval set never covered B. Same failure, different substrate.
- Reward hacking your own metrics. Optimizing for thumbs-up rates or session length reliably produces an agent that is pleasant and does not finish the job.
- Closed-loop selection bias. Learning only from users who complain means learning nothing from the larger group that quietly leaves. Population-scale analysis exists specifically to fix this.
- Update velocity outrunning measurement. Shipping harness changes faster than you can attribute their impact means you eventually cannot explain your own agent's behavior — the exact problem harness-layer learning was supposed to avoid.
Frequently asked
Questions
What is continual learning?
Continual learning is the ability of a machine learning system to keep acquiring new knowledge and skills after deployment, incorporating new data over time without losing what it already learned. The second half is the hard part: naive updates overwrite prior knowledge, a failure called catastrophic forgetting. In practice, systems achieve continual learning either by carefully updating model weights or by keeping the model frozen and updating the harness around it — prompts, tools, retrieval, memory, and skills.
Continual learning vs. continuous learning — what's the difference?
In machine learning the two terms are used interchangeably, and continual learning is the one the research literature settled on. The real confusion comes from a different field: in corporate learning and development, "continuous learning" means ongoing professional education for employees — courses, upskilling, training programs — and has nothing to do with machine learning. If that is what you were searching for, this is the wrong page. Everywhere else on this site, the term is used in the machine learning sense.
How is continual learning different from fine-tuning?
Fine-tuning is one technique for achieving continual learning, not a synonym for it. Fine-tuning updates model weights on new data in a discrete training run; continual learning is the broader goal of a system improving over time from ongoing experience. A single fine-tune is not continual learning — it is one update. And continual learning does not require fine-tuning at all: a system whose prompts, tools, retrieval, and memory are updated from production signal is learning continually with frozen weights.
Is continual learning the same as RLHF?
No. RLHF (reinforcement learning from human feedback) is a training method used to align a model to human preferences, typically as a phase during model development before release. Continual learning is about improvement after deployment, from ongoing production experience. The two intersect when a team runs RL against production signal on a recurring basis, which is a genuine form of weight-based continual learning — but standard RLHF is a one-time alignment step, not a loop.
What is catastrophic forgetting?
Catastrophic forgetting is when a neural network trained on new data loses performance on tasks it previously handled well — often dramatically, in a single training run. It happens because knowledge is distributed across the weights and the optimizer has no objective term protecting performance on data it can no longer see. Rehearsal, regularization, and architectural separation all reduce it. Keeping learnings outside the weights sidesteps it entirely, because nothing is being overwritten.
Do LLMs learn from conversations?
Not by default. A deployed LLM's weights are frozen: it does not remember your last conversation, and nothing you say changes the model for anyone else. What looks like learning is usually one of two things — in-context learning, where the model conditions on information placed in its context window for that request, or an explicit memory system that stores facts and retrieves them into context later. Both are real and useful. Neither modifies the model.
Continual learning vs. online learning — what's the difference?
Online learning means updating a model incrementally as each new example arrives, rather than retraining in batches. It is a training regime, defined by when updates happen. Continual learning is defined by a constraint: retaining old capabilities while acquiring new ones, typically across a sequence of distinct tasks or distributions. An online learner that quietly forgets last month's distribution is doing online learning and failing at continual learning.
How does Moda do continual learning?
Moda is the continual learning layer for AI agents, operating entirely on the harness rather than the weights. We ingest production conversations over OpenTelemetry, cluster the full population into an intent taxonomy, detect behavioral failures against a defined taxonomy (tool misuse, context loss, reasoning loops, goal drift, hallucinations, agent laziness), attribute each one to a specific harness component, and turn them into validated improvements. Because the learnings live outside the weights, they are portable across models, readable by humans, and revertible without a retraining run.
See continual learning on your traffic.
Moda turns production conversations into the production signal these loops need: intent clusters, behavioral failure exemplars, frustration root causes.