The Continual Learning Loop

The Transition to Lifelong Learning in AI Agents

A chatbot lived for one session. An agent now runs for months, and a model that stops learning at its last training step goes stale as the APIs, the codebase and the user's habits change.

Both of the old answers fail. Fine-tune once, and the knowledge freezes on that day. Retrain from scratch every few months, and every cycle costs offline compute, a curated mix of old and new data, and downtime, for every user you serve. Any update to the weights can also erase what the model already knew.

The continual learning loop sits between them. It updates the agent in small steps as feedback arrives, so every interaction becomes a training signal. The hard part is the balance. The agent has to stay plastic enough to learn the new thing and stable enough to keep the old ones.

Visual 1: The loop. Stages 1 to 7 repeat. The rest of this post follows it around once: what breaks at each stage, and how teams keep it stable.

The Architecture of Catastrophic Forgetting

Stage 5 is where things break. Train a network on task A, then take gradient steps on task B, and the weights move toward B alone. Knowledge in a transformer is spread across billions of shared parameters, so even a small update disturbs what A depended on. Performance on A drops, often sharply.

The damage sits in two places. Early attention heads lose focus, and their attention spreads over tokens that do not matter. Deeper MLP blocks and expert routers get overwritten, and with them the reasoning paths a task relied on.

Visual 2: Two failure regions, and a caveat: not every drop is lost knowledge.

That caveat is spurious forgetting. The first steps on a new task disturb the model's habit of activating the right functions before they erase any knowledge. Freezing the bottom layers keeps that habit. In the paper that named the effect, it lifted accuracy on old tasks from 11% to 44%.

To fight forgetting you have to measure it. Record $R_{i,j}$, the score on task $j$ after training through task $i$, and read three numbers off that matrix.

In plain terms: for each earlier task, compare its score at the end of the sequence with its score right after it was learned. A negative average is forgetting.

Mitigation Families: From Parameter Regularization to Replay Buffers

Every fix for forgetting belongs to one of four families.

Regularization makes it expensive to move the weights that mattered before. Elastic Weight Consolidation (EWC) is the original. It scores each weight's importance for task A with the Fisher information, then charges for moving the important ones.

In plain terms: train on task B as usual, but pay a penalty for moving any weight that mattered for task A, and pay the most for the weights that mattered most. $F_i$ is the diagonal of the Fisher information, estimated from squared gradients on task A's data.

The diagonal is an approximation, so it sometimes protects the wrong weights. "EWC Done Right" repairs the estimate. L2-SP is simpler and pulls every weight back toward the pretrained starting point.

Replay mixes old experience into the new training. That can be summaries of past runs in the context window, the most surprising outcomes replayed first, or a rehearsal schedule that follows a forgetting curve. When old data cannot be kept at all, self-distillation lets the model teach itself: the copy that sees a demonstration in context teaches the copy that does not.

Parameter isolation gives each task its own low-rank adapter (LoRA) and never touches the base model. O-LoRA keeps each new adapter orthogonal to the earlier ones, so a new task cannot overwrite an old direction. L-MoE treats the adapters as experts and lets a small gate mix them per token.

Visual 3: O-LoRA. The green update cannot disturb the purple subspace. The red one can.

Hypernetworks skip training altogether. Text-to-LoRA and Doc-to-LoRA read a task description or a document and write the adapter in a single forward pass.

Contextual Memory and Token-Space Learning

Not everything belongs in the weights. A user's current preference or the state of an open incident changes daily, and baking it into parameters is slow and hard to undo. Letta and Mem0 keep that kind of knowledge in a memory the agent manages itself, with tools to read, write, update and summarize its own notes.

Text memory is readable, works with any model, and cannot interfere with the weights. Forgetting is deleting a line.

Visual 4: Two lanes. The fast lane is text, the slow lane is gradients, and the base model stays frozen.

Reinforcement Learning with Verifiable Rewards (RLVR)

In the slow lane the training signal has changed too. The model is rewarded when a verifier says yes: a unit test passes, a script runs, a proof checks. That is Reinforcement Learning with Verifiable Rewards (RLVR), covered in the RLVR write-up. Continual RLVR runs it one task at a time instead of retraining on everything. Two things make that safe: online RL forgets less than fine-tuning, and a few old prompts replayed in each batch keep the old tasks alive.

Visual 5: One continual RLVR step, from the batch to the policy update.

The Production Agent Loop and Data Flywheels

Whatever the update rule, the rest of Visual 1 stays the same. NVIDIA's flywheel paper writes it as four verbs: monitor, analyze, plan, execute.

Monitor: collect traces, tool calls and outcomes. Analyze: filter them, because most telemetry is noise. Heuristics, a judge model and implicit signals such as a merged commit or a thumbs down do the sorting. Plan: turn each failure into a lesson. In the Reflexion pattern the agent, or a stronger model, writes down why the attempt failed and produces a corrected one. Execute: update the adapter or take an RLVR step, run the gate, redeploy.

The gains compound. An agent's hundredth code review is better than its first, because the loop has absorbed the codebase's style and edge cases.

Guardrails, Eval Suites, and Distribution Shift Detection

A loop that updates itself can also poison itself: bad data, drift away from its alignment, or slow forgetting that no single update shows. So every candidate runs against a golden set of core tasks. If any of them drops, the update is blocked and rolled back.

The loop also needs to know when an update is due. Watch the traffic.

In plain terms: bucket a feature, compare today's share of each bucket ($A_b$) with the baseline's share ($E_b$), and add up the mismatch. Below 0.1 is stable. Above 0.25 the traffic has moved and the model needs a refresh.

Infrastructure Shape: Multi-LoRA Serving and State Management

Gated updates produce hundreds of adapters, one per user or team, and each one must be served. One base model stays resident on the GPU and adapters are swapped in per request. PagedAttention keeps the KV cache in small blocks, like virtual memory pages, so unused reservations stop locking the GPU. A new adapter runs as a shadow on live traffic first and goes live only if it beats the current one without regressing.

How to Implement It in Claude

You cannot update Claude's weights, and you do not need to. The loop runs in the fast lane of Visual 4: the same seven stages, but stage 5 writes text, and that text is what Claude reads at the start of the next session.

In Claude Code, CLAUDE.md holds instructions you write, auto-memory holds notes Claude writes for itself, and Skills package a procedure it loads on demand. Hooks run a command at every tool call and at the end of a turn, which is where the loop collects its telemetry. In the Messages API, the memory tool gives Claude a /memories directory that your code stores. Managed Agents mount a memory store and version every edit. The self-improving agents write-up covers the governance around such a loop.

Visual 6: The same loop with the Claude pieces at each stage. Only stage 5 differs from Visual 1.

The skeleton in Python is short.

from anthropic import Anthropic
from anthropic.lib.tools import BetaAbstractMemoryTool

client = Anthropic()
memory = FileMemory("./memories")   # your BetaAbstractMemoryTool subclass over a directory

def serve(task):                    # stage 1: Claude works with its memory attached
    runner = client.beta.messages.tool_runner(
        model="claude-opus-5", max_tokens=16000, tools=[memory],
        messages=[{"role": "user", "content": task}],
    )
    return runner.until_done()

def learn(trace, passed):           # stages 3 to 5: a failed trace becomes one memory line
    if passed:
        return
    serve(f"This attempt failed:\n{trace}\n"
          "Write one lesson to /memories/lessons.md so the next attempt avoids it.")

def promote(frozen_eval):           # stages 6 and 7: the gate, then commit or revert
    if score(frozen_eval, "./memories") < score(frozen_eval, "./memories@HEAD"):
        git("checkout", "--", "./memories")   # rollback is a diff, not a checkpoint restore
    else:
        git("commit", "-am", "learned from production")

The tool type is memory_20250818 and needs no beta header. FileMemory, score and git are yours: a directory-backed memory class, the frozen eval runner, and a shell-out. The judge in stage 3 can be a second Claude call with the transcript and a rubric.

What We Learned

• The loop is the product. Deploy, collect, filter, reflect, update, gate, redeploy.
• Forgetting has a map. Early attention loses focus, deep layers get overwritten, and the first drop is lost alignment, so freeze the bottom.
• Measure it. Backward transfer is the number to watch. Negative means forgetting.
• Four families of fixes. Regularize, replay, isolate, or generate the adapter.
• Two lanes. Text memory for what changes daily, adapters for what should stick.
• Continual RLVR forgets less, and a few replayed prompts keep old tasks alive.
• Gate every update with a golden set, PSI on the traffic, and a shadow rollout.
• In Claude, stage 5 writes text. Hooks collect, Claude judges, memory files hold the lesson, git rolls it back.

Sources and further reading

• Forgetting and its metrics: Gradient Episodic Memory (BWT, FWT, ACC), Spurious Forgetting
• Regularization: EWC, EWC Done Right, L2-SP
• Replay: Contextual Experience Replay, SuRe, FOREVER, Self-Distillation Enables Continual Learning
• Isolation and hypernetworks: O-LoRA, L-MoE, Text-to-LoRA, Doc-to-LoRA
• Token space: Letta on continual learning in token space, Mem0
• Continual RLVR: Continual Reasoning Gym, RL's Razor, DAPO, Intuitor, Reflexion
• Production loops: Adaptive Data Flywheel, Cursor Tab online RL, Llama 4, Stripe's payments model
• Serving: PagedAttention
• Claude: the memory tool, Claude Code memory, hooks, Agent Skills