Renting Parameters: The Engineering, Science, and Political Economy of Continual Learning

0. Prologue

Continual learning isn't sexy. Not even a little. Before the Transformer paper came out, DeepMind was already arguing about catastrophic forgetting in PNAS: a neural network learns a new task and wipes out the old ability, so we have to find some way to hold on to the weights we care about¹.

After LLMs, nobody talked about the term for years. Suddenly everyone was pre-training, then everyone was doing reinforcement learning, then everyone was doing agents and harnesses, and outside of the community's complaining, the big labs didn't seem to care much whether a model was nice to use, or how well it suited its user. If you had to name the same idea, you'd say fine-tuning, adaptation, personalization, but it always carried a whiff of 2023.

Why, in mid-2026, is the word back?

Base models are pushing toward 10T parameters, and the frontier labs take turns on stage, a new flagship every few months, each refresh delivering almost exactly the same capabilities. Everyone is tired. A model that big, nobody can fine-tune it; a lineup turning over that fast, nobody has time to fine-tune². On the leaderboards they're unbeatable. Then you actually use one in your workflow, and who hasn't sworn at it a few times: "why do you understand nothing about me?"

So continual learning gets dug back up. The base models are good enough now. If one could adjust itself while in use, keep learning, and eventually really "grow into" my shape, wouldn't that be something?

But at least as of September 2026, that vision hasn't arrived. Everyone is only "renting" an agent, not really "owning" one.

1. Continual learning, four paths

First, full-parameter retraining. It's similar to continued pre-training (CPT), and it exists only in fantasy. In reality your data isn't as good as the base model's pre-training data, its distribution isn't as well chosen, and it can even turn around and hurt general reasoning ability. Setting all that aside, the biggest problem: even the big labs struggle to spare that much compute, to adapt a model for one domain, or even one individual, at that price and with no way of knowing whether it's right.

Second, parameter-efficient fine-tuning (PEFT). Freeze the base model, train a small patch of new parameters. In 2019, Adapter added small modules between Transformer layers³. In 2021, Prefix-Tuning learned only a continuous vector placed in front of each layer's input, under one thousandth of the full model's parameters, with results close to full fine-tuning⁴. Half a year later, LoRA (Low-Rank Adaptation) constrained the update to the product of two low-rank matrices, cutting trainable parameters by four orders of magnitude and shrinking a GPT-3 175B checkpoint from 350GB to 35MB⁵. Then it was engineering: which layers to hang LoRA on, how high to set the learning rate⁶.

Third, don't touch the parameters, lean on retrieval and memory. First, how do parameters support memory at all? That takes a short detour into the memorization-generalization paradox.

In 2017, Google showed that deep networks will memorize even random labels, which suggests a model can just memorize without finding any pattern⁷. Eight years later, ICML 2025⁸ was still calling back to it: big models memorize their training data and route around thinking, while small models can extrapolate but can't hang on to facts.

Meta later found in 2024 that each parameter stores exactly 2 bits of knowledge, unchanged even when quantized to int8⁹. Gradient descent settles this by itself. Memorizing one sample moves a handful of parameters, so the loss drops fast; learning a rule moves a whole stretch of parameters, so it's much slower. On the architecture side, we can run the same parameters over the same input again and again, forcing it to execute an algorithm instead of reciting one¹⁰. So where do big parameter counts still win? Goodfire and Anthropic say it's the long tail. Training-data engineering has matured, and once data got diverse, a small model's neurons can only handle high-frequency tasks. Long-tail tasks stay out of reach even when a solution exists, and they interfere with high-frequency ability¹¹. In other words: "small models will never catch up to big models."

But "your long tail and my long tail don't seem to be the same thing," and nobody can guarantee they'll always be part of "the majority's long tail." So "move the knowledge out of the parameters, leave the parameters to generalize" became a paradigm, and retrieval, external memory, hierarchical memory followed. DeepSeek went so far as to build external memory into the architecture, using an Engram module and constant-time table lookup to carry static knowledge, beating a pure MoE baseline across the board at equal parameters and equal compute¹². But retrieval can't handle the trade-offs. It gives you facts, not preferences. "Remembering something you said earlier for no reason at all in a new session — that's meaningless."

That leads to the fourth path, the cheapest layer: prompts. Once prompt engineering grew into context engineering, with all the engineering discipline that implies, Skills arrived as the culmination. Progressive disclosure, reading on demand, protocol-based use, instructions kept separate from scripts: for a while it was the meta answer to the long tail. Set that aside for now. And the crowd that used to work on ontologies and knowledge graphs started partying too, spinning out countless File-based Memory projects, Andrej Karpathy's LLM Wiki, and Google's Open Knowledge Format.

There's also a fifth, niche path: intervene directly in the model's internal activations at inference time (Steering), add a vector¹³ or a decoding constraint¹⁴. Plenty of papers, but whether it really works better than prompts still needs testing.

Behind the four paths sit two problems. One is scientific: catastrophic forgetting. The other is engineering: adapter, prompt, or harness, they all hang off the base model, and once the base model changes the effect can fall apart; and if what you had hanging there was an adapter, it simply stops working.

2. The Bitter Lesson: humans never learn

Let's go back for a moment to the Good Old Times right after LLMs blew up. How did you do continual learning back then? Prompting was a gimmick (chain-of-thought had just been discovered, and after that you could publish by adding just about anything to a prompt), and it didn't look serious enough. The consensus was that you fine-tune. You do parameter-efficient fine-tuning (Parameter-Efficient Fine-Tuning, PEFT). You do LoRA.

A problem that looked closed, and academia spent another three years circling it. From 2021 to 2024: how to allocate rank, whether the matrix structure was elegant enough, whether several LoRAs could be merged, whether a merged model could be routed, whether routing could then pick dynamically per input. Each question worth another paper. There's a retrospective on Xiaohongshu (which is in fact what set this piece in motion) that calls those three years PEFT's "era of filigree"¹⁵.

Around 2023 the filigree hardened into a slogan: the base model is an operating system, LoRA is an app. The story landed perfectly: the OS handles the low-level complexity, apps stay thin and plug-and-play, and in between lies the whole commercial fantasy of an app store. LoraHub used gradient-free optimization to combine off-the-shelf LoRA modules into tasks none of them had learned¹⁶; multi-LoRA serving let hundreds and thousands of adapters share one base model, and the whole industry, top to bottom, believed it: intelligence can be made pluggable.

"Companies big and small raced out plugin markets of every stripe plus hot-swappable all-in-one intelligence boxes. They loved this scheme, simply loved it; it was exactly the flavour the enterprise market wanted then, and the whole of academia caught the mood of alchemists' optimism."

Everyone knows how the rest goes. We did not end up in the future LoraHub predicted.

Compositionality was the first to collapse. Adding several task vectors together does not reliably lift all the tasks at once — the merged model always trails the specialist version of each task¹⁷. Model merging on its own (Model Merging) has a theoretical ceiling too: as you merge more, the marginal returns fall¹⁸, and interference between parameters costs the merged result a lot of points¹⁹. Even when two LoRAs are strictly orthogonal in subspace, merging them still gives no semantic decoupling; once they're stirred together, most of the time it is just a shallow pattern laid on top.

Then came Reflection 70B, the climax of the story. In September 2024 it did PEFT fine-tuning on Llama 3.1 70B with self-reflection data. Its leaderboard scores took off, it got packaged as the strongest open model, and to this day the weights are not open. A few days later OpenAI released o1, and four months after that DeepSeek-R1 went open source, and only then did everyone see that thinking was nowhere near that trivial. Reflection degraded into a museum piece, which says people really did believe that if you hung LoRA in the right places, you could pry a new capability out of the base model.

PEFT did not become the App Store of the AI world... It did not prove that intelligence can be split into dozens of low-rank plugins and then freely recombined like Lego. Nor did it prove that real reasoning hides in some mysterious low-rank subspace, waiting for a grad student with 8 GPUs to dig it out. Intelligence has no shortcut.

Richard Sutton said in "The Bitter Lesson" that over the long run, methods leaning on human cleverness lose to compute and scale²⁰. The grand old PEFT narrative is just one more piece of evidence for it.

But the only lesson humans take from history is that humans never take lessons.

3. Back from the dead

The fever broke. So where is LoRA's value, really? Academia has slowly been "correcting" it in two directions.

First, the ceiling. Mathematically, the low-rank parameter space is only a strict subset of the full fine-tuning space, and a small one, of measure zero²¹. The other side of it: when the difference between task and base is itself low-rank, LoRA is strictly better than full fine-tuning²². The dividing line is roughly data volume. Around a few thousand samples, LoRA is the better buy; only when data climbs to the millions and looks more like continued-pre-training-style knowledge injection does full fine-tuning overtake it²³. Thinking Machines Lab applied LoRA to every layer and set the learning rate ten times higher than full fine-tuning's, and found that in the vast majority of post-training settings LoRA matches full fine-tuning exactly on sample efficiency and final performance; for reinforcement learning, rank-1 is enough, because the reward of a single episode often carries less than a bit of information. Which is to say: only when you inject enormous amounts of new knowledge into the model does LoRA start to fall behind²⁴.

So LoRA is far from worthless. Want the model to learn something entirely new (a whole new language, a wildly rare word combination)? Don't count on a small parameter budget. Want the model to recombine what it already has, a person's tone, a style of decision-making? That's exactly where LoRA has the edge. The original LoRA paper measured it: the fine-tuning delta $\Delta W$ precisely amplifies the directions that matter in the base weights but get no emphasis⁵. A rank-4 delta picks out roughly 4 such directions per layer and amplifies them by as much as 20-odd times; at rank 64 the amplification drops back to about 2x. What LoRA learns has always been a re-weighting of representations that already exist.

Second, the shape. The answer is not merging. It's routing. Spectral analysis shows that merging LoRAs in sequence accumulates "intruder dimensions" that almost never show up in full fine-tuning, and the more you merge, the faster it degrades²⁵. That same year Mixture of LoRA (MoL) used a gating network to combine a frozen pool of LoRAs²⁶, and over the next two years this line kept evolving, until mid-2026, when Mind Lab put the pedal to the floor: in June the research paper "On the Scaling of PEFT" made the case, and in August the technical report Macaron-V1 shipped it:

1. DeepSeek-R1-Distill-Qwen-32B with a 0.07B LoRA attached improves 20.61% normalised on AIME 2025, beating the full-parameter reinforcement learning of the 1.5B model in the same family²⁷;
2. The same base (Qwen3-30B), the same data, the same hyperparameters: 198 LoRAs trained with nothing different but perturbations like the order the data arrived in, then majority-voted, taking AIME24 from 36.4% to 48.7%²⁷;
3. The product shape the report gives: the whole GLM-5.2 744B base stays frozen and the capabilities go into four roughly 1B LoRA experts (Chat for conversation, Agent for daily-life assistance, Coding, GenUI). No separate routing model at all; the entry expert L0 picks the expert per request itself. A new capability means hanging one more LoRA; the old ones do not move at all²⁸.

What they are betting on: (1) the strength of the prior may matter more than the number of trainable parameters; (2) different LoRA variants under majority voting can produce a kind of swarm intelligence. For now, training a "personal LoRA" per user is still a future direction in the report; what they actually deliver is four fixed experts. The story is deafening: "personal models at the million scale, persistent." On the enterprise side they open training and hosting as an API, billed by token; on the consumer side they built Macaron, a personal agent with deep memory, and what runs in production is exactly this MoL architecture, one frozen base with four fixed experts shared by all users, routed by user turn; and "each user's own preferences, memory and tools" are for now served by a bolt-on Agentic Memory system, with "one LoRA per person" still on the way. Subscriptions, about two weeks since commercial launch, $10 million ARR.

But even if "one LoRA per person" works, the experts, the memory, the user preferences all grow on the same base. What happens when the base goes stale? What about users who want the capabilities of a newer base? They can only go quiet for now (in public materials, at least). GLM is already at 5.3; their whole stack still stands on 5.2. One generation is fine. Over the long run?

On swapping base models and LoRA, there's a 2024 paper called PortLLM: do a normal LoRA fine-tuning run on the old base and get a patch; then when the vendor continued-pre-trains (CPT) a new base, add the patch straight in, matrix addition. On what grounds? The paper takes the ideal answer, "fine-tune again on the new base", and decomposes it algebraically: $\text{ideal} = \text{new base} + \text{old patch} + \text{a residual}$, where the residual measures how much the patch would have changed. Measured, it comes out 66 times smaller than the main term on average, and about 96 times smaller at the most. In other words, CPT (simulated here) really does move the base a long way, but the direction your task needs barely moves²⁹. The results: direct addition loses less than a point against re-fine-tuning, and a few tasks come out ahead; it holds across the Mistral, Llama and Gemma families, on consecutive multi-generation base updates simulated by different authors. But patches only migrate within a family. Llama to Qwen, cross-family, is explicitly future work.

At the other end, OpenAI has announced it is winding down the fine-tuning API, stopping new fine-tuning jobs from January 2027³⁰. Closed-source fine-tuning is narrowing. Open base models plus a learnable parameter layer are alive again.

4. The laws of development, and what the masses need

Readers have probably been carrying a question the whole way: isn't the hottest thing of the past year Skill? Who's out there playing with parameter space?

True enough. As of September 2026, Skills are close to an iron law for the long tail.

Skill isn't technically elegant. It's satisfying to use. Writing fuzzy preferences down as clear rules is itself a pleasure, the "look how good I am at this" kind. Add one constraint line to an md file, and as long as the model is strong enough, the effect is immediate. As Tim O'Reilly put it, "implicit knowledge, executable for the first time"³¹. Rules are natural language: readable, editable, auditable. As OpenClaw and Claude Code got hot, each one feeding the other straight into orbit, and with SkillHub and GitHub behind them, an ecosystem formed fast. Finish a Skill and you're holding a real piece of work in your hand, ready to hand out and spread around — unlike parameter tweaking, that "alchemy" where all you can do is stare helplessly at loss and accuracy.

Even better is the nationwide language carnival: Skill made "distillation" a hot word. Distill your experts, distill your colleagues, the person leaves, the Skill stays. It grew into a race among the big labs to mass-produce Skills, running in parallel with the tokenmaxxing race, and on to Zuckerberg putting screen monitors on employees to collect data and train models. FOMO, the media, the platforms all piling on: together they gave Skill real momentum.

Skill is great to use. What everyone keeps quiet about is that Skill has a ceiling: natural-language rules can't scale up.

In May this year, Jiayi Weng wrote the blog post "Learning Beyond Gradients": using an agent that maintains rules to play Breakout and control a simulated robot, arguing that piling up rules beats training a network. It gave continual learning an alternative road: gradient is not the only way to learn. Traditional heuristics, expert systems, "hand-written" rules can be learned and maintained too. "A lot of the heuristics of the past weren't useless, it's that nobody could afford to keep them alive," and coding agents have pushed the maintenance cost to nearly 0³².

But everything this paradigm got working falls into the box of "change it and the metric moves right away": game inputs show up in the score, code commits show up in the error trace. Where does natural language have that? "Don't say not-X-but-Y." "Don't use em dashes." "Don't reach for metaphor every other sentence." "Say something with substance." "Quit flinging abstract jargon." In the end you can only hand it to a human to slowly "taste" whether it fits. It's hard to write the model an exhaustive operating manual up front. Usually you only discover "this is wrong" and "that's wrong too" at the moment the model does something stupid and your blood pressure spikes.

Even for verifiable instructions, how well the model follows them decays as the prompt grows³³³⁴, so forget the unverifiable ones. And once context gets long, the model has attention sink (attention rushing to the beginning)³⁵ and lost in the middle³⁶; all you can do is pray the model still remembers. So we can't stuff 500 constraints into SKILL.md in one go. That runs against how the model works. And even if it did work, switch models or upgrade one (Prompt Migration) and hundreds or thousands of carefully polished bespoke rules all die at once³⁷. What's more, none of those "super prompts" that beat fine-tuning came from a human hand; every one was optimized by reinforcement learning³⁸. A prompt that can scale is no longer natural language.

Right now, for the industry's standard answer on "the long tail," Anthropic has given some complete demonstrations. Summarized: a single-model Agent Loop handles the trunk, Skills catch the long tail, tool calls cover existing systems, the Harness enforces rules, snapshot-style evaluation³⁹. It looks like things have converged on somewhere quite good. But we have to ask: can natural-language In-Context Learning (ICL) really catch every long tail? Put another way, what else can parameters carry that a prompt can't?

Trace one layer further down and there's a more theoretical question: when exactly are a prompt and fine-tuning equivalent? A batch of work since 2022 has answered it. A prompt's effect on attention is, in theory, isomorphic to one step of linear gradient descent⁴⁰:

The reverse holds too: people have hand-built a Transformer that runs one gradient-descent step per layer, and the trained model's behavior lines up as well⁴¹. So fine-tuning buys parameters, and piling up prompts rents them. The lease runs from the start of a context to the end of it. In the short run, the rented thing and the bought thing are the same thing.

What renting and buying can carry was never the same to begin with. Natural language carries the part that can be said out loud: steps, rules, constraints. Parameters carry a different kind of content: distributions, habits, experience, collocations, the fine calibration of tone, trade-offs, intuition, the posture you take when things are ambiguous. That stuff is messy, broad, personal. It will never write down cleanly as a prompt or a few-shot example. It can only go into the parameters, fed in by data⁴². And for now everyone just has to endure the pain of that part, because nearly the whole industry is converging on the same place. Dwarkesh Patel wrote "8 Predictions for the Era of Continual Learning" last month⁴³. Briefly:

1. Base models update every day, so they should be reviewed on a rolling basis;
2. Alignment should ensure a model self-supervises through continual updates and doesn't degrade under cross-user aggregation or federated learning;
3. Today's mainstream AI Minds number no more than five, they eat the same data, and they all look alike; once experience diverges, genuinely different models will emerge, escaping mode collapse;
4. The best model gets used the most, draws the most feedback, gets smarter, and the flywheel spins faster;
5. Which means the stronger the model, the earlier it should be let out, rather than hidden away;
6. Switching models is like firing an old employee who has accumulated years of experience; switching costs are enormous, and once you're locked in, the vendor can charge whatever it likes;
7. Users either lock their context into one provider or give up "the more you use it, the better it understands you"; the big labs will also subsidize users willing to license their sessions, and those who refuse to license get no access to the best models;
8. If personalization lives not in LoRA but in a fork of the full weights, then batch size=1 for a single user is something almost only the big labs can do.

His continual learning is, by the look of it, still "centralized distribution." This essay holds that continual learning, properly understood, is a combination of natural language and parameters, and that however the experience gets deposited, it should belong to the user. Zuckerberg said last month, in "The Future Is for Everyone"⁴⁴, that intelligence should belong to everyone too.

Continual learning should be a three-way split. Whatever can be written as rules goes to natural language, which answers the masses' need for controllability, explainability, and handoff. Whatever can be looked up goes to external memory, because every bit in the parameters should be spent sparingly, on patterns, and memorized facts have no business living in the weights. What's left, the part that can't be written as rules and can't be looked up, goes to parameters, and this road follows the laws of development: base models are swelling and speeding up, personalization demand is exploding, and neither of those answers to anyone's will. No two leaves are alike, and there is no single super-model that can cover every need. Waiting for such an AGI moment is futile and absurd.

Tech columnist Steve Yegge insists flat out: Skills are a bet against Sutton's "bitter lesson." If the bitter lesson holds as usual, the knowledge inside Skills will sooner or later be eaten by the base model. We don't buy it. No base model with unlimited parameters can exist that holds everyone's different taste, "human flavor," non-standardness, novelty, and macro-planning³¹. Skills is a very good solution, and also a limited one.

Back in early 2024, UC Berkeley's BAIR lab proposed going "from models to compound AI systems": more and more frontier results come out of multi-component systems rather than a single model⁴⁵. NVIDIA Research's position paper last year argued that heterogeneous collaboration between small task-specific models and large general-purpose models is the natural selection⁴⁶. And last month Will Sommer, an analyst at the consulting firm Gartner, went further: there is no model on the horizon that can be reliable and cheap at the same time across all scenarios, and the inference cost of a single agentic workflow will still rise fivefold by 2028⁴⁷. We need at least some "small models that meet our own needs."

From there it's easy to arrive at Personalization, which is in fact a fairly mature field. Reading the surveys of the past two years, it splits into four camps: (1) profile: compress user history into a profile and memories, retrieve into context at use time⁴⁸; (2) reasoning: leave the black-box model alone and align the reasoning path to the user at the chain-of-thought level⁴⁹; (3) reward: customize the evaluator, train each user their own reward model, and still scale per-user at test time⁵⁰; (4) parameters: fine-tune a small slice of weights for each user, the smallest camp of the four⁵¹. The first two are one model serving everyone, with all personalization computed on the spot, which is really "renting." The last two are "buying."

The one part that's probably genuinely hard is evaluation. Learning needs a signal, a baton. Whether a personalized result is good can only be judged accurately by the target user themselves⁵², and automatic evaluation metrics will hand you a falsely high correlation under a particular data composition⁵³. And don't forget Goodhart's Law: once a gold standard is found, the model reward-hacks it, and checking becomes a disaster.

5. Contested ground

The AI boom is largely a boom in word-coining, and a specifically anthropomorphic one: "neuron," "perceptron," "attention," "memory." They give everything a distant, cybernetic air, machines on the rise, biology gone bionic⁵⁴. I think it is worth connecting and distinguishing the continual learning this essay is about from a few fashionable terms: Sovereign AI, test-time training (TTT), and recursive self-improvement (RSI).

Sovereign AI

Jensen Huang was already out selling this one in 2023⁵⁵, and it meant real sovereignty, at the national level. The years since produced actual deployments: 01.AI with Kazakhstan, ModelBest with Laos, MiniMax with Saudi Arabia⁵⁶. Nowadays plenty of people stretch the concept to companies and individuals: the base model will never belong to us, so for business safety and to cut the cost of adapting to each new iteration, we need AI we can control, rather than being carved up and ordered around by the foundation-model vendors. There are many paths to that, and one of them is training from scratch, actually owning all the weights.

This essay shares the sovereign attitude behind "intelligence belongs to everyone," but I do not think every problem can be solved by training your own base. Narrow-task small models may work. But the general ability of a base model is extremely hard to replicate now that large models are heavy industry, so you may as well stand on the shoulders of giants.

Test-time training (TTT)

This is the branch I fear most, and its standard-bearer is Ilya Sutskever's SSI. Word is their TTT works well and is already on the way. SSI has always been secretive; all you can see is the long-term partnership with NVIDIA, compute for a next-generation platform, and about $8 billion raised over two years⁵⁷.

Still, we can piece together roughly what the TTT line looks like. The model does a few gradient updates in place while it reasons, so inference and learning become the same act; a prototype existed as early as 2020⁵⁸. By 2024 it was built inside the sequence layer: a layer's hidden state is itself a small model that trains itself on every stretch of input it reads, as a way to handle long context⁵⁹. Accepting the NeurIPS Test of Time award, Sutskever said internet data is fossil fuel and pre-training will end, no question about it⁶⁰. In the scattered remarks since, their plan comes out in four pieces: TTT built into the architecture; the model updating continuously in service, the session as the training stream; the learning signal coming from an internal value function rather than human labels; and scaling moved from the training side to inference-time compute. The core belief is that the boundary between training and inference should be dissolved entirely.

First: if closed-source TTT works, if a model can train in place while it talks, then anything bolted on outside gets eaten by the base model, and our own approach could, in turn, end up as the Bitter Lesson itself. So the first step to shipping it is evaluation, plus a justification for what particularity external parameters can actually carry, and we need to find real-world scenarios. Writing and language style might be a start.

Second: can the big labs push the marginal cost of hosting one set of personalized weights per user down to the order of what a recommender system spends maintaining user profiles? If they can, personalization becomes a native feature of the base model, folded into the subscription, and we really are the Bitter Lesson. If they cannot, external parameters have room to live.

A user profile is light. Weights are heavy. They need continuous training, they have to sit in VRAM, they break batching. Even if the cost comes down, a lab's objective is the average over all its users, which personalization is naturally at odds with, so one round of RL can only lift general ability, and everything that is not general (language style, for instance; coding agents genuinely do not talk like a human) is left to mode collapse.

Other vendors I cannot speak for, but Mind Lab is already pushing this. MinT turns model training into an API, with training, sampling, checkpoints, and storage billed side by side in the SDK under the name training token⁶¹. Inference is already open to individuals, and third-party gateways list prices⁶². Hosted training is mostly enterprise onboarding by invitation, with no public price list for training⁶¹. The faster the price of a training token falls, the closer personalized weights get to the range where they are simply swallowed in passing.

Recursive self-improvement (RSI)

The buzzword before RSI was reinforcement learning, and it has long since worked in math, code, and games: R-Zero has a Challenger and a Solver set each other problems from zero data and co-evolve⁶³. But the loop only turns when there is a signal, which is where reinforcement learning with verifiable rewards (RLVR) comes in. The moment a reward stops yielding new information, it plateaus⁶⁴.

RSI goes further, and now that coding agents are so much better, AI can write code and strategies that improve itself and build environments that improve itself, and so on. On September 10, SJTU and Theseus Labs put out a heavyweight survey, The Last AI Built by Humans⁶⁵. They define RSI as "turning experience and feedback into durable change, and improving one's own capability while also improving the very process by which that capability is improved," then sort RSI into L1 through L5 by how much decision-making power the AI holds over its own improvement. From there the survey argues that if experience accumulates only into capability, while the learning objective, the update rule, and the acceptance test still belong to humans, then it is still just continual learning. Only when the AI starts deciding how to modify, how to evaluate, how to consolidate, and the revised process gets reused in the next round, has it touched the threshold of RSI.

This essay argues that RSI does not have to be a giant system in a data center improving itself. It can also be the small patch beside each user. In other words, continual learning can be RSI at the individual level. Carried to its end, this line makes me think of Meta and KAUST's position paper from April, Neural Computers⁶⁶: a neural network can itself be a computer, internalizing compute, memory, and IO into the implicit runtime state the model learns, no longer depending on separate external hardware and software modules to host its state, and eventually settling into a general form of neural computation that executes stably, can be programmed, and reuses capability durably. Everyone ends up with a neural network on their desktop, evolving in your direction. I look forward to a future like that.

A springtime for hybrid architectures?

The future is definitely not one size fits all. There are already quite a few examples with real results behind them.

In July, PyroDash: a 4B Qwen3.5 starts working the problem, and the moment it finds it cannot handle where things are going, it spits out a "raise my hand" token. The system then packages the reasoning trace so far and hands it to a frozen GLM-5.2-FP8 to continue. One direction, at most once. The policy for when to ask for help is internalized in the small model itself; no external router, no gradients on the big model, just a standard text-completion interface⁶⁷. That "raise my hand" token is first initialized near natural breakpoints such as periods and newlines, then given a supervised cold start on the EasyHard-24k dataset, where a right answer counts as easy and a wrong answer that can still be reconstructed into a correct trace counts as hard. Last comes cost-aware RL, with cost written into the reward and a $\lambda$ knob that sets how often the model raises its hand. Tune $\lambda$ well and performance beats the frozen big model by 6.36 points at 20% lower cost, because by the time the big model takes over it sees not only the problem statement but the reasoning the small model has already organized.

Pyromind founder Kevin Ding, in a recent interview⁶⁸:

"The problems and scenarios in our world that need AI to solve… are already endless. New scenarios keep appearing, and the scenarios themselves are not static. Given that, using a model with finite parameters to generalize over infinite scenarios is, at least on the Transformer architecture, too big a challenge."

"The better the base model, the less pressure on the Worker Model. This architecture is itself a patch on top of the base model; the base evolving is good for it, not a threat. … Performance, cost, and privacy are hard to solve all at once just by calling a bigger model. Some tasks suit a direct call to the base model, some are better done on a local small model, and some need the two to collaborate. The stronger the base model, the more demand there is for routing, coordination, and cost-optimizing RL."

PyroDash can at least prove its advantage on math benchmarks, and on the strategic judgment this essay is of the same mind. But outside math, which tasks are "better done on a local small model"? In this essay's terms, which ones "have to be carried in local weights"? And how do you evaluate that? We have yet to answer.

On September 2, mostik.ai had models exchange hidden states directly, with no text in between. The sender is a 753B GLM-5.2 that only reads; the receiver is a 4B Qwen3.5 that does all the generation; between them sits a separately trained little bridge ("mostik" is Russian for "little bridge") that translates the big model's representation space to the small model. Both models stay frozen throughout, and only that one bridge in the middle trains⁶⁹. The bridge is not plug and play either. By their account, the two models' representation spaces align only partially, and the degree of alignment is predictable, so it can be trained, but every pair needs its own bridge.

With the bridge attached, the 4B model gains 25% accuracy, and doubles on hard samples; the pair uses 2.5x less compute than a mid-range model of the same class, beats text transfer by 10 points, and the earlier the handoff, the bigger the advantage. Two lessons. One: even sending text between models loses information, since each token carries only about 17 bits out of roughly 2MB of internal state, so writing personalization into the weights becomes close to inevitable. Two: the delta does not have to be bound tightly to the base; it can be a small model in the user's hands plus a bridge. Or does it have to be? Can we explore a method that decouples from the big model? That too is ours to answer.

The hybrid goes beyond small models. On September 10, DeepSeek released the technical report for V4.1-Flash, designing the division of labor into the skeleton of a single model. They observed that in agent workloads, frequent tool calls bring massive prefill, and the volume read far exceeds the volume written. So the Causal Encoder-Decoder architecture has the first half of the model's layers act as a "causal encoder" that handles reading, and when the second half generates, the KV is projected straight from the hidden states of the first half, and prefill compute nearly halves. On active parameters: reading a token activates 8B, writing a token activates 16B. Reads run stripped down, writes run fully loaded. Add sparse attention and an FP4 cache on top, and the global KV compresses to 890 bytes per token, a quarter of the previous generation⁷⁰.

You could say that any hybrid architecture will one day be replaced by one giant centralized ASI, so these current solutions, like Skills, work well but are not elegant, and may still be the Bitter Lesson. But the closer you get to elegance, the further you may end up from the truth.

Summary: the Monticule approach

Continual learning is an engineering problem, a scientific problem, and a social problem, and few fields have ever drawn this much interest out of me. We believe in the future of continual learning, especially the kind aimed at personal, custom models, so we want to start a Neo Lab, Monticule Tech: to keep exploring the boundary of what continual learning can do, and to take on the broad, scattered tasks of the real world, accumulating your implicit preferences, taste, experience, and judgment, delivering real social value, and thereby genuinely improving human wellbeing.

Taken together, we propose the Monticule approach:

• Adapt as the strong remote base keeps evolving, and gain from that evolution
• Fully decoupled from the base model, so it follows the base across family changes and leaves the user free to choose
• A strong remote base + a small parameter space that keeps adapting to the user + a natural-language space that keeps accumulating (knowledge and instructions included)
• Explore, argue for, and evaluate and verify which aspects of human intelligence the Monticule approach can carry

We are not going to commit to a fixed technical route. Any combination of techniques that gets there can count as the Monticule approach. The three lines, engineering, science, society, all end up supporting the same set of judgments. If the base keeps turning over, decouple from the base. If LoRA cannot produce new abilities, embrace a strong base. What natural-language Skill and Memory cannot hold, we make up for with the parameter space. And more and more evidence says hybrid architectures are the future. Our biggest belief: we do not believe the future holds an ASI that takes on everything, one that can cover all the complexity and diversity of human society and human intelligence. Intelligence belongs to everyone, and individuality need not bend to "renting parameters."

Will this route work? Can it become the next paradigm? Will a more general, more elegant solution swallow it, and turn it into the Bitter Lesson once again?

Continual learning is nothing new, but in the age of base agent models, no vendor has yet fully shipped a mature solution that lands. It is hard enough, complex enough, broad enough, and we believe Monticule can make its own contribution.

References