On November 5, 2026, three Google researchers posted Muir, a new language-model design, to arXiv, the preprint server. One is also affiliated with the University of Texas at Austin. Their 1.34-billion-parameter model read continuous texts of 10 million tokens, or word fragments, using a fixed 25 MB of working memory. By the authors’ calculations, a Transformer of the same size, the architecture behind today’s large models, would need a 2.1 TB cache to do the same.
That cache limits how much text a model can read in one pass, its “context,” when you run it on your own machines. Muir is still research: the models tested are small and were trained on texts of only 4,096 tokens. Their weights, the files that make up a trained model, have not been released.
A memory that does not grow with the text
A Transformer keeps a record of every token it has read in its KV cache, short for “keys and values.” The cache grows with every new token. For 10 million tokens, the authors put it at 2.1 TB with 16-bit numbers, or 0.32 TB with numbers compressed to 2.5 bits.
Muir instead uses linear attention, a variant of the step where a model looks back at earlier tokens. It compresses everything read so far into a fixed-size state, updated at each token. The state takes up 25 MB across Muir’s 24 layers, or processing stages, whether the text is 4,000 or 10 million tokens long.
The authors point out the catch: a fixed-size state cannot hold everything. The more the model writes into it, the more new information blurs the old. A family called delta-rule models reduces the problem. Before storing a value under a key, a kind of label, they erase what the state already held under that key.
Three changes to an existing model
Muir builds on Gated DeltaNet-2 (GDN-2), a delta-rule model published on May 21, 2026, by Ali Hatamizadeh, Yejin Choi and Jan Kautz. NVIDIA’s research lab hosts its code. GDN-2 controls erasing and writing separately through “gates,” learned valves that set how much of each happens.
First change: by the authors’ measurements, the state is about 13 times more sensitive to what gets written than to what gets erased. So Muir shrinks the erase gates and packs all its gates into one compact block. The saved parameters widen another part of the network by up to 30.4%, keeping the total parameter count unchanged.
Second change: some parts of the state fade slowly and build up large values. They dominate what the model reads back and push it to erase harder, which risks overwriting useful information. Muir adds a read gate to rebalance them, modeled on a signal-processing technique called the matched filter. The gate adjusts at every token, except in a “static readout” variant.
Third change: new training math that stays numerically stable, with no runaway errors, on the large blocks of text that TPUs, Google’s AI chips, need to run efficiently.
The results the authors report
The authors trained models of 166 million to 1.34 billion parameters from scratch on eight TPU7x chips. The data was 100 billion tokens from FineWeb-Edu, a dataset of educational web pages. They compared Muir with GDN-2 and with a Transformer of the same size.
Perplexity measures how well a model predicts the next token: the lower, the better. On this measure, Muir scores 1.8% to 2.0% better than GDN-2 and 2.5% to 3.6% better than the Transformer. On common-sense reasoning tests, its gain is 1.9 points at most.
The models also read three texts of 10 million tokens, each made of documents placed end to end. The authors measured perplexity over ever-longer windows of text, clearing the memory between them. Without extra training, Muir’s perplexity drops to 8.11 with 64,000-token windows and stays there up to a single 10-million-token window.
GDN-2 levels off at a worse 8.31. The Transformer rises to 10.51 with 64,000-token windows, then runs out of chip memory.
The passkey test, RULER S-NIAH-1, hides a seven-digit number in a long filler text of repeated sentences. The model has to find it.
Without extra training, the static-readout variant still succeeds 90.0% of the time at 1 million tokens. After a short round of extra training on 64,000-token texts, it reaches 99.4% at that length. GDN-2 collapses beyond 4,000 to 8,000 tokens.
On 32,000-token texts, Muir trains as fast as GDN-2 and 2.7 times faster than the Transformer. Below about 11,000 tokens, the Transformer still trains faster, the authors acknowledge.
Two variants behind the headline numbers
The paper’s abstract puts two results side by side, but they come from different variants. Only the words “with static readout” tell them apart. The steady perplexity over 10 million tokens comes from the full version. The passkey result at 1 million tokens comes from the static-readout variant.
Their conclusion acknowledges a trade-off. For the same budget, the static readout is better at holding on to a single distant piece of information. The full version’s dynamic readout is better at questions that span several passages.
Without extra training, the full version finds the passkey 85.6% of the time at 32,000 tokens, but only 26.2% at 64,000 and 12.5% at 1 million. The static-readout variant’s perplexity drifts slightly on continuous text until it gets extra training.
A steady perplexity score also does not prove that the model uses all 10 million tokens. With its memory cleared every 64,000 tokens, Muir already scores 8.11, the same as when it reads all 10 million in one pass.
The study’s limits
Parameters measure a model’s size, and Muir’s largest test model has 1.34 billion. Qwen3.8-Flash-Next, an open model Alibaba released in August, has 125 billion, though only 6 billion are used for each token.
The 2.1 TB figure is a calculation. The authors ran the Transformer without splitting its computation into chunks. In the 10-million-token and passkey tests, it runs out of chip memory at 128,000 tokens and up. All measurements were made on Google’s TPUs.
The arXiv page links to neither a code repository nor the weights. An appendix does include reference code, written for JAX, Google’s numerical computing library. The paper is a first version, not yet peer reviewed.
In a November 6 article for AI Weekly, Alexis Dufresne notes that all the figures come from the authors. The piece adds that Google has not said whether Muir will go into a product. An editor’s note calls the result worth watching if another team can reproduce it.
What it means
The Gated DeltaNet family, which includes GDN-2, already runs in local models. Alibaba’s Qwen models, from Qwen3.5 through Qwen3.8, have used a hybrid that mixes Gated DeltaNet layers with full-attention layers, the kind a Transformer uses. The Qwen team says so in the code repository for Qwen3.8-Flash-Next, released on August 26, 2026.
Qwen3.8-Flash-Next previews the design of Alibaba’s upcoming Qwen4 generation. Only one in four of its layers uses attention. The other three keep a fixed-size state, while the attention layers keep a cache that grows with the text. The model runs on llama.cpp, a free program for running models locally.
Muir has not been tested in that kind of hybrid. Its authors float the idea of pairing the static-readout variant with sliding-window attention. That type of attention looks only at the most recent tokens, so its memory stays fixed too.
On a local machine, the model’s weights share memory with its cache, and they already take up a lot of it. Muse Spark Open, Meta’s open model announced with a 1-million-token context, has 816 billion parameters. Meta and Apple suggest running it on two linked Mac Studio desktops with 512 GB of memory each. On October 15, NVIDIA presented a 74-billion-parameter model compressed to fit in 24 GB with 128,000 tokens of context.
Open questions remain: how Muir behaves at tens of billions of parameters, on real tasks with long contexts, and inside a hybrid model. Beyond 8,000 tokens, the authors measured only perplexity and the passkey test. They say Muir’s core runs on graphics cards with GDN-2’s existing code, unchanged, but publish no measurements on them.
What to watch next: a release of the weights or a code repository, and a reproduction of the results by an outside team. Another signal would be Muir’s gates turning up in open models from Qwen, NVIDIA or Moonshot AI. Moonshot makes Kimi Linear, an earlier design the paper cites. After that would come support in llama.cpp and vLLM, another program for running models.
Sources
- Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout, arXiv (Tushaar Gangavarapu, Gil Tabak and Clemens JS Schaefer), November 5, 2026
- Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention, arXiv (Ali Hatamizadeh, Yejin Choi and Jan Kautz), May 21, 2026
- On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability, arXiv (Qwen team), August 31, 2026
- Qwen3.8-Flash-Next, Qwen (GitHub), August 26, 2026
- Google’s Muir cuts perplexity 3.6% vs Transformer at 1.34B, AI Weekly, November 6, 2026
Header image: MACCUS chart, data from Gangavarapu, Tabak and Schaefer, “Muir”, arXiv 2611.08376, November 5, 2026.
