Attention ISN'T all you need?! New Qwen3 variant Brumby-14B-Base leverages Power Retention technique
When the transformer architecture was introduced in 2017 in the now seminal Google paper "Attention Is All You Need," it became an instant cornerstone of modern artificial intelligence. Every major large language model (LLM) — from OpenAI's GPT series to Anthropic's Claude, Google's Gemini, and Meta's Llama — has been built on some variation of its central mechanism: attention, the mathematical operation that allows a model to look back across its entire input and decide what information matters most.Eight years later, the same mechanism that defined AI’s golden age is now showing its limits. Attention is powerful, but it is also expensive — its computational and memory costs scale quadratically with context length, creating an increasingly unsustainable bottleneck for both research and industry. As models aim to reason across documents, codebases, or video streams lasting hours or days, attention becomes the architecture’s Achilles’ heel.On October 28, 2025, the little-known AI startup Manifest AI introduced a radical alternative. Their new model, Brumby-14B-Base, is a retrained variant of Qwen3-14B-Base, one of the leading open-source transformer models.But while many variants of Qwen have been trained already, Brumby-14B-Base is novel in that it abandons attention altogether. Instead, Brumby replaces those layers with a novel mechanism called Power Retention—a recurrent, hardware-efficient architecture that stores and updates information over arbitrarily long contexts without the exponential memory growth of attention.Trained at a stated cost of just $4,000, the 14-billion-parameter Brumby model performs on par with established transformer models like Qwen3-14B and GLM-4.5-Air, achieving near-state-of-the-art accuracy on a range of reasoning and comprehension benchmarks.From Attention to Retention: The Architectural ShiftThe core of Manifest AI’s innovation lies in what they call the Power Retention layer. In a traditional transformer, every token computes a set of queries (Q), keys (K), and values (V), then performs a matrix operation that measures the similarity between every token and every other token—essentially a full pairwise comparison across the sequence. This is what gives attention its flexibility, but also what makes it so costly: processing a sequence twice as long takes roughly four times the compute and memory.Power Retention keeps the same inputs (Q, K, V), but replaces the global similarity operation with a recurrent state update. Each layer maintains a memory matrix S, which is updated at each time step according to the incoming key, value, and a learned gating signal. The process looks more like an RNN (Recurrent Neural Network) than a transformer: instead of recomputing attention over the entire context, the model continuously compresses past information into a fixed-size latent state.This means the computational cost of Power Retention does not grow with context length. Whether the model is processing 1,000 or 1,000,000 tokens, the per-token cost remains constant. That property alone—constant-time per-token computation—marks a profound departure from transformer behavior.At the same time, Power Retention preserves the expressive power that made attention successful. Because the recurrence involves tensor powers of the input (hence the name “power retention”), it can represent higher-order dependencies between past and present tokens. The result is an architecture that can theoretically retain long-term dependencies indefinitely, while remaining as efficient as an RNN and as expressive as a transformer.Retraining, Not RebuildingPerhaps the most striking aspect of Brumby-14B’s training process is its efficiency. Manifest AI trained the model for only 60 hours on 32 Nvidia H100 GPUs, at a cost of roughly $4,000 — less than 2% of what a conventional model of this scale would cost to train from scratch.However, since it relied on a transformer-based model, it's safe to say that this advance alone will not end the transformer AI-era.As Jacob Buckman, founder of Manifest AI, clarified in an email to VentureBeat: “The ability to train for $4,000 is indeed only possible when leveraging an existing transformer model,” he said. “Brumby could not be trained from scratch for that price.”Still, Buckman emphasized the significance of that result: “The reason this is important is that the ability to build on the weights of the previous generation of model architectures is a critical accelerant for the adoption of a new modeling paradigm.” He argues this demonstrates how attention-free systems can catch up to transformer performance “for orders-of-magnitude less” investment.In the loss curves released by Manifest AI, Brumby’s training loss quickly converges to that of the Qwen3 baseline within 3,000 training steps, even as the architecture diverges significantly from its transformer origins. Although Brumby-14B-Base began life as Qwen3-14B-Base, it did not remain identical for long. Manifest AI fundamentally altered Qwen3’s architecture by removing its attention layers—the mathematical engine that defines how a transformer model processes information—and replacing them with their new “power retention” mechanism. This change restructured the model’s internal wiring, effectively giving it a new brain while preserving much of its prior knowledge.Because of that architectural swap, the existing Qwen3 weights no longer fit perfectly. They were trained to operate within a transformer’s attention dynamics, not the new retention-based system. As a result, the Brumby model initially “forgot” how to apply some of its learned knowledge effectively. The retraining process—about 3,000 steps of additional learning—served to recalibrate those weights, aligning them with the power retention framework without having to start from zero.A helpful way to think about this is to imagine taking a world-class pianist and handing them a guitar. They already understand rhythm, harmony, and melody, but their hands must learn entirely new patterns to produce the same music. Similarly, Brumby had to relearn how to use its existing knowledge through a new computational instrument. Those 3,000 training steps were, in effect, its crash course in guitar lessons.By the end of this short retraining phase, Brumby had regained its full performance, reaching the same accuracy as the original Qwen3 model. That quick recovery is what makes the result so significant: it shows that an attention-free system can inherit and adapt the capabilities of a transformer model with only a fraction of the training time and cost.The benchmark progression plots show a similar trend: the model rapidly approaches its target accuracy on core evaluations like GSM8K, HellaSwag, and MMLU after only a few thousand steps, matching or even slightly surpassing Qwen3 on several tasks.Benchmarking the BrumbyAcross standard evaluation tasks, Brumby-14B-Base consistently performs at or near parity with transformer baselines of comparable scale.TaskBrumby-14BQwen3-14BGLM-4.5-AirNemotron Nano (12B)ARC0.890.940.920.93GSM8K0.880.840.830.84GSM8K (Platinum)0.870.880.850.87HellaSwag0.770.810.850.82MATH0.620.540.470.26MBPP0.570.750.730.71MMLU0.710.780.770.78MMLU (Pro)0.360.550.510.53While it lags slightly behind transformers on knowledge-heavy evaluations like MMLU-Pro, it matches or outperforms them on mathematical reasoning and long-context reasoning tasks—precisely where attention architectures tend to falter. This pattern reinforces the idea that recurrent or retention-based systems may hold a structural advantage for reasoning over extended temporal or logical dependencies.Hardware Efficiency and Inference PerformanceBrumby’s power retention design offers another major advantage: hardware efficiency.Because the state update involves only local matrix operations, inference can be implemented with linear complexity in sequence length. Manifest AI reports that their fastest kernels, developed through their in-house CUDA framework Vidrial, can deliver hundreds-fold speedups over attention on very long contexts.Buckman said the alpha-stage Power Retention kernels “achieve typical hardware utilization of 80–85%, which is higher than FlashAttention2’s 70–75% or Mamba’s 50–60%.” (Mamba is another emerging “post-transformer” architecture developed by Carnegie Mellon scientists back in 2023 that, like Power Retention, seeks to eliminate the computational bottleneck of attention. It replaces attention with a state-space mechanism that processes sequences linearly — updating an internal state over time rather than comparing every token to every other one. This makes it far more efficient for long inputs, though it typically achieves lower hardware utilization than Power Retention in early tests.)Both Power Retention and Mamba, he added, “expend meaningfully fewer total FLOPs than FlashAttention2 on long contexts, as well as far less memory.” According to Buckman, the reported 100× speedup comes from this combined improvement in utilization and computational efficiency, though he noted that “we have not yet stress-tested it on production-scale workloads.”Training and Scaling EconomicsPerhaps no statistic in the Brumby release generated more attention than the training cost.A 14-billion-parameter model, trained for $4,000, represents a two-order-of-magnitude reduction in the cost of foundation model development.Buckman confirmed that the low cost reflects a broader scaling pattern. “Far from diminishing returns, we have found that ease of retraining improves with scale,” he said. “The number of steps required to successfully retrain a model decreases with its parameter count.” Manifest has not yet validated the cost of retraining models at 700B parameters, but Buckman projected a range of $10,000–$20,000 for models of that magnitude—still far below transformer training budgets.He also reiterated that this approach could democratize large-scale experimentation by allowing smaller research groups or companies to retrain or repurpose existing transformer checkpoints without prohibitive compute costs.Integration and DeploymentAccording to Buckman, converting an existing transformer into a Power Retention model is designed to be simple. “It is straightforward for any company that is already retraining, post-training, or fine-tuning open-source models,” he said. “Simply pip install retention, change one line of your architecture code, and resume training where you left off.”He added that after only a small number of GPU-hours, the model typically recovers its original performance—at which point it gains the efficiency benefits of the attention-free design. “The resulting architecture will permit far faster long-context training and inference than previously,” Buckman noted.On infrastructure, Buckman said the main Brumby kernels are written in Triton, compatible with both NVIDIA and AMD accelerators. Specialized CUDA kernels are also available through the team’s in-house Vidrial framework. Integration with vLLM and other inference engines remains a work in progress: “We have not yet integrated Power Retention into inference engines, but doing so is a major ongoing initiative at Manifest.”As for distributed inference, Buckman dismissed concerns about instability: “We have not found this difficulty to be exacerbated in any way by our recurrent-state architecture. In fact, context-parallel training and GPU partitioning for multi-user inference both become significantly cleaner technically when using our approach.”Mission and Long-Term VisionBeyond the engineering details, Buckman also described Manifest’s broader mission. “Our mission is to train a neural network to model all human output,” he said. The team’s goal, he explained, is to move beyond modeling “artifacts of intelligence” toward modeling “the intelligent processes that generated them.” This shift, he argued, requires “fundamentally rethinking” how models are designed and trained—work that Power Retention represents only the beginning of.The Brumby-14B release, he said, is “one step forward in a long march” toward architectures that can model thought processes continuously and efficiently.Public Debate and Industry ReceptionThe launch of Brumby-14B sparked immediate discussion on X (formerly Twitter), where researchers debated the framing of Manifest AI’s announcement. Some, including Meta researcher Ariel (@redtachyon), argued that the “$4,000 foundation model” tagline was misleading, since the training involved reusing pretrained transformer weights rather than training from scratch.“They shuffled around the weights of Qwen, fine-tuned it a bit, and called it ‘training a foundation model for $4k,’” Ariel wrote.Buckman responded publicly, clarifying that the initial tweet had been part of a longer thread explaining the retraining approach. “It’s not like I was being deceptive about it,” he wrote. “I broke it up into separate tweets, and now everyone is mad about the first one.”In a follow-up email, Buckman took a measured view of the controversy. “The end of the transformer era is not yet here,” he reiterated, “but the march has begun.” He also acknowledged that the $4,000 claim, though technically accurate in context, had drawn attention precisely because it challenged expectations about what it costs to experiment at frontier scale.Conclusion: A Crack in the Transformer’s Wall?The release of Brumby-14B-Base is more than an engineering milestone; it is a proof of concept that the transformer’s dominance may finally face credible competition. By replacing attention with power retention, Manifest AI has demonstrated that performance parity with state-of-the-art transformers is possible at a fraction of the computational cost—and that the long-context bottleneck can be broken without exotic hardware.The broader implications are twofold. First, the economics of training and serving large models could shift dramatically, lowering the barrier to entry for open research and smaller organizations. Second, the architectural diversity of AI models may expand again, reigniting theoretical and empirical exploration after half a decade of transformer monoculture.As Buckman put it: “The end of the transformer era is not yet here. Our release is just one step forward in a long march toward the future.”
