Attractor Models: Solving the Recurrence Loop for Scalable Reasoning

TL;DR

Attractor Models rethink how AI "thinks" before it speaks. Instead of a fixed-depth pass (standard Transformers) or a fragile loop (Recurrent models), Attractor Models treat token generation as a search for an equilibrium. By combining a proposal backbone with a fixed-point attractor, this architecture achieves superior perplexity and reasoning capabilities at a fraction of the compute and memory cost of traditional looped models.

Background: The Problem with Iterative Thinking

Modern LLMs generate tokens in a single, rigid forward pass. While "Chain-of-Thought" allows for sequential reasoning, it forces the model to externalize every step into discrete text. Researchers have tried Looped Transformers—reusing the same layers to refine internal representations—but these models are notoriously hard to train. They consume massive memory (linear to the number of loops) and often "break" if you try to run more loops at inference than seen during training.

The authors of "Solve the Loop" identified a critical insight: Looped trajectories eventually converge to fixed points. If the goal is convergence, why not solve for it directly?

Methodology: The Backbone and The Attractor

The Attractor Model splits the workload into two specialized modules:

1. The Backbone: A standard Transformer that makes a high-quality "initial guess" ($ilde{y}_0$) in the output embedding space.
2. The Attractor: A smaller, weight-tied module that takes that guess and iteratively refines it until it reaches a stable state (the attractor), where further iteration no longer changes the value.

Figure 1: Unlike standard looped models that start from zero or noise, Attractor Models warm-start the search from a meaningful backbone proposal.

Constant Memory through Implicit Differentiation

Crucially, Attractor Models use the Implicit Function Theorem. Instead of storing every intermediate step's activations for backpropagation (which kills memory), they calculate gradients based on the final equilibrium. This makes training memory O(1) relative to complexity—a game-changer for scaling.

Breakthrough Phenomenon: Equilibrium Internalization

One of the most striking findings of this paper is what the authors call Equilibrium Internalization.

During training, the attractor module acts as a "teacher." Because the backbone is trained to produce a proposal that the attractor will refine, the backbone eventually learns to jump straight to the answer. By the end of training, the backbone's "initial guess" is often so close to the final fixed point that the attractor has almost nothing left to do.

Figure 2: At test time, the model is essentially converged at T=1, meaning it provides "deep thinking" quality at "fast thinking" speeds.

Experiments: Scaling and Reasoning

The researchers tested Attractor Models in two extreme regimes:

1. Large-Scale Language Modeling

At the 770M parameter scale, Attractor Models achieved a "Pareto improvement" over Transformers. Specifically, the 770M model outperformed a 1.3B parameter Transformer trained on twice the data.

2. Hard Reasoning (Sudoku & Mazes)

On tasks like Sudoku-Extreme, where frontier models like GPT-o3 and Claude 3.7 fail completely (0% accuracy), a tiny 27M Attractor Model reached 91.4% accuracy.

Figure 3: Comparison of performance on hard reasoning tasks. Attractor Models solve problems where frontier models collapse.

Critical Insight & Conclusion

Attractor Models prove that we don't need deeper models; we need smarter convergence. By framing representation as an equilibrium, we get the best of both worlds: the reasoning power of deep, iterative refinement during training, and the blinding speed of a streamlined backbone during inference.

The Takeaway: Recurrence shouldn't be a chain; it should be a destination. By teaching models to "internalize" their internal search for an answer, we move one step closer to architectures that can reason effectively without exploding in size.

Keep an eye on this space—if "Equilibrium Internalization" holds for multi-billion parameter models, the era of fixed-layer Transformers might be nearing its end.