01 / Recurrent computation
The CTM Transformer
Finding
Beats the matched 8-layer Transformer by 0.040 nats on both seeds — but ties the best recurrent-depth Transformer (+0.014) under a pre-registered 0.02 rule
A Transformer whose own layers take on Continuous Thought Machine mechanisms (arXiv:2505.05522) — internal ticks, neuron-level models, synchronization-driven attention — zero-initialized so it starts exactly as its baselines, then compared against the best recurrent-depth Transformer (arXiv:2502.05171) at matched size, 400M tokens, two seeds.
- Withdrew my own earlier headline: the 0.157-nat tick gain came from a non-residual readout — with a residual path the ticks were worth 0.012
- Diagnosed a separate-head CTM's dead ticks: the gain was a quadratic feature of the start state, with tiny tick states and decays stuck at 0 trapping the synchronization readout
- Tuned a recurrent-depth recipe that beats its matched Transformer by 0.054 nats on both seeds and lands within ~0.02 of a 20-layer Transformer at equal compute
- Now running per-mechanism ablations (sync queries, sync output, neuron-level models), rules fixed before the runs
Status: Active[GitHub ↗]