01 / Recurrent computation
The CTM Transformer
Finding
Every CTM variant beats the matched 8-layer Transformer by 0.040–0.056 nats but ties the best recurrent-depth Transformer under a pre-registered 0.02 rule — and tick-scaling extrapolation turned out to come from recurrence, not the CTM
A Transformer whose own layers take on Continuous Thought Machine mechanisms (arXiv:2505.05522) — internal ticks, neuron-level models, synchronization-driven attention — zero-initialized so it starts exactly as its baselines, then compared against the best recurrent-depth Transformer (arXiv:2502.05171) at matched size, 400M tokens, two seeds, and stress-tested on parity, adaptive compute and reasoning tasks inside text.
- Ablated each mechanism: sync queries, sync output and neuron-level models each tie the recurrent-depth baseline on their own; combining all three is slightly worse, at ~2.9× its cost per tick
- Reproduced the CTM's central claim on parity — a model that runs more ticks on longer inputs and reaches 3× the trained length — then showed it becomes reliable once training ticks grow with input length, and more often without the CTM mechanisms (4/10 seeds) than with them (2/10)
- Pre-registered nulls, reported as nulls: CTM loss across ticks (diagnosed a state-norm collapse), attention sink, homeostasis, certainty exit, per-token tick budgets (caught a label leak in my own difficulty signal) — language loss is flat beyond 8 ticks in every arm
- Now: reasoning tasks embedded in text at d704 — addition is learned (CTsq 99% in-distribution) but nothing extrapolates; running addition-reliability replicates and a reasoning-heavy data mix
Status: Active[GitHub ↗]