Small Tower-of-Hanoi transformers, Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B linearly encode a geometrically faithful Sierpinski-triangle state-space representation, and in the large models it decays over long planning

- Pereira and Zuidema train small GPT-2-style transformers (6 layers, d=128) on flat-to-flat Tower-of-Hanoi traces and show that they develop a linearly decodable, geometrically faithful, causally involved representation of the puzzle state space (the Sierpinski triangle). A linear distance-matching probe on the SEP-token residual stream reaches Spearman 0.94 (Pearson 0.90) at layer 5 on all 81 four-disk configurations, with a persistent Spearman-Pearson gap (the probe preserves distance order but inflates the separation between the three largest-disk sub-triangles); per-disk probes (row spans as encoding subspaces) show principal angles of about 55 degrees at SEP versus about 77 degrees at move tokens. Activation patching of the whole SEP residual stream transfers state to the recipient's output (mostly partially), so the causal evidence is graded and not surgical on the 2D representation. In two large reasoning models, Qwen3.6-27B and DeepSeek-R1-Distill-Qwen-32B, the same probe at the final prompt token gives Spearman about 0.935 with perfect nearest-state accuracy, but the joint configuration is less decodable at the commitment point and a factored per-disk encoding partly returns during move emission; steering with the prompt-time activation turns about half of Qwen3.6-27B's failures into optimal solutions (26 of 48 at the best strength) but barely helps DeepSeek-R1-Distill-Qwen-32B (6 of 72). Single puzzle (n=4), several figures from single seeds. Filed under linear-subspace as a linearly decodable state geometry (the Sierpinski triangle itself is a fractal graph embedding, not a linear subspace); the paper states only 'Qwen3.6-27B' without a variant, mapped to the existing qwen3.6-27b-instruct entry. [pereira-zuidema-2026-transformers-struggle-to-use-their-emergent-world-models-revisiting-the-tower-of-hanoi]