Grid-walker toy transformer — Math in LLMs Research Map

MATH · IN · MODELS

Grid-walker toy transformer

Brenner, Knosche & Scherf (2026); custom-trained from scratch on 2D-lattice random walks

Structures found in this family (2)

Linear Subspace, Attention–MLP Sufficiency Staging Hypothesis

By model (1)

Grid-walker decoder transformer (L4/H4/d_model=128, HookedTransformer)

Linear Subspace, Attention–MLP Sufficiency Staging Hypothesis

Observations (1)

Attention builds a shared world-state coordinate system; the MLP specializes it

Papers

Predictive Statistics Shape Emergent World Representations of Grid Walkers (2026)