A Sudoku transformer builds substructure-level linear constraint directions

measured in 1 paper

Kniazev & Fijalkow train an 8-layer transformer on Sudoku solving traces and probe its residual stream at the clues-end token [kniazev-fijalkow-2026] Per-cell digit probes plateau at 80% (imperfect linear separability), but per-cell candidate probes reach 1.0 exact-match in mid-layers [kniazev-fijalkow-2026] Candidate probe directions align by shared substructure (cosine ~0.33 for one shared row/column/box, ~0.65 for two), consistent with each being a linear combination of substructure-level directions [kniazev-fijalkow-2026] Substructure "digit present in row/column/box" probes reach perfect accuracy, and patching only their direction components flips the top-1 prediction in 99% of cases [kniazev-fijalkow-2026] Mid-layer heads specialize by substructure, and a final-MLP set of monosemantic "naked-single" neurons commits the answer [kniazev-fijalkow-2026]

Structure

Linear Direction, Constraint-Algebra Basis Hypothesis

Context

world models, constraint satisfaction, Sudoku, toy/synthetic sequence model, attention circuits, monosemantic neurons, mechanistic interpretability

Confirmed in models

Sudoku Transformer (8 layers, 8 heads, d_model=576) · Sudoku Transformer

Method

Linear probing, Activation patching, Causal interventions (steering), Direct Logit Attribution (DLA), Logit Lens

Papers

Transformers Linearly Represent Highly Structured World Models — Kniazev, Roman, Fijalkow, Nathanaël2026 · arXiv:2605.18847