Real trained transformers linearly embed the theoretically predicted fractal belief-state geometry

Shai, Marzen, Teixeira, Gietelink Oldenziel & Riechers derive the Mixed-State Presentation (MSP) analytically for known epsilon-machines such as Mess3 and RRXOR, a curved, often fractal, self-similar image of the belief simplex under repeated Bayesian updating [shai-etal-2024-transformers-represent-belief-state-geometry-residual-stream] Real GPT-2-style transformers trained from scratch by gradient descent on sequences generated by these processes linearly embed the predicted MSP geometry in their residual stream, recovered via linear regression from theoretically-derived belief coordinates to the network's own activations [shai-etal-2024-transformers-represent-belief-state-geometry-residual-stream] Next-token predictions decompose as the predicted barycentric combination of per-vertex predictive distributions, weighted by the recovered belief coordinates, matching the operational signature of the Belief State Geometry Hypothesis [shai-etal-2024-transformers-represent-belief-state-geometry-residual-stream]