Definition

Fix a network’s activation space $\mathbb{R}^d$ and a discrete labeling of inputs by some ground-truth latent state $y \in \mathcal{Y}$ (e.g. the full configuration of a game board). The labeled classes $\{x : y(x) = c\}_{c\in\mathcal{Y}}$ exhibit nonlinear world-model decodability if a nonlinear probe $f_\theta$ (e.g. a 1-2 hidden-layer MLP) trained on activations reliably predicts $y$, while the same classes fail the Linear Separability test — no single hyperplane per class (or small set of hyperplanes) achieves comparable accuracy. Reliability is typically established two ways: decoding accuracy of $f_\theta$ against the true $y(x)$, and causal validity — intervening on the activations to flip $f_\theta$‘s decoded state and checking that the network’s own downstream behavior (e.g. its next move) changes consistently with the intervention, ruling out the probe merely fitting an epiphenomenal correlate.

Relative to linear-separability

This is the negative/nonlinear counterpart of Linear Separability, not an unrelated finding: the same labeled classes are being tested, but the question is whether the representation itself only becomes causally identifiable class-by-class once probed with a nonlinear function, versus already being linearly readable in the model’s own coordinate basis. A representation can migrate from one regime to the other under a change of basis (not of the network) — the canonical case in this map is Othello-GPT, whose board-state representation was first reported nonlinear in this basis, then shown to be linear once probed in the network’s own relative (mine/yours) frame rather than the human-preconceived absolute (black/white) one — see othello-relative-frame-direction. Nonlinear decodability by itself is therefore evidence of some systematic, causally load-bearing structure, but does not by itself imply either linear or nonlinear geometry is the “true” shape — only that no linear readout was found in the basis tested.

Key evidence

The original Othello-GPT world-model finding: small-MLP probes reliably decode the full board state from a real trained transformer, and activation patching that flips the decoded state causally changes subsequent move predictions. Li, Hopkins, Bau, Viégas, Pfister & Wattenberg (2023) train an 8-layer, 8-head, 512-dim GPT-style transformer on Othello move sequences (synthetic self-play and championship human games) and show nonlinear (MLP) probes decode the full board state reliably at every move — a nonlinear emergent world-model representation. Activation-patching interventions that flip the probe-decoded board state causally change the model’s subsequent move predictions (validated via move-legality checks, ~0.01% error synthetic / ~5.17% championship, and latent-saliency-map analysis). This is the foundational paper behind Nanda, Lee & Wattenberg (2023)‘s later linear reframing of the same phenomenon in the model’s relative frame (already in this map, see othello-relative-frame-direction) and behind Yuan & Søgaard (2025)‘s cross-architecture Procrustes-alignment follow-up. See li-etal-2023-nonlinear-probes-decode-a-full-othello-board-state-from-othello-gpts-activations-and-activation-patching-that-flips-the-decoded-state-causally-changes-subsequent-move-predictions.

A real architecture-dependent split between the linear and nonlinear regimes, without any change of probing basis. Zhang (2026) probes two real RL world models trained on real Atari Breakout/Pong replay data — IRIS (discrete VQ-VAE-token transformer) and DIAMOND (continuous diffusion UNet) — for game-state variables (ball/paddle position, score) using both linear and nonlinear (MLP) probes at every layer. IRIS is approximately linearly decodable throughout (selectivity gap between MLP and linear probe accuracy, Delta <= 0.06 at every layer), while DIAMOND shows an “inverted-V”: linear decodability peaks only at the UNet’s compression bottleneck and requires a nonlinear probe everywhere else in the network — the same discrete-vs-continuous architectural distinction determines which regime a given real world model falls into, rather than a change of basis. Causal validation: activation patching along the probe-derived direction shifts predicted next-state token distributions consistent with the intervention (r=0.97), and multi-baseline token-ablation on IRIS’s spatial token grid identifies score/brick-region tokens as most causally important (rank consistency rho > 0.92). See zhang-2026-real-atari-world-models-iris-is-linearly-decodable-throughout-while-diamond-needs-nonlinear-probes-except-at-its-unet-bottleneck-both-causally-validated-via-activation-patching.

Key papers

• Li, Hopkins, Bau, Viégas, Pfister & Wattenberg (2023). Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task. arXiv:2210.13382 — origin of the finding above.
• Nanda, Lee & Wattenberg (2023). Emergent Linear Representations in World Models of Self-Supervised Sequence Models. arXiv:2309.00941 — shows the same OthelloGPT board state becomes linear under a change of probing basis; see othello-relative-frame-direction and Linear Representation Hypothesis.