A truth-encoding linear subspace emerges via a two-phase memorize-then-linearly-separate training dynamic, and the same behavioral signature appears in a real pretrained LM

- Ravfogel et al. build a transparent, one-layer transformer toy model that reproduces truth subspaces end-to-end, isolating a concrete route by which the linear subspaces separating true from false statements (previously only observed post-hoc by probing) can arise: a data distribution where factual statements co-occur with other factual statements, so encoding a truth bit lowers language-modeling loss. [ravfogel-etal-2025-emergence-linear-truth-encodings] - In the toy setting, training shows a two-phase learning dynamic: the network first memorizes individual factual associations in a few steps, then -- over a much longer horizon -- learns to linearly separate true from false, which further lowers the LM loss; this dissociates memorization and the emergence of the linear subspace as two distinct training-dynamics phases. [ravfogel-etal-2025-emergence-linear-truth-encodings] - The authors present the toy model as a proof of concept and note it is 'not likely that the same mechanism applies to real LMs'. In pretrained Llama3-8B on CounterFact (SPEAKSLANGUAGE relation) they report that a linear classifier reaches over 95% accuracy separating true from counterfactual instances on all middle and last layers, that two false context sentences (vs. one true) raise the NLL of the correct attribute by 1.52 (about 4.55x lower probability; 128 tuples), and that adding a mean-difference steering vector alpha(mu_T - mu_F) at layer 11 (alpha = 3.0) tends to raise the correct-attribute probability under false context. The two-phase dynamic itself is shown only in the authors' own small from-scratch transformers (synthetic data and CounterFact-based training), not in Llama3-8B. [ravfogel-etal-2025-emergence-linear-truth-encodings]