Linear "modal difference vectors" that discriminate possible, improbable, impossible, and nonsensical sentences exist in language-model activations, emerge in a consistent order across training steps, layers, and model scale, and projections onto them track human categorization judgments

- Lepori et al. identify linear directions ("modal difference vectors", built by contrastive activation addition: the mean difference of final-token hidden states over minimal pairs, one vector per pair of modal categories among probable, improbable, impossible, inconceivable) from the Hu et al. (2025b) minimal-pair dataset in GPT2-{Small, Medium, Large, XL}, Llama-3.2-{1B, 3B}, OLMo-2-{1B, 7B, 13B} and Gemma-2-{2B, 9B}. For models with at least 2B parameters (Llama-3.2-3B, OLMo-2-7B/13B, Gemma-2-2B/9B) the vectors classify held-out pairs from three other datasets at least as well as sentence probability and better than principal-component and random-vector baselines; for models under 2B results are mixed, with probability sometimes better. [lepori-etal-2025-modal-difference-vectors] - The vectors emerge in a consistent order as models become more competent (training steps in OLMo-2-7B, layers, parameter count): inconceivable is separated first, then probable vs. impossible, then probable vs. improbable, and finally improbable vs. impossible. Projections onto three of the vectors form a 3-dimensional feature space whose logistic-regression fit models graded human categorization distributions better than probability, principal-component and random baselines, and ablating those features hurts the fit for the finest-grained dataset. Steering with the vectors (Gemma-2-2B and Llama-3.2-3B, 30 prefixes) is only preliminary evidence of a causal effect. [lepori-etal-2025-modal-difference-vectors]