Morphology stays linearly separable while lexical identity turns nonlinear with depth

measured in 1 paper

Li & Subramani train ridge-regression linear probes and nonlinear MLP probes across 25 pretrained models to predict word lemma and inflectional features [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Lemma is linearly separable in early layers (0.8-1.0) but linear accuracy declines toward final layers while MLP accuracy declines much less, so lexical identity becomes increasingly nonlinearly encoded [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Inflectional morphology stays linearly separable at 0.9-1.0 uniformly across every layer, with MLP probes adding no benefit [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models] Control-task selectivity confirms inflection is a generalizable linear abstraction (selectivity 0.4-0.6) while lemma decodability reflects near-zero-selectivity memorization [li-subramani-2025-finding-lexical-identity-and-inflectional-morphology-in-modern-language-models]

Structure

Linear Separability

Context

a depth-dependent dissociation between two lexical properties' encoding geometry -- one staying linear throughout, the other becoming progressively nonlinear -- within the same set of models, control-task selectivity distinguishing genuine linear abstraction from probe memorization for each property separately

Confirmed in models

BERT-base-uncased · BERT, BERT-large-uncased · BERT, GPT-2-small · GPT, GPT-2-Large · GPT, GPT-2-XL · GPT, Pythia-6.9B · Pythia, OLMo 2 7B · OLMo 2, Gemma-2-2B · Gemma, Qwen2.5-1.5B-Instruct · Qwen, Llama-3.1-8B · Llama

Method

Linear probing, Mlp probing

Papers

Finding Lexical Identity and Inflectional Morphology in Modern Language Models — Li, Michael, Subramani, Nishant2025 · arXiv:2506.02132