Structural probes show staged phonemic-then-lexical-then-syntactic subspaces

Orhan et al. train Hewitt-Manning distance-based structural probes to recover phoneme, WordNet lexical-semantic and syntactic-tree subspaces from wav2vec2 (tiny/base/large) and text models (Llama-2-13B, Pythia), tracked across pretraining [orhan-etal-2026-emergence-of-phonemic-syntactic-and-semantic-representations-in-artificial-neural-networks] Both modalities show staged emergence: phonemic structure first, then lexical, then syntactic, mirroring child language-acquisition order but needing 2-4 orders of magnitude more data [orhan-etal-2026-emergence-of-phonemic-syntactic-and-semantic-representations-in-artificial-neural-networks] On Llama-2-13B the syntactic and semantic subspaces are near-orthogonal (mean-squared cosine of principal angles 0.038-0.042), so they are distinct low-dimensional subspaces [orhan-etal-2026-emergence-of-phonemic-syntactic-and-semantic-representations-in-artificial-neural-networks] A contrastive topology-preserving probe confirms the distance-based probe validity [orhan-etal-2026-emergence-of-phonemic-syntactic-and-semantic-representations-in-artificial-neural-networks]