In LLaMA-style language models trained from scratch, grammatical concept vectors become more predictive of acceptability and occupy a higher-dimensional subspace after delayed (grokking-like) generalization on BLiMP phenomena

- Muckatira et al. train LLaMA-style decoder-only language models, define an exposure-based proxy train/validation split on BLiMP minimal pairs, and observe delayed generalization across five grammatical phenomena; comparing checkpoints before and after generalization (main analyses on the 60M model, 3 seeds; 35M and 130M only as robustness checks for the behavioural lag; concept vectors are final-layer last-token differences between acceptable and unacceptable sentences), mean-difference concept vectors generally become more separable (AUROC, with phenomenon-dependent strength; Regular Subject-Verb Agreement 2 stays close to chance) and the effective rank of the mean-centered concept-vector matrix is higher after generalization than at step 100 in all five phenomena (it fluctuates during the transition window), which the authors read as the concept being "represented across a higher-dimensional subspace". [muckatira-etal-2026-pretraining-grokking-analogue] - Attention from the critical token to the relevant context token concentrates in a small number of heads. The dimensionality result is reported without a published control for rank growth over training in general, so it should be read as a directional finding about these small from-scratch models. [muckatira-etal-2026-pretraining-grokking-analogue]