Studies the eigenvectors/eigenvalues of the (normalized) word co-occurrence or pointwise mutual information matrix to predict the geometry of learned representations before training any model.
Used in (5 observations)
structure: Circle · models: Grokking Modular-Arithmetic Transformer (2 layers, 4 heads, pre-LN, d_model=128, mod 113/149/197) · paper: Circuit Synchronization Precedes Generalization: A Causal Precursor to Grokking
structure: Affine Subspace · models: Llama-2-7B, Llama-2-13B, Llama-2-70B, Pythia-6.9B, Gemma-2-2B, EmbeddingGemma, word2vec (trained on Wikipedia) · paper: Language Models Represent Space and Time, Symmetry in Language Statistics Shapes the Geometry of Model Representations
structure: Concept lattice (Formal Concept Analysis half-space model), Linear Direction · models: Gemma-2B, word2vec · paper: Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence
structure: Concept Crystals (Parallelogram/Trapezoid Structure) · models: · paper: On the Emergence of Linear Analogies in Word Embeddings
structure: 1D continuum manifold · models: Gemma-2-2B, EmbeddingGemma, word2vec (trained on Wikipedia), Claude 3.5 Haiku · paper: Symmetry in Language Statistics Shapes the Geometry of Model Representations, When Models Manipulate Manifolds: The Geometry of a Counting Task