A PCA spiral in Llama hidden states aligns with the periodic table
measured in 1 paperLei & Cooper apply PCA to Llama-3-8B and Llama-3.1-70B hidden states on chemistry prompts, finding a 3D spiral in the top components that tracks the periodic table's atomic-number ordering [lei-cooper-2025-layerwise-recall-and-the-geometry-of-interwoven-knowledge-in-llms] Per-property spiral-fit consistency scores reach up to about 0.90 for some element properties and lower (around 0.65) for others [lei-cooper-2025-layerwise-recall-and-the-geometry-of-interwoven-knowledge-in-llms] Layer-wise linear probing shows middle layers encode element attributes continuously and overlapping while deeper layers sharpen them into more categorically separated clusters [lei-cooper-2025-layerwise-recall-and-the-geometry-of-interwoven-knowledge-in-llms]
Structure
Context
a PCA-derived 3D trajectory whose shape is checked against an independent external conceptual ordering (atomic number / periodic table position) rather than only against a behavioral proxy, a depth-dependent continuous-to-categorical transition in linear-probe separability for related attributes, distinct from (but structurally similar to) other depth-wise separation/ superposition findings in this map
Confirmed in models
Method
Papers
Layerwise Recall and the Geometry of Interwoven Knowledge in LLMs — Lei, Ge, Cooper, Samuel J.