MATH · IN · MODELS

BLOOM

BigScience

Structures found in this family (4)

Linear Direction, Anisotropy, Intrinsic-dimension profile across depth, Platonic Representation Hypothesis

By model (6)

BLOOM-3B · 3B
Anisotropy, Intrinsic-dimension profile across depth, Platonic Representation Hypothesis
BLOOM-560M · 560M
Anisotropy, Platonic Representation Hypothesis, Linear Direction
BLOOM-1.1B · 1.1B
Platonic Representation Hypothesis
BLOOM-1.7B · 1.7B
Platonic Representation Hypothesis
BLOOM-176B · 176B
Linear Direction
BLOOM-7.1B · 7.1B
Platonic Representation Hypothesis

Observations (5)

LLMs implement word2vec-style relational vector offsets, causally patchable
Decoder-only models peak anisotropy in middle layers
Embedding intrinsic dimension expands early in pretraining, then compresses
Cross-model and cross-modal representational alignment increases with scale and competence
The Tuned Lens replaces the logit lens with a trained per-layer affine correction

Papers

Language Models Implement Simple Word2Vec-style Vector Arithmetic (2024), The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024), The Platonic Representation Hypothesis (2024), Eliciting Latent Predictions from Transformers with the Tuned Lens (2023)