MATH · IN · MODELS
methods / Causal Validation / PCA-derived semantic-subspace ablation, validated against a randomized-subspace control

PCA-derived semantic-subspace ablation, validated against a randomized-subspace control

Techniqueintermediate

Extracts a low-rank PCA subspace spanning a semantic category (e.g. country names, climate vocabulary) from a set of related embeddings, projects it out of a target embedding, and quantifies the causal drop in a downstream linear-probe's decodability — validated by comparison against ablating an equal-rank random subspace, so the effect can be attributed to the specific semantic content rather than merely to removing any generic low-rank chunk of the space.

Used in (2 observations)

structure: Linear Direction · models: GloVe (Wikipedia + Gigaword, uncased), word2vec (Google News, 300d) · paper: World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings
structure: Anisotropy · models: all-MiniLM-L6-v2, all-MiniLM-L12-v2, all-mpnet-base-v2, paraphrase-mpnet-base-v2, BGE-base-en-v1.5, BGE-large-en-v1.5, E5-large-v2, multilingual-e5-large, e5-mistral-7b-instruct, SFR-Embedding-Mistral, BERT-base-uncased, RoBERTa-base, ELECTRA-base, mBERT (BERT-base, Multilingual Cased), GPT-2-small, Pythia-410M, Qwen2.5-1.5B, Qwen2.5-7B, Mistral-7B · paper: Anisotropy Decides Cosine vs. Rank Metrics for Text Embeddings