Partial Least Squares (PLS)

methods / Dimensionality Reduction / Partial Least Squares (PLS)

Techniqueintermediate

Supervised dimensionality reduction: projects activations onto the low-dimensional basis that maximizes covariance with a known target value (e.g. the number, or a date), rather than PCA's target-blind maximum-variance basis.

Used in (3 observations)

A grid-organized cell subspace supports relational binding in LLMsmeasured

structure: Linear Subspace · models: Llama-3-8B-Instruct, Qwen3-8B · paper: Cell-Based Representation of Relational Binding in Language Models

Answer-correctness sits in a 3-8D linear subspace, causally steerablemeasured

structure: Linear Subspace, Linear Direction · models: GPT-2 Small, GPT-2 Medium, GPT-2 Large, Qwen2-1.5B-Instruct, Qwen2-7B-Instruct, Llama-3.2-1B-Instruct, Llama-3.2-3B-Instruct, Mistral-7B-Instruct-v0.3, Gemma-2-2B-it · paper: The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models

Numbers lie on a logarithmically-compressed 1D number linemeasured

structure: 1D continuum manifold · models: Pythia-2.8B, Llama-2-7B, Llama-3.1-8B, Llama-3.2-1B, GPT-2-Large, Mistral-7B, Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-3B, Llama-3.2-3B-Instruct, Llama-3.2-3B, Gemma-2-2B-it, Gemma-2-2B, Llama-3.1-70B-Instruct, Llama-3-8B-Instruct, Llama-3-8B, Mistral-7B-Instruct-v0.3, Qwen2.5-7B-Instruct · paper: Number Representations in LLMs: A Computational Parallel to Human Perception, Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling, Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models, LLMs Know More About Numbers than They Can Say