INLP subspace + AlterRep counterfactual push

methods / Causal Validation / INLP subspace + AlterRep counterfactual push

INLP subspace + AlterRep counterfactual push

Techniqueadvanced

Trains a set of mutually-orthogonal linear-classifier directions spanning a feature's 'subspace' (Iterative Null-space Projection), then generates counterfactual activations by pushing a real hidden vector further to one side of every classifier's separating hyperplane in that subspace (AlterRep) — a subspace-level, multi-direction generalization of a single diff-in-means push, applied mid-forward-pass.

Used in (10 observations)

AlterRep push on BERT's RC-boundary subspace shifts agreement errorsmeasured

structure: Linear Subspace · models: BERT-base-uncased, BERT-large-uncased, BERT-medium, BERT-small, BERT-mini, BERT-tiny · paper: Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction

Amnesic probing misleads at low class count and high dimensionmeasured

structure: Linear Subspace · models: RoBERTa-large-MNLI, RoBERTa-large-MNLI + HELP fine-tuning, RoBERTa-large-MNLI + double fine-tuning (HELP + NLI_XY), BERT-base-uncased fine-tuned on SNLI, BERT-base-uncased fine-tuned on SNLI + HELP · paper: Interventional Probing in High Dimensions: An NLI Case Study

A boundedness subspace causally shifts Russian aspect predictionsmeasured

structure: Linear Subspace · models: ruBERT-base, ruBERT-large, ruRoBERTa-large · paper: Probing the Category of Verbal Aspect in Transformer Language Models

Decodability does not imply behavioral use, shown by amnesic probingmeasured

structure: Linear Subspace · models: BERT-base-uncased · paper: Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

The gender subspace spans dozens of directions, not onemeasured

structure: Linear Subspace · models: GloVe (Wikipedia + Gigaword, uncased), DeepMoji encoder, BERT-base-uncased · paper: Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection

BERT causally uses a multi-dimensional number subspace for agreementmeasured

structure: Linear Subspace · models: BERT-base-uncased · paper: Probing for the Usage of Grammatical Number

INLP fails its own control while Mean Projection and LEACE passmeasured

structure: Linear Subspace · models: BERT-base-uncased · paper: Improving Causal Interventions in Amnesic Probing with Mean Projection or LEACE

Mnestic probing recovers signal where amnesic removal found nonemeasured

Nonlinear interventions are more reliable than linear subspace erasuremeasured

structure: Linear Subspace · models: BERT-base-uncased, GPT-2-small, Pythia-160M, Pythia-1.4B, Pythia-6.9B, Llama-3.2-3B-Instruct · paper: How Reliable are Causal Probing Interventions?

Temporal knowledge drift is a linear direction orthogonal to correctness and uncertaintymeasured

structure: Linear Direction · models: Llama-2-7B-Chat, Mistral-7B-Instruct-v0.1, Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Gemma-2-9B-it, Gemma-2-2B-it · paper: The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations