User gender, race and class directions from linear probes in four LLMs track population statistics and causally steer outputs

Logistic-regression probes on last-token residual activations of synthetic dialogues decode user gender, race and class with AUC-ROC above 0.98 (Magistral 24B, Qwen3 14B, GPT-OSS 20B) and above 0.92 (OLMo2-1B) at the best layer [bouchaud-ramaciotti-2025] Best layers are 20/40 (Magistral), 33/40 (Qwen3), 13/28 (GPT-OSS) and 13/16 (OLMo2); a one-hidden-layer MLP probe on OLMo gave no significant improvement over the linear probe [bouchaud-ramaciotti-2025] Applied to occupation mentions, the gender readout correlates with the fraction of women employed (Spearman 0.866 Qwen, 0.823 GPT-OSS, 0.785 Magistral) and the class readout with median hourly wage (0.701 GPT-OSS, 0.430 Magistral, 0.396 Qwen) [bouchaud-ramaciotti-2025] Adding the probe-derived attribute direction to the residual stream at the best layer shifts answers monotonically with the coefficient, e.g. parents' origin from Nigeria toward France along the race direction [bouchaud-ramaciotti-2025] Evidence is correlational plus steering on synthetic first-person prompts with binary attributes; the authors state it does not establish causation from training data to representation [bouchaud-ramaciotti-2025]