A projected-out toxic subspace beats DPO more sample-efficiently

measured in 1 paper

Uppaal et al. introduce ProFS: from toxic/non-toxic pair embeddings, remove the corpus-mean direction, then SVD the centered differences and keep the top-k (ScreeNot-selected: k=2 GPT-2, k=10 larger models) as a toxic-subspace projector [uppaal-etal-2024-profs-detox-model-editing-dpo] Projecting it out of MLP value matrices once offline drops GPT-2 toxicity 48.0% to 26.8% using 500 pairs, below DPO 36.4% at 2,000 pairs, with comparable perplexity [uppaal-etal-2024-profs-detox-model-editing-dpo] Under label noise up to 50% flipped, DPO degrades monotonically while ProFS stays flat, a consequence of the SVD being provably invariant to individual pair sign-flips [uppaal-etal-2024-profs-detox-model-editing-dpo] The fraction of a single DPO gradient step explained by the toxic subspace rises with sample size and depth, framing ProFS as a denoised DPO step [uppaal-etal-2024-profs-detox-model-editing-dpo]

Structure

Linear Subspace

Context

corpus-mean-direction confound in raw contrastive-pair differences, removed by explicit centering before SVD, ScreeNot automatic rank selection for the toxic subspace (k=2 vs. k=10), SVD sign-flip invariance as the mechanistic explanation for noise robustness vs. gradient-based DPO, offline one-time weight-space projection vs. inference-time activation-space erasure (contrast with LEACE/INLP/RLACE), single-DPO-gradient-step variance increasingly explained by the toxic subspace as N and depth grow

Confirmed in models

GPT-2-Medium · GPT, Mistral-7B · Mistral, Mistral-7B-SFT-Beta · Mistral, OPT-6.7B · OPT, GPT-J-6B · GPT-J

Method

ProFS (Projection Filter for Subspaces) weight-subspace projection

Papers

Model Editing as a Robust and Denoised Variant of DPO: A Case Study on Toxicity — Uppaal, Rheeya, Dey, Apratim, He, Yiting, Zhong, Yiqiao, Hu, Junjie2024 · arXiv:2405.13967