A delta-loss crosscoder isolates 1-2 causal directions per fine-tuning organism

measured in 1 paper

Kassem et al. train a BatchTopK crosscoder with a delta-loss prioritizing directions that change between a base and fine-tuned model, ranked by Relative Decoder Norm [kassem-etal-2026-delta-crosscoder-robust-model-diffing] Across 10 fine-tuning model organisms in Llama-3.1-8B-Instruct, Gemma-2-9B-IT, and Qwen2.5-7B, it isolates exactly 1-2 causal directions per organism [kassem-etal-2026-delta-crosscoder-robust-model-diffing] It recovers the causal direction in 10/10 cases with 0% false positives, versus 40% and 60% false-positive rates for DSF and BatchTopK baselines [kassem-etal-2026-delta-crosscoder-robust-model-diffing] Adding or subtracting the isolated latent's decoder vector at inference causally mitigates the fine-tuned behavior [kassem-etal-2026-delta-crosscoder-robust-model-diffing]

Structure

Linear Direction

Context

delta-loss crosscoder, fine-tuning-organism diffing, directional false-positive rate

Confirmed in models

Llama-3.1-8B-Instruct · Llama, Gemma-2-9B-it · Gemma, Qwen2.5-7B · Qwen

Method

Sparse crosscoders, Causal interventions (steering)

Papers

Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes — Kassem, Aly M., Jiralerspong, Thomas, Rostamzadeh, Negar, Farnadi, Golnoosh2026 · arXiv:2603.04426