ActAdd: contrast-pair activation differences added at the first token positions steer topic, sentiment and toxicity in GPT-2-XL, OPT-6.7B and LLaMA-3-8B

Turner et al. compute a steering vector as the difference of the residual-stream activations of a prompt pair (e.g. "Love" minus "Hate") and add it at the first token positions of a later forward pass, in GPT-2-XL, OPT-6.7B and LLaMA-3-8B; the single-token pair vectors act as one vector of the activation space [turner-etal-2024-actadd] Topic steering reaches more than 90% success against a baseline of about 2%, and detoxification lowers the toxicity score from .134 to .112 on OPT and from .114 to .108 on LLaMA-3 [turner-etal-2024-actadd] For multi-token contrast pairs the steering vector is a sequence of per-position differences and no pooled single vector is tested, so the evidence concerns single-token pairs; the linear-direction reading itself is only an interpretive hypothesis in the paper (Sec. 5) [turner-etal-2024-actadd]