Negation and intensification deflect sentiment RNNs into an orthogonal 2D subspace

Maheswaranathan & Sussillo extend fixed-point analysis with input-Jacobian modulation to explain how negation and intensification modify sentiment-RNN meaning [maheswaranathan-sussillo-2020-contextual-processing] Modifier words deflect the hidden state into a 2D subspace orthogonal to the line attractor (rather than moving along it as valence words do), with the top-2 PCs explaining 96.2% of variance [maheswaranathan-sussillo-2020-contextual-processing] The modifier subspace has two internal timescales (tau 2.6 and 4.3 tokens), with negators and intensifiers in distinguishable regions [maheswaranathan-sussillo-2020-contextual-processing] Projecting the initial state out of the 2D modifier subspace drops accuracy while a random 2D projection has no effect, a placebo-controlled causal ablation, and an augmented bag-of-words baseline with modifier convolution recovers over 90% of the RNN's gain [maheswaranathan-sussillo-2020-contextual-processing]