Sentiment RNNs converge on a readout-aligned line-attractor mechanism

Maheswaranathan et al. reverse-engineer trained LSTM, GRU, UGRNN, and vanilla RNNs on binary sentiment (Yelp, IMDB, SST) by finding ~500 approximate fixed points and linearizing each via its Jacobian [maheswaranathan-etal-2019-line-attractor] Training compresses the state onto a low-dimensional subspace (top 2-3 PCs explain ~90% of a trained LSTM's variance), and the fixed points lie along an approximately 1D manifold [maheswaranathan-etal-2019-line-attractor] The manifold is a genuine line attractor: state velocity decays to zero, the top Jacobian eigenvalue clusters near |lambda|~1 with a time constant of hundreds-to-thousands of tokens, and its direction aligns with the readout weights above a random null [maheswaranathan-etal-2019-line-attractor] Positive- and negative-valence words produce large opposite-sign projections onto the top eigenmode while neutral words project near zero, implementing evidence accumulation along the attractor [maheswaranathan-etal-2019-line-attractor] The same mechanism recurs across all four architectures and three datasets despite independent training, a universality of mechanism [maheswaranathan-etal-2019-line-attractor]