A single sentiment neuron in a byte-level mLSTM linearly encodes and steers sentiment

Radford et al. train a 4,096-unit byte-level multiplicative LSTM unsupervised as a next-byte model on ~82M Amazon reviews [radford-etal-2017-sentiment-neuron] A logistic-regression probe reaches 91.8% on binary SST, and a single hidden unit's activation alone reaches comparable accuracy [radford-etal-2017-sentiment-neuron] Clamping/overwriting the sentiment neuron during generation flips the sentiment of the model's own generated review text [radford-etal-2017-sentiment-neuron] Later replication found ablating the neuron only marginally affects downstream classification, questioning how exclusively sentiment is localized to one unit [radford-etal-2017-sentiment-neuron]