A summarization motif routes sentiment through non-valenced tokens

Using iterative path-patching and value-weighted attention-pattern analysis, Tigges et al. trace how sentiment information flows through GPT2-small (and Pythia-2.8B) [tigges-etal-2023-linear-sentiment] A "summarization motif" accumulates sentiment onto intermediate non-valenced tokens such as commas and periods rather than only onto the sentiment-bearing words [tigges-etal-2023-linear-sentiment] These summarization points act as intermediate stores that later attention heads read from to produce the final sentiment prediction [tigges-etal-2023-linear-sentiment]