Baseline-corrected cosine similarity between intermediate hidden states of semantically-equivalent inputs (different languages, notations, code, images, audio) is elevated at middle layers relative to a non-equivalent-pair baseline, logit-lens decoding shows non-English/non-textual inputs pass through tokens of the model's dominant pretraining language, and causally patching an intermediate representation with its dominant-language equivalent predictably redirects downstream output across data types

Wu, Yu, Yogatama, Lu & Kim (2024) test whether LMs route semantically-equivalent inputs through a shared intermediate representation anchored by the dominant pretraining language, across Llama-2-7B/70B, Llama-3-8B/70B, and Baichuan-2. Baseline-corrected cosine similarity of intermediate hidden states for English-Chinese translation pairs is elevated at middle layers relative to a non-translation-pair control; logit-lens projections show intermediate representations of non-English and non-textual (code, arithmetic, vision, audio) inputs decode to the dominant language's semantically-equivalent tokens. Causal activation patching -- replacing an intermediate representation with the dominant-language- equivalent representation -- predictably shifts downstream output across all tested data types.