LLMs bind belief triples via Ordering-ID subspaces localized by patching

Prakash et al. show Qwen2.5-14B-Instruct, Llama-3-70B-Instruct and Llama-3.1-405B-Instruct bind character-object-state triples by co-locating reference information as low-rank Ordering IDs in the state-token residual stream, plus a visibility ID for theory-of-mind [prakash-etal-2025-language-models-use-lookbacks-to-track-beliefs] Interchange-intervention (activation patching) localizes visibility source-reference alignment to layers 10-23 and the strongest binding-lookback alignment to layers 33-38 [prakash-etal-2025-language-models-use-lookbacks-to-track-beliefs] Answer-payload information is present only after layer 31, consistent across all 80 test cases [prakash-etal-2025-language-models-use-lookbacks-to-track-beliefs] The lookback mechanism duplicates reference information into an address and a pointer, retrieved later via attention [prakash-etal-2025-language-models-use-lookbacks-to-track-beliefs]