LLMs consolidate entity-state tracking into a local binary subspace

Tang et al. train linear probes on last-token residual activations (the "boxes" dataset) to test global versus local entity-state tracking [tang-etal-2026-do-language-models-track-entities-across-state-changes] Global 8-way state probes reach only ~0.3 accuracy versus ~0.9 for local binary probes, so the model consolidates state locally rather than tracking a full global state [tang-etal-2026-do-language-models-track-entities-across-state-changes] INLP null-space projection of the state-tag direction causally changes completions [tang-etal-2026-do-language-models-track-entities-across-state-changes] A learned sparse boolean mask over the residual PCA basis ("subspace patching") reveals a shared positional-information subspace at layers 15-25 in CodeLlama-13B [tang-etal-2026-do-language-models-track-entities-across-state-changes]