In a real pretrained Mamba-370M state-space model's Indirect Object Identification circuit, name-entity information appears linearly represented at Layer 39's SSM input (averaged-activation replace / subtract-and-add edits overwrite the output name), after Layer 39's convolution shifts the relevant name one position forward

- Ensign & Garriga-Alonso adapt pre-Mamba mechanistic-interpretability techniques (including an adapted positional Edge Attribution Patching circuit-discovery tool) to a real pretrained Mamba-370M (state-spaces/mamba-370m, 48 layers, trained on The Pile) state-space model, partially reverse-engineering the circuit responsible for the Indirect Object Identification (IOI) task. [ensign-garriga-alonso-2024-mamba-ioi-circuit] - Layer 39 is identified as a key bottleneck for the circuit, and its convolutional component shifts the relevant name token one sequence position forward ahead of the SSM's own state update. [ensign-garriga-alonso-2024-mamba-ioi-circuit] - The authors hypothesize and report evidence that name entities are linearly represented in Layer 39's SSM: replacing the SSM input at a name's position with the averaged activation of a different name, or subtracting the current name's average and adding the other's, flips the output logits to the desired name (subtract-and-add: over 95% of the time; Sec. 4.3), though this is framed as suggestive and only for the first three name positions, and the intervention is on the SSM input (blocks.39.hook_ssm_input) rather than the recurrent state itself. The paper presents this as initial evidence that circuit-based mechanistic-interpretability tools work for the Mamba architecture. [ensign-garriga-alonso-2024-mamba-ioi-circuit]