A trained rank-1 steering module rebalances multimodal attention

Bi et al. quantify a Layer-wise Modality Attention Ratio showing textual tokens dominate attention during generation in LLaVA-style multimodal LLMs under full fine-tuning and standard PEFT [bi-etal-2024-llava-steering-visual-subspace-modality-rebalancing] They introduce MoReS, a trained reversible linear map applied to visual tokens at every layer that projects into a rank-d subspace, transforms there, and projects back, with the base LLM frozen -- a learned add-on module, not a structure found in free activations [bi-etal-2024-llava-steering-visual-subspace-modality-rebalancing] Composed into LLaVA Steering models (Phi-2, Vicuna-7B/13B), it matches or exceeds LoRA/Adapter/OFT/IA3 across visual and VQA benchmarks at 287-1,150x fewer trainable parameters, reducing POPE hallucination [bi-etal-2024-llava-steering-visual-subspace-modality-rebalancing] A rank ablation (d in 1,2,4,8) finds the trained rank-1 module already scores highest (81.8 avg) with the fewest parameters, and steering just 1% of visual tokens is optimal [bi-etal-2024-llava-steering-visual-subspace-modality-rebalancing]