In-context learning induces higher intrinsic dimension than fine-tuning

Janapati & Ji apply TwoNN to last-token representations of Llama-3-8B, Llama-2-13B, Llama-2-7B, and Mistral-7B-v0.3 across 8 tasks, comparing ICL, LoRA fine-tuning, and zero-shot [janapati-ji-2024-learning-paradigms-via-intrinsic-dimension] ICL with k>=5 demonstrations induces a consistently higher intrinsic-dimension profile across all layers than SFT or zero-shot [janapati-ji-2024-learning-paradigms-via-intrinsic-dimension] This holds even though SFT reaches higher task accuracy (e.g. MMLU 0.542 vs ICL 0.531), dissociating performance from manifold dimensionality [janapati-ji-2024-learning-paradigms-via-intrinsic-dimension] ID versus number of demonstrations is non-monotonic, rising then plateauing past k approximately 5-10 [janapati-ji-2024-learning-paradigms-via-intrinsic-dimension]