Decoder-only models peak anisotropy in middle layers

measured in 1 paper

- Decoder-only models develop a bell-shaped anisotropy profile peaking in the middle layers, unlike encoders' flatter profile, measured by an SVD anisotropy score sigma_1^2/sum(sigma_i^2). [razzhigaev-etal-2024-shape-of-learning] - Intrinsic dimension (TwoNN) follows a two-phase pattern of early growth then decline, and decoder anisotropy rises through pretraining before stabilizing. [razzhigaev-etal-2024-shape-of-learning] - Spans encoders (BERT, RoBERTa, ALBERT) and decoders (OPT, Llama-2 and Llama-2-chat, GPT-2, GPT-J, Falcon-7B and Falcon-7B-Instruct, BLOOM, Pythia); observational on enwik8 character-level embeddings. [razzhigaev-etal-2024-shape-of-learning]

Structure

Anisotropy

Context

encoder-decoder-contrast

Confirmed in models

BERT-base-uncased · BERT, RoBERTa-base · RoBERTa, ALBERT-base-v1 · ALBERT, GPT-2-small · GPT, GPT-J-6B · GPT-J, OPT-13B · OPT, Llama-2-7B · Llama, Llama-2-7B-Chat · Llama, BLOOM-560M · BLOOM, BLOOM-3B · BLOOM, Pythia-2.8B · Pythia, Falcon-7B · Falcon, Falcon-7B-Instruct · Falcon

Method

SVD, Intrinsic dimension estimation (TwoNN)

Papers

The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models — Razzhigaev, Anton, Mikhalchuk, Matvey, Goncharova, Elizaveta, Oseledets, Ivan, Dimitrov, Denis, Kuznetsov, Andrey2024 · arXiv:2311.05928