One unconstrained linear map injects vision embeddings into frozen GPT-J
measured in 1 paperMerullo et al. fit a single unconstrained linear projection (no orthogonality constraint, trained by SGD) mapping vision-encoder embeddings (CLIP RN50x16, NFRN50, BEiT-Large) into frozen 6B GPT-J's token-embedding space [merullo-etal-2023-linearly-mapping-image-to-text-space] Only the linear map is trained; the mapped vectors are fed as soft prompts into the untouched, frozen GPT-J [merullo-etal-2023-linearly-mapping-image-to-text-space] This causally drives competitive zero-shot image captioning (NoCaps/COCO CIDEr-D) and few-shot VQA with no fine-tuning of either model [merullo-etal-2023-linearly-mapping-image-to-text-space] It demonstrates a measured linear relationship between two independently-trained representation spaces plus a genuine behavioral effect [merullo-etal-2023-linearly-mapping-image-to-text-space]