Re-expressing embeddings as cosine similarity to a fixed anchor set yields a rotation/reflection/rescaling-invariant coordinate system in which decoders and classifiers trained on one independently-trained encoder can be stitched, with zero fine-tuning, onto a completely different independently-trained encoder's representations of the same domain

Moschella, Maiorca, Fumero, Norelli, Locatello & Rodolà (2022) define the relative representation of an input x as its vector of cosine similarities to a fixed set of anchor points from the same domain: r_x = (sim(e_x, e_a1), ..., sim(e_x, e_a|A|)), with sim typically cosine similarity S_C(a,b) = ab/(||a|| ||b||) (vector quantization is explored as an alternative in an appendix). This construction is invariant to any rotation, reflection, or positive rescaling applied uniformly to the encoder's output space -- exactly the transformations that make two independently-trained encoders' raw coordinates incomparable -- with translation invariance recoverable separately via a normalization layer (e.g. InstanceNorm). Diagnostically, across ~2,000 independently-trained Cora GCN classifiers varying seed, epoch count, layer depth, dropout, activation, optimizer, learning rate, and embedder type, the Pearson correlation between each model's relative- representation cosine similarity to a fixed reference model and its own classification accuracy is 0.955 (after filtering models with best validation accuracy below 0.5) -- models that reach similar accuracy converge, in relative coordinates, to highly similar representations. Constructively, decoders/classifiers trained on one encoder's relative representations are spliced with zero fine-tuning onto a different, independently-trained encoder's relative representations of the same domain ("zero-shot stitching"), and recover most non-stitched performance while absolute-representation stitching performs at or near chance throughout: image reconstruction MSE (Table 3, 5 seeds) drops from an Absolute-Stitch baseline of 97.79/120.54/86.74/97.17 (MNIST/F-MNIST/CIFAR-10/CIFAR-100, AE) to a Relative-Stitch 2.83/6.37/5.39/18.03; cross-lingual text classification F1 (Table 4, RoBERTa, Amazon Reviews coarse, English decoder) drops from 91.54 (English, non-stitched) to 43.67/54.41/48.72 (Spanish/French/Japanese, Absolute-Stitch) but only to 90.06/82.78/ 78.49 with Relative-Stitch (translated anchors); cross-architecture image classification (Table 6, CIFAR-100/ImageNet1k, ViT-Base decoder + ViT-Small-ResNet50 encoder direction) goes from Absolute-Stitch 6.21/0.07 to Relative-Stitch 81.42/44.72 (and the reverse direction 4.69/0.08 to 84.46/62.21). The paper explicitly summarizes: "Stitching with absolute representations yields performances comparable to random guessing across the board." Typical anchor-set sizes are 300 (word embeddings, Cora) or 500 (cross-architecture vision, matching the larger ViT-Base's 768 dimensions); an ablation (Figure 6/Appendix A.2) shows performance improves monotonically with more anchors for frozen encoders but is less stable for end-to-end-trained ones (Cora), which the authors attribute to greater susceptibility to model collapse. This is among the earliest constructive demonstrations of representational convergence across independently-trained networks: unlike vec2vec/mini-vec2vec, which *learn* a translation map between two embedding spaces, this shows that for a sufficiently invariant choice of coordinates (cosine similarity to a shared anchor set), no learned map is required at all -- only a shared anchor domain, no paired correspondence, and no training.