DuoRec diagnoses and corrects anisotropic item-embedding degeneration in sequential recommenders

- In SASRec-style Transformer sequential recommenders, the authors observe "representation degeneration": item embeddings projected to 2D by SVD form a narrow cone (rare items bunch in a narrow region, positive along one axis), and the normalized singular values of the item embedding matrix drop quickly, which the authors call approximately "extreme low rank" and anisotropic. [qiu-etal-2021-duorec-representation-degeneration] - DuoRec's contrastive regularization (dropout-based model-level augmentation as unsupervised positives plus same-target-item sequences as supervised positives) gives item embeddings that are distributed more uniformly around the origin with more slowly decaying singular values (Figs. 1, 3, 4; Amazon Clothing and Sports), and the uniformity loss of sequence representations decreases during DuoRec training (Fig. 5). [qiu-etal-2021-duorec-representation-degeneration] - DuoRec attains higher HR@5/10 and NDCG@5/10 than SASRec, BERT4Rec, S3Rec and CL4SRec on all five datasets (Amazon Beauty, Sports, Clothing, MovieLens-1M, Yelp); the paper does not isolate the embedding geometry as the cause of the accuracy gain beyond ablations over contrastive-sample variants (UCL, SCL, full). [qiu-etal-2021-duorec-representation-degeneration]