Weight-tied embeddings degenerate into a narrow cone

- In real trained NLG models with tied input-output embeddings, the word vectors degenerate into a narrow cone with nearly all pairwise cosine similarities positive. [gao-etal-2019-representation-degeneration-problem] - The cause is MLE under weight tying: non-target (especially rare and unseen) word embeddings are pushed toward a shared negative direction of the hidden states, formally when the convex hull of hidden states excludes the origin (made likely by layer normalization). [gao-etal-2019-representation-degeneration-problem] - A cosine-similarity regularizer, MLE-CosReg (gamma=1, penalizing mean pairwise cosine, not "MMI"), flattens the singular-value spectrum and improves WMT'14 BLEU (+1.08 En-De, +0.93 De-En) and WikiText-2 perplexity (52.0 to 50.0). [gao-etal-2019-representation-degeneration-problem] - Studied on Transformer base/big NMT (WMT'14 En-De/De-En) and an AWD-LSTM LM (WikiText-2). [gao-etal-2019-representation-degeneration-problem]