Linear analogies hold when shifted PMI is constant

- A linear word analogy holds exactly iff a co-occurrence shifted PMI, csPMI(x,y) = PMI(x,y) + log p(x,y), is constant across all pairs in the analogy set, with the words contextually coplanar. [ethayarajh-etal-2019-linear-word-analogies] - Equivalently, squared Euclidean distance between two word vectors is a linear function of negative csPMI, which is why analogies form approximate parallelograms. [ethayarajh-etal-2019-linear-word-analogies] - Confirmed on SGNS vectors trained on Wikipedia: distance vs negative csPMI correlates at Pearson r=0.502 (0.514 normalized), and analogy accuracy tracks csPMI variance at r=-0.70. [ethayarajh-etal-2019-linear-word-analogies] - The derivation is for SGNS only; the authors state it cannot be repeated for GloVe (learned biases are unidentified), so GloVe is treated theoretically, not tested. [ethayarajh-etal-2019-linear-word-analogies]