= Word embedding
A <word embedding> represents a word by a <vector>, usually learned so that relationships in the data become useful relationships between <vectors>. A vocabulary of $V$ words can be stored as a <matrix> with $V$ rows and $d$ columns: selecting a word selects one row. Unlike a separate indicator for each word, a dense representation can share predictive information across words.
For example, training a <neural language model> can place words occurring in similar contexts near one another, making evidence about “cat” useful when predicting contexts involving “dog”. This is statistical similarity, not a guarantee of synonymy: opposite words can occur in similar contexts too. The 2003 neural probabilistic model learned these <vectors> jointly with next-word probabilities.<A Neural Probabilistic Language Model by Yoshua Bengio and others>[abstract and §2, shared word-feature matrix]{ref}
Back to article page
