Word embedding

A word embedding represents a word by a vector, usually learned so that relationships in the data become useful relationships between vectors. A vocabulary of $V$ words can be stored as a matrix with $V$ rows and $d$ columns: selecting a word selects one row. Unlike a separate indicator for each word, a dense representation can share predictive information across words.

For example, training a neural language model can place words occurring in similar contexts near one another, making evidence about “cat” useful when predicting contexts involving “dog”. This is statistical similarity, not a guarantee of synonymy: opposite words can occur in similar contexts too. The 2003 neural probabilistic model learned these vectors jointly with next-word probabilities.^[ref]