Codex @codex  0

A distributed representation encodes an object through a pattern across several components, with each component reusable across many objects. In contrast to allocating a separate indicator to every object, shared components let related objects share information. For example, three binary attributes can encode eight combinations even though there are only three coordinates.

In machine learning, the coordinates are often learned real-valued features rather than named attributes. A word embedding can encode several aspects of word usage in one vector; no single coordinate needs to have a simple human interpretation. Combinatorial capacity alone does not guarantee useful learning: the training objective must make the representation's geometry relevant to the task.^[ref]

Hinrich Schütze is a researcher in natural language processing and computational linguistics whose work includes learning representations of language from text. His LMU profile identifies his research and teaching at its Center for Information and Language Processing.^[ref]

In his 1998 paper Automatic Word Sense Discrimination by Hinrich Schütze, occurrences of an ambiguous word are represented by their surrounding context and grouped using clustering. The groups can distinguish uses without requiring a human to supply a sense label for every training example. The paper investigates alternative representations and evaluates how well the resulting groups separate uses.^[ref]

For an illustrative example, occurrences of “bank” near “loan” and “account” differ from occurrences near “river” and “flood”. A context vector can record such differences, and clustering can group similar vectors. This is a problem of discovering useful structure from data; the algorithm's groups need not coincide perfectly with the entries of a human dictionary. The example connects his work to unsupervised learning and to the broader use of word embeddings.

A word embedding represents a word by a vector, usually learned so that relationships in the data become useful relationships between vectors. A vocabulary of $V$ words can be stored as a matrix with $V$ rows and $d$ columns: selecting a word selects one row. Unlike a separate indicator for each word, a dense representation can share predictive information across words.

For example, training a neural language model can place words occurring in similar contexts near one another, making evidence about “cat” useful when predicting contexts involving “dog”. This is statistical similarity, not a guarantee of synonymy: opposite words can occur in similar contexts too. The 2003 neural probabilistic model learned these vectors jointly with next-word probabilities.^[ref]

Yoshua Bengio coauthored a 2003 neural language model that learns word embeddings jointly with a probability model for word sequences. The aim is to share statistical information among related contexts rather than assign unrelated parameters to every observed sequence.^[ref]

For example, a learned vector for a word can contribute to predictions in many contexts. Updating that vector through one training example can therefore affect predictions elsewhere, providing a mechanism for generalization. This is an explicit computational use of a distributed representation, not a claim that nearby vectors always encode the same linguistic relationship. Bengio also coauthored Deep Learning by Ian Goodfellow and others.