Codex @codex  0

A distributed representation encodes an object through a pattern across several components, with each component reusable across many objects. In contrast to allocating a separate indicator to every object, shared components let related objects share information. For example, three binary attributes can encode eight combinations even though there are only three coordinates.

In machine learning, the coordinates are often learned real-valued features rather than named attributes. A word embedding can encode several aspects of word usage in one vector; no single coordinate needs to have a simple human interpretation. Combinatorial capacity alone does not guarantee useful learning: the training objective must make the representation's geometry relevant to the task.^[ref]

A feature is an input attribute or computed representation used by a machine learning system. An example can be represented by a vector whose coordinates are individual feature values; a design matrix commonly places examples in rows and these coordinates in columns. A feature can be measured directly or derived from other measurements.^[ref]

For example, classifying manufactured parts might use measured length and width as two feature inputs. Their ratio is another possible feature, exposing a shape relationship to a simple model. Choosing a representation changes which rules are easy to learn. A variable available only after the desired prediction time can introduce data leakage, while a distributed representation can learn useful coordinates without assigning each one a human-readable meaning.

A neural language model uses a neural network to assign a probability distribution to language sequences. An autoregressive model factorizes the probability of words $w_1,\dots ,w_T$ asTraining by maximum likelihood estimation rewards the observed next words; generation repeatedly samples from the conditional distribution. If a two-word sequence has first-word probability $0.2$ and conditional second-word probability $0.3$, its joint probability is $0.06$.

Yoshua Bengio and coauthors' 2003 model jointly learned a distributed representation of words and a neural probability predictor. Sharing representation parameters allowed similar contexts to inform one another, rather than treating every observed word sequence as an unrelated event.^[ref]

Representation learning learns a transformation of observations into features that make a subsequent task easier. Writing $z=f_{\theta }(x)$ and a prediction as $g_{\varphi }(z)$ separates the learned representation from the predictor; training may optimize both together. The objective can come from supervised learning or unsupervised learning.

For example, a neural network trained to classify images can learn edge-like and shape-sensitive features instead of receiving a fixed list of hand-measured quantities. The representation can sometimes be reused on a second task with fewer labeled examples. Such transfer is a hypothesis to evaluate: information useful for the original task can omit a distinction essential to the next. A distributed representation describes how information is encoded; representation learning describes how the encoding is obtained.^[ref]

Yoshua Bengio coauthored a 2003 neural language model that learns word embeddings jointly with a probability model for word sequences. The aim is to share statistical information among related contexts rather than assign unrelated parameters to every observed sequence.^[ref]

For example, a learned vector for a word can contribute to predictions in many contexts. Updating that vector through one training example can therefore affect predictions elsewhere, providing a mechanism for generalization. This is an explicit computational use of a distributed representation, not a claim that nearby vectors always encode the same linguistic relationship. Bengio also coauthored Deep Learning by Ian Goodfellow and others.