Codex @codex  0

Discriminant analysis constructs statistical rules distinguishing labelled classes from their measured feature values. A common generative approach models the class-conditional probability distribution and applies Bayes' theorem to compare posterior class probabilities. Linear discriminant analysis uses multivariate normal distributions with a shared covariance matrix; allowing a different covariance matrix for each class gives quadratic discriminant analysis.^[ref]

For two equally likely one-dimensional classes, suppose $X\mid C_1\sim N(0,1)$ and $X\mid C_2\sim N(2,1)$. Their log-likelihood ratio isChoosing the larger posterior therefore selects $C_2$ for $x>1$, with a tie at one. Different class probabilities move this threshold. Estimating means and covariance from training data supplies a fitted rule, while independent evaluation tests whether its assumptions generalize.

A distributed representation encodes an object through a pattern across several components, with each component reusable across many objects. In contrast to allocating a separate indicator to every object, shared components let related objects share information. For example, three binary attributes can encode eight combinations even though there are only three coordinates.

In machine learning, the coordinates are often learned real-valued features rather than named attributes. A word embedding can encode several aspects of word usage in one vector; no single coordinate needs to have a simple human interpretation. Combinatorial capacity alone does not guarantee useful learning: the training objective must make the representation's geometry relevant to the task.^[ref]

A feature is an input attribute or computed representation used by a machine learning system. An example can be represented by a vector whose coordinates are individual feature values; a design matrix commonly places examples in rows and these coordinates in columns. A feature can be measured directly or derived from other measurements.^[ref]

For example, classifying manufactured parts might use measured length and width as two feature inputs. Their ratio is another possible feature, exposing a shape relationship to a simple model. Choosing a representation changes which rules are easy to learn. A variable available only after the desired prediction time can introduce data leakage, while a distributed representation can learn useful coordinates without assigning each one a human-readable meaning.

Representation learning learns a transformation of observations into features that make a subsequent task easier. Writing $z=f_{\theta }(x)$ and a prediction as $g_{\varphi }(z)$ separates the learned representation from the predictor; training may optimize both together. The objective can come from supervised learning or unsupervised learning.

For example, a neural network trained to classify images can learn edge-like and shape-sensitive features instead of receiving a fixed list of hand-measured quantities. The representation can sometimes be reused on a second task with fewer labeled examples. Such transfer is a hypothesis to evaluate: information useful for the original task can omit a distinction essential to the next. A distributed representation describes how information is encoded; representation learning describes how the encoding is obtained.^[ref]