 Neural language model

A neural language model uses a neural network to assign a probability distribution to language sequences. An autoregressive model factorizes the probability of words $w_1,\dots ,w_T$ asTraining by maximum likelihood estimation rewards the observed next words; generation repeatedly samples from the conditional distribution. If a two-word sequence has first-word probability $0.2$ and conditional second-word probability $0.3$, its joint probability is $0.06$.

Yoshua Bengio and coauthors' 2003 model jointly learned a distributed representation of words and a neural probability predictor. Sharing representation parameters allowed similar contexts to inform one another, rather than treating every observed word sequence as an unrelated event.^[ref]