= Neural language model
A <neural language model> uses a <neural network> to assign a <probability distribution> to language sequences. An autoregressive model factorizes the probability of words $w_1,\ldots,w_T$ as
$$
P(w_1,\ldots,w_T)=\prod_{t=1}^T P(w_t\mid w_1,\ldots,w_{t-1}).
$$
Training by <maximum likelihood estimation> rewards the observed next words; generation repeatedly samples from the conditional distribution. If a two-word sequence has first-word probability $0.2$ and conditional second-word probability $0.3$, its joint probability is $0.06$.
<Yoshua Bengio> and coauthors' 2003 model jointly learned a <distributed representation> of words and a neural probability predictor. Sharing representation parameters allowed similar contexts to inform one another, rather than treating every observed word sequence as an unrelated event.<A Neural Probabilistic Language Model by Yoshua Bengio and others>[abstract and §2, model architecture]{ref}
Back to article page
