A neural language model uses a neural network to assign a probability distribution to language sequences. An autoregressive model factorizes the probability of words asTraining by maximum likelihood estimation rewards the observed next words; generation repeatedly samples from the conditional distribution. If a two-word sequence has first-word probability and conditional second-word probability , its joint probability is .
Yoshua Bengio and coauthors' 2003 model jointly learned a distributed representation of words and a neural probability predictor. Sharing representation parameters allowed similar contexts to inform one another, rather than treating every observed word sequence as an unrelated event.[ref]
A Neural Probabilistic Language Model by Yoshua Bengio and others. abstract and §2, model architecture
Articles by others on the same topic
There are currently no matching articles.