= Perplexity
{title2=$\operatorname{PPL}=\exp[-N^{-1}\sum_i\ln q_i]$}
<Perplexity> measures how much <probability> a model assigns to observed outcomes, expressed as the inverse <geometric mean> of those probabilities. For a sequence evaluated by a <neural language model>, let $q_i$ be the model's <conditional probability> of the actually observed token given its allowed context. Then
$$
\operatorname{PPL}
=\left(\prod_{i=1}^Nq_i\right)^{-1/N}
=\exp\!\left(-\frac1N\sum_{i=1}^N\ln q_i\right).
$$
It is therefore exponentiated mean negative <log-likelihood>. If <cross-entropy> is expressed in <bits> rather than natural-log units, the equivalent expression is $\operatorname{PPL}=2^{H_{\rm bits}}$.<A Neural Probabilistic Language Model by Yoshua Bengio and others>[§2, printed p.1141, opening definition of perplexity]{ref}
A uniform predictor over $V$ possible tokens assigns each observed token probability $1/V$, giving perplexity $V$. For an unequal two-token example with assigned probabilities $1/2$ and $1/4$,
$$
\operatorname{PPL}=\bigl((1/2)(1/4)\bigr)^{-1/2}
=\sqrt8\simeq2.828.
$$
The result need not be an integer or an actual vocabulary size. It is an effective branching measure defined by the likelihood calculation.
On a fixed evaluation representation and dataset, lower <perplexity> means higher <geometric mean> probability for the realized outcomes. Assigning zero probability to any realized outcome makes the unsmoothed sequence perplexity infinite. Correctness of the most probable token alone does not determine the metric: the probabilities assigned to all realized outcomes matter.
For an empirical example, the 2003 neural-language-model paper reports test perplexity $268$ for its MLP1 row, compared with $321$ for the five-gram back-off comparison row. These are measurements under the paper's Brown-corpus setup and token-counting conventions.<A Neural Probabilistic Language Model by Yoshua Bengio and others>[§4.1, printed p.1148, token conventions; Table 1, p.1149, test-perplexity column]{ref}
Changing tokenization, vocabulary, punctuation or sentence-boundary treatment changes the outcomes and the divisor $N$. Per-word and per-subword scores therefore cannot be compared as if they measured the same event space. <Perplexity> is also distinct from <BLEU>, which compares generated text with references rather than directly averaging the probability of observed tokens.
Back to article page
