BLEU (source code)
= BLEU
{c}
{title2=$\mathrm{BLEU}=\mathrm{BP}\exp(\sum_n w_n\log p_n)$}
<BLEU> is a reference-based metric for evaluating <machine translation>. It combines <modified n-gram precision> at several lengths using a <geometric mean>, then applies a <brevity penalty> to discourage very short outputs. <Kishore Papineni> and colleagues introduced it as a way to make repeated translation evaluation less costly than repeatedly collecting human judgments.<BLEU: a Method for Automatic Evaluation of Machine Translation by Kishore Papineni and others>[Abstract and §1.1; §§2.1–2.3, score construction]{ref}
For each candidate <n-gram>, count matches only up to its maximum occurrence count in a single reference. Sum these clipped matches across the corpus and divide by the total number of candidate <n-grams> of that length to obtain $p_n$. This clipping prevents repeated use of a common word from earning unlimited matches. Corpus aggregation uses counts, rather than simply averaging independently calculated sentence scores.<BLEU: a Method for Automatic Evaluation of Machine Translation by Kishore Papineni and others>[§2.1, footnote 2; §2.1.1, corpus modified precision]{ref}
With candidate length $c$, effective reference length $r$ and weights summing to one,
$$
\mathrm{BP}=\begin{cases}1&c>r,\\e^{1-r/c}&c\leq r,\end{cases}
\qquad
\mathrm{BLEU}=\mathrm{BP}\exp\!\left(\sum_{n=1}^N w_n\log p_n\right).
$$
The original baseline uses $N=4$ and $w_n=1/4$. Effective reference length sums the closest reference sentence lengths for the candidates.<BLEU: a Method for Automatic Evaluation of Machine Translation by Kishore Papineni and others>[§§2.2.2–2.3, brevity penalty, effective length and baseline weights]{ref}
For a synthetic one-pair calculation, take reference `the cat is on the mat` and candidate `the cat is on mat`. The clipped precisions are
$$
p_1=1,\quad p_2=\frac34,\quad p_3=\frac23,\quad p_4=\frac12.
$$
Here $c=5$, $r=6$ and the precision product is $1/4$, giving
$$
\mathrm{BLEU}=e^{-0.2}(1/4)^{1/4}
=\frac{e^{-0.2}}{\sqrt2}\simeq0.57893.
$$
This is about $57.893$ on a scale multiplied by one hundred; it is not $57.893\%$ translation accuracy.
The unsmoothed score becomes zero when a required precision is zero. Short sentences make this particularly conspicuous, while legitimate rephrasing can reduce overlap. State tokenization, casing, reference set, smoothing and scale when comparing results. The example illustrates the arithmetic; the paper emphasizes corpus-level use rather than treating each sentence score as an independent human-quality judgment.<BLEU: a Method for Automatic Evaluation of Machine Translation by Kishore Papineni and others>[§2.1.1, footnote 4; §2.1.3, footnote 5]{ref}
Back to article page
