Sanov theorem

Let $X_1,\dots ,X_n$ be independent and identically distributed on a finite alphabet with full-support probability mass function $Q$, and let ${\hat{P}}_n$ be their empirical distribution. Sanov theorem gives a large deviation principle on the probability simplex, with rate functionusing natural logarithms. For any measurable set $A$, with interior $A^{\circ }$ and closure $\overline{A}$ relative to that probability simplex,The method of types explains the exponent: a type class has roughly $e^{n{H}_e(P)}$ words, each with probability $e^{-n(H_e(P)+D_e(P\Vert Q))}$; the number of types grows only polynomially.^[ref]

For Bernoulli distribution observations with success probability $1/5$, the event ${\overline{X}}_n\ge 1/2$ therefore has probability $\exp [-n\log (5/4)+o(n)]$. The exponent is $D_e(\mathrm{Bernoulli}(1/2)\Vert \mathrm{Bernoulli}(1/5))$; atypical empirical laws closest in relative entropy dominate the rare event.