= Information projection and rare sample means
Take <independent and identically distributed> draws from the uniform three-outcome reference above. An observed sample <mean> at least $1/2$ is unusual because the source <expected value> is zero. For two draws, the only successful ordered pairs are $(0,1),(1,0),(1,1)$, giving <probability> $3/9=1/3$.
For long samples, the relevant <empirical distributions> have <mean> at least $1/2$. The <method of types> assigns a profile $q$ a leading probability cost $e^{-nD(q\Vert p)}$. The least costly admissible profile is the <information projection on three outcomes>: in this example the minimizing <mean> is on the boundary $1/2$, and the leading exponent is approximately $0.197378$ <nats> per draw. The exponential scale is not an exact finite-sample probability formula.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others/Chapter 12: Information Projection and Large Deviation>[§12.1, Theorem 12.1, equation (12.2), printed p. 131; §12.2, Theorem 12.3, printed p. 134]{ref}
The attraction of this connection is that a constrained optimization problem predicts the most probable way an unusual aggregate observation can occur. It links <relative entropy>, <expected value> constraints and rare <empirical distributions> without identifying a rare event with just one exact attainable <type class>.
Back to article page
