Relative entropy minimization under a mean constraint ($q_i^{\ast }=p_i{e}^{\theta a_i}/{\sum }_j{p}_j{e}^{\theta a_j}$)

Let $p_i>0$ be a probability mass function on a finite alphabet with numerical outcomes $a_i$. Prescribe a mean $m$ strictly between the smallest and largest outcomes. Exponential tilting giveswith $\theta$ chosen so that ${\sum }_i{q}_i^{\ast }{a}_i=m$. For any other feasible law $r$, direct substitution yieldsRelative entropy nonnegativity therefore proves optimality and uniqueness. At an extreme prescribed mean, feasible laws may have zero probabilities; a finite tilting parameter need not realize them, although a limit can.

Take independent and identically distributed draws from the uniform three-outcome reference above. An observed sample mean at least $1/2$ is unusual because the source expected value is zero. For two draws, the only successful ordered pairs are $(0,1),(1,0),(1,1)$, giving probability $3/9=1/3$.

For long samples, the relevant empirical distributions have mean at least $1/2$. The method of types assigns a profile $q$ a leading probability cost $e^{-nD(q\Vert p)}$. The least costly admissible profile is the information projection on three outcomes: in this example the minimizing mean is on the boundary $1/2$, and the leading exponent is approximately $0.197378$ nats per draw. The exponential scale is not an exact finite-sample probability formula.^[ref]

The attraction of this connection is that a constrained optimization problem predicts the most probable way an unusual aggregate observation can occur. It links relative entropy, expected value constraints and rare empirical distributions without identifying a rare event with just one exact attainable type class.