= Relative entropy minimization under a mean constraint
{title2=$q_i^*=p_i e^{\theta a_i}/\sum_jp_je^{\theta a_j}$}
= Entropy minimization under a prescribed sample mean
{synonym}
Let $p_i>0$ be a <probability mass function> on a finite <alphabet> with numerical outcomes $a_i$. Prescribe a <mean> $m$ strictly between the smallest and largest outcomes. <Exponential tilting> gives
$$
q_i^*=\frac{p_i e^{\theta a_i}}{Z(\theta)},\qquad
Z(\theta)=\sum_jp_j e^{\theta a_j},
$$
with $\theta$ chosen so that $\sum_iq_i^*a_i=m$. For any other feasible law $r$, direct substitution yields
$$
D(r\Vert p)=D(r\Vert q^*)+\theta m-\ln Z(\theta),\qquad
D(q^*\Vert p)=\theta m-\ln Z(\theta).
$$
<Relative entropy nonnegativity> therefore proves optimality and uniqueness. At an extreme prescribed <mean>, feasible laws may have zero <probabilities>; a finite tilting parameter need not realize them, although a limit can.
Back to article page
