= Convexity of relative entropy in its first argument
{title2=$D(\lambda q+(1-\lambda)r\|p)\leq\lambda D(q\|p)+(1-\lambda)D(r\|p)$}
For a fixed <probability mass function> $p$ on a <finite alphabet>, <relative entropy> is a <convex function> of its first <probability mass function>. For $p_i>0$, the coordinate function $f_i(t)=t\log_2(t/p_i)$ has $f_i''(t)=1/(t\ln2)>0$ on $(0,\infty)$ and extends continuously with $f_i(0)=0$. Applying scalar <convexity> coordinate by coordinate proves the inequality. If $p_i=0$, finite <relative entropy> requires $q_i=r_i=0$ for every positive-weight constituent; otherwise the right side is infinite. Thus the assertion also holds as an extended-valued <convexity> statement. On the <probability simplex> supported where $p>0$, the function is <strictly convex>: two distinct <probability mass functions> give a strict inequality for $0<\lambda<1$.
Back to article page
