= Solution
For <probability mass functions> on the <finite alphabet>, take <logarithms> to base two and define the <relative entropy> by
$$
\boxed{D(q\|p)=\sum_{i=1}^mq_i\log_2\frac{q_i}{p_i}.}
$$
A term with $q_i=0$ is zero, including $p_i=0$; a term with $q_i>0=p_i$ is $+\infty$. Fix $p_i>0$ and write $f_i(t)=t\log_2(t/p_i)$. For $t>0$,
$$
f_i''(t)=\frac1{t\ln2}>0,\qquad \lim_{t\downarrow0}f_i(t)=0.
$$
Thus $f_i$ is a <convex function> on $[0,\infty)$ by continuous extension to zero. For $0\leq\lambda\leq1$, applying this scalar <convexity> to each coordinate and summing proves
$$
D(\lambda q+(1-\lambda)r\|p)\leq\lambda D(q\|p)+(1-\lambda)D(r\|p).
$$
If $p$ has zero coordinates, the finite-valued domain consists of <probability mass functions> supported on $\{i:p_i>0\}$; it is convex and the same proof applies there. Outside that domain the right side is infinite whenever an offending <probability mass function> has a positive mixture weight, so the extended-valued <convexity> assertion still holds. At mixture weights zero or one the inequality is an identity, without needing the expression $0\cdot\infty$.
Back to article page
