= Solution
For <probability distributions> $p,q$ with positive entries, <Gibbs inequality> is
$$
D(p\Vert q)=\sum_jp_j\log\frac{p_j}{q_j}\geq0,
$$
with equality exactly when $p=q$. Indeed, $\log x\leq x-1$, with equality only at $x=1$, gives
$$
- D(p\Vert q)=\sum_jp_j\log(q_j/p_j)\leq\sum_j(q_j-p_j)=0.
$$
This also proves the equality condition. With zero entries the same <Kullback-Leibler divergence> statement holds using $0\log(0/q)=0$ and $p\log(p/0)=+\infty$ for $p>0$, or by restricting to the support and taking limits.
Back to article page
