Han's inequality for relative entropy

If $P={⨂}_{i=1}^n{P}_i$ is a product measure, $Q$ is any probability measure, and $X^{(i)}$ omits coordinate $i$, then

For a random vector $X_{1:n}$,This is equivalent to Han's inequality for relative entropy after expansion against a product reference measure.

For a discrete random vector with $n$ coordinates, this is the average information entropy per coordinate over all subsets of size $k$. Independent coordinates give the same average for every $k$, even when their individual entropies differ. If all coordinates are copies of one random variable $Z$, the average is $h(Z)/k$.

For discrete random variables, normalized subset entropy decreases with subset size. For finite entropies, the chain rule for information entropy and conditioning reduces entropy give ${\sum }_{i\in S}h(X_{S\setminus \{i\}})\ge (|S|-1)h(X_S)$. Sum this over all $k$-subsets $S$. Every $(k-1)$-subset occurs $n-k+1$ times, and $k\left(\genfrac{}{}{0ex}{}{n}{k}\right)=(n-k+1)\left(\genfrac{}{}{0ex}{}{n}{k-1}\right)$ cancels the normalization factors. This proves $h_k^{(n)}\le h_{k-1}^{(n)}$. If a marginal entropy is infinite, every one of these subset averages is infinite; otherwise all entropies used in the proof are finite.

The strong formfollows by iterating entropy submodularity.