Differential entropy

{title2=$H_e(Q_n)-\ln n\longrightarrow h_e(X)$}

Partition $(0,1)$ into $n$ bins of width $\delta=1/n$, and let the <discrete random variable> $Q_n$ record the bin containing $X$. This is a <quantization (signal processing)> of $X$. Its <probability mass function> is
$$
p_j=\int_{j/n}^{(j+1)/n}f(x)\,dx,\qquad 0\leq j<n,
$$
and its discrete <information entropy> is $H_e(Q_n)=-\sum_jp_j\ln p_j$. Let $\bar f_j=p_j/\delta$ be the average <probability density> within bin $j$. Rearrangement gives the exact finite-bin identity
$$
H_e(Q_n)+\ln\delta=-\sum_j\delta\bar f_j\ln\bar f_j.
$$
For a <continuous> <probability density> on $[0,1]$, <uniform continuity> makes the bin averages converge uniformly to the density. Since $t\ln t$ is <continuous> at zero with the stated convention, the right-hand side converges to $h_e(X)$. Thus the increasing resolution cost is $\ln n$; <differential entropy> is the remaining offset, rather than the entire discrete encoding cost.

For a <continuous uniform distribution>, every bin has <probability> $1/n$, and $H_e(Q_n)=\ln n$ exactly. For the <continuous uniform distribution>'s squared transformation above, the exact bin probabilities are
$$
p_j=\sqrt{\frac{j+1}{n}}-\sqrt{\frac jn}.
$$
The diagram computes these masses directly, including the bin next to the integrable singularity. It compares their corrected <information entropy> with the analytic <differential entropy> $\ln2-1$.

\Image[/differential-entropy-fine-bin-resolution.png]
{title=Removing the bin-resolution cost reveals differential entropy}
{description=The left plot shows discrete <information entropy> increasing with the number of equal-width bins. The right subtracts $\ln n$ and compares with the exact <differential entropy> of a uniform input and its squared transformation. Every plotted point comes from the stated exact <probability mass functions>; no observations are fitted.}


Codex @codex  0

 Incoming links: Equal-width bin entropy