Differential entropy ($h(q)=-\int q(x)\log q(x)dx$)

Differential entropy describes the spread of a continuous probability distribution relative to the volume scale of its coordinates. For a real random variable $X$ with probability density function $f$, use natural logarithms and writeThe convention is $0\ln 0=0$. For a finite value, require $\int f|\ln f|<\infty$; if both positive and negative contributions are infinite, the expression is undefined. Unlike a probability, a probability density can exceed one. Consequently differential entropy can be negative. Base-two logarithms give $h_2=h_e/\ln 2$, in bits instead of nats.^[ref]

This quantity does not give the exact number of bits needed to store a real number. A recording device supplies a resolution; equal-width bin entropy connects that resolution to discrete information entropy. The useful continuous quantity is what remains after removing the increasing cost of specifying finer bins. Its dependence on coordinates is visible already in differential entropy of a uniform law.

For a continuous uniform distribution on an interval of length $L>0$, the probability density is $1/L$. Direct integration givesThus a uniform law on an interval of length $1/2$ has differential entropy $-\ln 2$, while length two gives $\ln 2$. Both laws have total probability one. The shorter interval has larger probability density, making $-\ln f(X)$ negative; this is neither a negative probability nor a negative discrete information entropy.

For a normal distribution with mean $m$ and positive variance ${\sigma }^2$, expand the logarithm of its probability density function:The second term has expected value $1/2$, hence $h_e(X)=\frac{1}{2}\ln (2\pi e{\sigma }^2)$. Translation of the mean has no effect, while shrinking the standard deviation lowers differential entropy. A standard normal distribution has approximately $1.41894$ nats. This connects the continuous spread calculation with entropy power, whose normalization matches a Gaussian distribution's variance.

Let $X$ have a probability density function on an open interval, and let $g$ be continuously differentiable, strictly increasing there, with $g^{\prime }(x)>0$. The change of variables formula givesIf the differential entropy and the expected absolute logarithm of $g^{\prime }(X)$ are finite, taking the expectation of the negative logarithm proves the displayed formula, for a fixed base $b>1$. Local stretching reduces probability density; its contribution to differential entropy is the expected logarithmic stretch. The map is invertible, so this change does not mean that the transformation destroyed information about $X$.

Take $X$ with continuous uniform distribution on $(0,1)$ and set $Y=X^2$. This is an increasing invertible transformation on that interval. Its cumulative distribution function is $F_Y(y)=\sqrt{y}$, givingAlthough the probability density diverges near zero, it is integrable. The logarithmic stretch is integrable too:The inverse function $X=\sqrt{Y}$ recovers every input in the interval. The changed differential entropy records a changed coordinate volume, rather than a failure of recovery.

// Target: probability-and-statistics.bigb

For real $a\ne 0$ and $b$, the change of variables formula gives $f_{aX+b}(ax+b)=f_X(x)/|a|$. ThereforeThe absolute value also covers a decreasing transformation. If a numerical length changes from metres to centimetres, $a=100$ and the differential entropy changes by $\ln 100\approx 4.60517$ nats, although the physical uncertainty is the same. The reference coordinate scale must therefore accompany an absolute differential entropy value.

Partition $(0,1)$ into $n$ bins of width $\delta =1/n$, and let the discrete random variable $Q_n$ record the bin containing $X$. This is a quantization of $X$. Its probability mass function isand its discrete information entropy is $H_e(Q_n)=-{\sum }_j{p}_j\ln p_j$. Let ${\bar{f}}_j=p_j/\delta$ be the average probability density within bin $j$. Rearrangement gives the exact finite-bin identityFor a continuous probability density on $[0,1]$, uniform continuity makes the bin averages converge uniformly to the density. Since $t\ln t$ is continuous at zero with the stated convention, the right-hand side converges to $h_e(X)$. Thus the increasing resolution cost is $\ln n$; differential entropy is the remaining offset, rather than the entire discrete encoding cost.

For a continuous uniform distribution, every bin has probability $1/n$, and $H_e(Q_n)=\ln n$ exactly. For the continuous uniform distribution's squared transformation above, the exact bin probabilities areThe diagram computes these masses directly, including the bin next to the integrable singularity. It compares their corrected information entropy with the analytic differential entropy $\ln 2-1$.

Apply the same invertible differentiable transformation to two probability distributions with densities $f$ and $r$. Both transformed densities acquire the same Jacobian determinant factor. Write their transformed densities as $f_g$ and $r_g$. Their ratio is unchanged, and the integration measure transforms with the numerator density. Consequently their relative entropy is unchanged:The densities describe two alternative laws in the same coordinate, with a nonzero Jacobian determinant on their common domain, and the expression is assumed well defined. This explains why comparing two laws can survive a change of measurement units even though either law's differential entropy changes. Mutual information, formulated as relative entropy from a product of marginals, has the same invariance under invertible changes to either observed coordinate.

Differential entropy is also the entropy term in the evidence lower bound for continuous variational inference. Its coordinate dependence must be combined consistently with that of the model density; changing units does not justify changing only one term of the objective.

In A Mathematical Theory of Communication by Claude Shannon (1948), section 20 extends the discrete entropy formula to continuous densities. Properties 8–9 explicitly discuss coordinate dependence, possible negative values and linear rescaling. Claude Shannon relates this coordinate convention to the engineering use of entropy differences, rather than treating an isolated continuous entropy as an invariant count of messages.^[ref]

The entropy power of an $n$-dimensional random vector with finite differential entropy in bits is the variance per coordinate of an isotropic Gaussian distribution with the same differential entropy. The normalization makes $N(X)={\sigma }^2$ when $X$ has covariance matrix ${\sigma }^2{I}_n$.

// Target: probability-and-statistics.bigb

For independent $n$-dimensional random vectors with densities and finite differential entropies, the entropy power inequality statesThe sum's differential entropy is required to be defined; a value $+\infty$ makes the inequality immediate. Equality for finite nondegenerate densities occurs for Gaussian distributions with proportional covariance matrices.

// Target: probability-and-statistics.bigb

For independent positive random variables $Y_1,Y_2$ with densities, finite differential entropies and finite ${\mu }_i=\mathbb{E}{\log }_2{Y}_i$, applying the entropy power inequality to $Z_i=\ln Y_i$ givesIndeed $h(Z_i)=h(Y_i)-{\mu }_i$, while $h(Y_1{Y}_2)=h(Z_1+Z_2)+{\mu }_1+{\mu }_2$, provided the sum's differential entropy is defined. Positivity alone does not ensure these expressions exist: $Y=e^T$ with $T$ having Cauchy distribution has undefined $\mathbb{E}{\log }_{2}Y$.

// Target: algebra.bigb