= Differential entropy
{title2=$h(q)=-\int q(x)\log q(x)\,dx$}
{wiki}
<Differential entropy> describes the spread of a continuous <probability distribution> relative to the volume scale of its coordinates. For a real <random variable> $X$ with <probability density function> $f$, use <natural logarithms> and write
$$
\boxed{h_e(X)=-\int_{\mathbb R} f(x)\ln f(x)\,dx=-\mathbb E[\ln f(X)].}
$$
The convention is $0\ln0=0$. For a finite value, require $\int f|\ln f|<\infty$; if both positive and negative contributions are infinite, the expression is undefined. Unlike a <probability>, a <probability density> can exceed one. Consequently <differential entropy> can be negative. Base-two <logarithms> give $h_2=h_e/\ln2$, in <bits> instead of <nats>.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others/Chapter 1: Information Measures: Entropy and Divergence>[§1.3, Definition 1.5, equation (1.19) and warning, printed p. 18]{ref}
This quantity does not give the exact number of <bits> needed to store a real number. A recording device supplies a resolution; <equal-width bin entropy> connects that resolution to discrete <information entropy>. The useful continuous quantity is what remains after removing the increasing cost of specifying finer bins. Its dependence on coordinates is visible already in <differential entropy of a uniform law>.
Back to article page
