 Differential entropy

Differential entropy describes the spread of a continuous probability distribution relative to the volume scale of its coordinates. For a real random variable $X$ with probability density function $f$, use natural logarithms and writeThe convention is $0\ln 0=0$. For a finite value, require $\int f|\ln f|<\infty$; if both positive and negative contributions are infinite, the expression is undefined. Unlike a probability, a probability density can exceed one. Consequently differential entropy can be negative. Base-two logarithms give $h_2=h_e/\ln 2$, in bits instead of nats.^[ref]

This quantity does not give the exact number of bits needed to store a real number. A recording device supplies a resolution; equal-width bin entropy connects that resolution to discrete information entropy. The useful continuous quantity is what remains after removing the increasing cost of specifying finer bins. Its dependence on coordinates is visible already in differential entropy of a uniform law.

Differential entropy is a concept in information theory that extends the idea of traditional (or discrete) entropy to continuous probability distributions. While discrete entropy measures the uncertainty associated with a discrete random variable, differential entropy quantifies the uncertainty of a continuous random variable.