Entropy functional ($\mathrm{Ent}(Z)$)

For a nonnegative integrable random variable $Z$ on a probability space with probability measure $P$, its entropy functional isusing $0\log 0=0$ and allowing the value $+\infty$. If $m>0$, define the probability measure $Q$ by $dQ/dP=Z/m$. Substitution into the definition of relative entropy givesThus the normalization factor $m$ is essential. Jensen inequality shows that the entropy functional is nonnegative; it vanishes precisely when $Z$ is constant almost surely. If $m=0$, nonnegativity forces $Z=0$ almost surely and the entropy functional is zero.

For an event $A$ of probability $p>0$, take $Z=1_A$. Then $Q=P(\cdot \mid A)$ and ${\mathrm{Ent}}_P(Z)=p\log (1/p)$. This example connects the entropy functional to the information cost of conditioning on a rare event.

A probability measure $\mu$ with a Dirichlet form ${\mathcal{E}}_{\mu }$ satisfies a logarithmic Sobolev inequality with constant $c_{\mathrm{LS}}>0$ iffor every $g$ in the domain of the Dirichlet form. The entropy functional measures how far $g^2$ is from a constant, while the Dirichlet form measures its spatial variation. For a symmetric Markov semigroup with generator $L$, the convention is ${\mathcal{E}}_{\mu }(g,g)=-\int gLgd\mu$ when this expression is defined.

For standard Gaussian measure, $L=\Delta -x\cdot \nabla$ and Gaussian integration by parts gives ${\mathcal{E}}_{\mu }(g,g)=\int \Vert \nabla g{\Vert }^{2}d\mu$. The Gaussian logarithmic Sobolev inequality has optimal constant two in this convention. Rescaling the Dirichlet form rescales the constant inversely, so a numerical constant needs its energy convention. The logarithmic Sobolev inequality implies hypercontractivity and also supports concentration inequalities through the Herbst argument.^[ref]

// Target: analysis.bigb

Differentiate $\log \Vert u{\Vert }_q$ with $u=P_{t}f$ and a changing exponent. The derivative is $q^{\prime }\mathrm{Ent}(u^q)/(q^2\int u^q)-\mathcal{E}(u^{q-1},u)/\int u^q$. Apply the logarithmic Sobolev inequality to $u^{q/2}$ and the power inequality for symmetric Markov energies. Choosing $q^{\prime }=4(q-1)/c_{\mathrm{LS}}$ makes the upper bound zero.

// Target: probability-and-statistics.bigb