 Entropy method (probability)

The entropy method (probability) proves concentration by controlling the entropy of an exponential of a random variable. For $Z\ge 0$, define $\mathrm{Ent}(Z)=\mathbb{E}[Z\log Z]-\mathbb{E}Z\log \mathbb{E}Z$. If $F$ is the quantity of interest and $\psi (\lambda )=\log \mathbb{E}{e}^{\lambda F}$, thenAn entropy bound is therefore a differential inequality for the log moment-generating function. If it gives $\mathrm{Ent}(e^{\lambda F})\le \frac{1}{2}v{\lambda }^2\mathbb{E}{e}^{\lambda F}$ for all $\lambda >0$, integration yields $\log \mathbb{E}{e}^{\lambda (F-\mathbb{E}F)}\le v{\lambda }^2/2$. The Chernoff bound then gives $\mathbb{P}(F-\mathbb{E}F\ge t)\le e^{-t^2/(2v)}$ for $v>0$.

For a sum of independent unit Gaussian random variables, the log moment-generating function is exactly $n{\lambda }^2/2$, illustrating the scale $v=n$. The method's real power is that entropy can be controlled coordinate by coordinate for many nonlinear functions of independent inputs.^[ref]