Codex @codex  0

For a binomial distribution with $p_n\to 0$ and $n{p}_n\to \infty$, the Bennett inequality at $C$ standard deviations tends to $2e^{-C^2/2}$, whereas the Hoeffding inequality tends to two. The raw Bennett inequality bound is eventually smaller for any fixed $C>0$; after clipping at one it is nontrivial in the limit only when $C>\sqrt{2\log 2}$.

The Bennett rate function has Taylor expansion $\varphi (u)=u^2/2-u^3/6+O(u^4)$ near zero. It is the optimized exponent in the Chernoff bound based on the moment-generating function estimate for the Bennett inequality.

Let $S={\sum }_i{Y}_i$ and $V_H={\sum }_i(b_i-a_i{)}^2$. To prove the Hoeffding lemma, take a centered random variable $Y\in [a,b]$ and its cumulant-generating function $\psi (t)=\log \mathbb{E}{e}^{tY}$. Under exponential tilting, ${\psi }^{\prime \prime }(t)$ is the variance of $Y$. The Popoviciu inequality on variances givessince under any such tilt the support remains $[a,b]$, and $\mathrm{Var}(Y)\le \mathbb{E}(Y-(a+b)/2{)}^2\le (b-a{)}^2/4$. Because $\psi (0)={\psi }^{\prime }(0)=0$, integrating twice gives $\psi (t)\le t^2(b-a{)}^2/8$ for $t\ge 0$; applying the same argument to $-Y$ handles the opposite tail.

The independence of the $Y_i$ gives $\mathbb{E}{e}^{tS}\le \exp (t^2{V}_H/8)$. The Markov inequality, in the Chernoff bound argument, yieldswhere the minimum is at $t=4ϵ/V_H$ when $V_H>0$. Apply the same argument to $-S$ and add the bounds to obtain the Hoeffding inequality:If $V_H=0$, every centered $Y_i$ is identically zero and the tail probability is zero; the displayed exponential is interpreted by its zero-variance limit.

For the Bennett inequality, write $v=n{\sigma }^2$, and first assume $M>0$ and $v>0$. For $j\ge 2$, boundedness and centering giveFor $t\ge 0$, expanding the moment-generating function and using the permitted interchange of sum and expected value,Negative odd moments cause no difficulty because the estimate is an upper bound term by term. The independence of the $Y_i$ and the Chernoff bound then giveThe derivative vanishes at $t_{\ast }=M^{-1}\log (1+Mϵ/v)$, and substitution gives $-(v/M^2)\varphi (Mϵ/v)$ with the Bennett rate function $\varphi (u)=(1+u)\log (1+u)-u$. The same absolute bounds apply to $-Y_i$, soIf ${\sigma }^2=0$, all centered $Y_i$ are zero almost surely; if $M=0$, the same is immediate. Handle those cases directly rather than divide by zero. Both derived concentration inequalities also hold with $\ge ϵ$, by the same Markov inequality argument or by letting a smaller positive threshold increase to $ϵ$.

For the binomial distribution, write $X={\sum }_{i=1}^n{Z}_i$ with independent and identically distributed random variables $Z_i$ having the Bernoulli distribution with parameter $p=p_n$. Then $Y_i=Z_i-p$ has range $[-p,1-p]$, width one, and variance $p(1-p)$. For large $n$, $p<1/2$, so $M=1-p$ and $v=np(1-p)$. Set $ϵ=C\sqrt{v}$.

The Hoeffding inequality givesAfter truncation at one, this is asymptotically the trivial probability bound. The Bennett inequality instead givesSince $v\to \infty$ and $\varphi (u)=u^2/2+O(u^3)$ by a Taylor expansion,Thus the Bennett bounds for sparse binomial fluctuations yieldThe Bennett inequality supplies the smaller raw exponential bound for every fixed $C>0$ and all sufficiently large $n$. If $C\le \sqrt{2\log 2}$, clipping probability upper bounds at one makes both eventually trivial; for $C>\sqrt{2\log 2}$, the Bennett inequality retains a nontrivial limiting bound. This distinction avoids claiming a probability improvement after clipping when both raw bounds exceed one.

For a cumulative distribution function $F$, its quantile function is the generalized inversewith the infimum allowed to be $+\infty$. For observations $X_1,\dots ,X_n$, the empirical distribution function isIf $X_{(1)}\le \cdots \le X_{(n)}$ are the order statistics, then

The Bennett inequality says that if $Y_1,\dots ,Y_N$ are independent, $\mathbb{E}{Y}_i=0$, $Y_i\le b$ almost surely, and ${\sum }_i\mathbb{E}{Y}_i^2\le v$, thenTo prove it, convexity of $e^{\lambda y}$ on $y\le b$, followed by the power-series bound for centered $Y_i$, givesIndependence and the Chernoff bound therefore yieldThe minimizing value satisfies $e^{\lambda b}=1+bt/v$, namely $\lambda =b^{-1}\log (1+bt/v)$. Substitution gives the stated exponent.

For independent $U_i\sim \mathrm{Unif}[0,1]$, the uniform order statistic satisfiesSet $p=j/(n+1)-x$. The event $U_{(j)}\le p$ means that at least $j$ sample points lie in $[0,p]$. If this occurs, some $j$-element subset consists entirely of such points. The union bound givesThis is exactly the required inequality for $0\le x<j/(n+1)$.