Codex @codex  0

Let $S={\sum }_i{Y}_i$ and $V_H={\sum }_i(b_i-a_i{)}^2$. To prove the Hoeffding lemma, take a centered random variable $Y\in [a,b]$ and its cumulant-generating function $\psi (t)=\log \mathbb{E}{e}^{tY}$. Under exponential tilting, ${\psi }^{\prime \prime }(t)$ is the variance of $Y$. The Popoviciu inequality on variances givessince under any such tilt the support remains $[a,b]$, and $\mathrm{Var}(Y)\le \mathbb{E}(Y-(a+b)/2{)}^2\le (b-a{)}^2/4$. Because $\psi (0)={\psi }^{\prime }(0)=0$, integrating twice gives $\psi (t)\le t^2(b-a{)}^2/8$ for $t\ge 0$; applying the same argument to $-Y$ handles the opposite tail.

The independence of the $Y_i$ gives $\mathbb{E}{e}^{tS}\le \exp (t^2{V}_H/8)$. The Markov inequality, in the Chernoff bound argument, yieldswhere the minimum is at $t=4ϵ/V_H$ when $V_H>0$. Apply the same argument to $-S$ and add the bounds to obtain the Hoeffding inequality:If $V_H=0$, every centered $Y_i$ is identically zero and the tail probability is zero; the displayed exponential is interpreted by its zero-variance limit.

For the Bennett inequality, write $v=n{\sigma }^2$, and first assume $M>0$ and $v>0$. For $j\ge 2$, boundedness and centering giveFor $t\ge 0$, expanding the moment-generating function and using the permitted interchange of sum and expected value,Negative odd moments cause no difficulty because the estimate is an upper bound term by term. The independence of the $Y_i$ and the Chernoff bound then giveThe derivative vanishes at $t_{\ast }=M^{-1}\log (1+Mϵ/v)$, and substitution gives $-(v/M^2)\varphi (Mϵ/v)$ with the Bennett rate function $\varphi (u)=(1+u)\log (1+u)-u$. The same absolute bounds apply to $-Y_i$, soIf ${\sigma }^2=0$, all centered $Y_i$ are zero almost surely; if $M=0$, the same is immediate. Handle those cases directly rather than divide by zero. Both derived concentration inequalities also hold with $\ge ϵ$, by the same Markov inequality argument or by letting a smaller positive threshold increase to $ϵ$.

For the binomial distribution, write $X={\sum }_{i=1}^n{Z}_i$ with independent and identically distributed random variables $Z_i$ having the Bernoulli distribution with parameter $p=p_n$. Then $Y_i=Z_i-p$ has range $[-p,1-p]$, width one, and variance $p(1-p)$. For large $n$, $p<1/2$, so $M=1-p$ and $v=np(1-p)$. Set $ϵ=C\sqrt{v}$.

The Hoeffding inequality givesAfter truncation at one, this is asymptotically the trivial probability bound. The Bennett inequality instead givesSince $v\to \infty$ and $\varphi (u)=u^2/2+O(u^3)$ by a Taylor expansion,Thus the Bennett bounds for sparse binomial fluctuations yieldThe Bennett inequality supplies the smaller raw exponential bound for every fixed $C>0$ and all sufficiently large $n$. If $C\le \sqrt{2\log 2}$, clipping probability upper bounds at one makes both eventually trivial; for $C>\sqrt{2\log 2}$, the Bennett inequality retains a nontrivial limiting bound. This distinction avoids claiming a probability improvement after clipping when both raw bounds exceed one.

By the definition of exponential tilting,On $A_n$ one has $S_n\le cn$, so for $\lambda \ge 0$,Therefore

Set $Y_n=\log Z_n$ andDefine a probability measure ${\mathbb{P}}^{\ast }$ by the positive densityThe density has expectation one and is strictly positive, so ${\mathbb{P}}^{\ast }$ is equivalent to $\mathbb{P}$. By exponential tilting, the variables $Y_k$ remain independent under ${\mathbb{P}}^{\ast }$ and have distribution $N({\mu }_{\ast },{\sigma }^2)$. Thus $Z_k$ has a log-normal distribution andFor the discounted stock ${\tilde{S}}_n=S_n/S_n^0$,Hence ${\mathbb{P}}^{\ast }$ is an equivalent martingale measure.

Fix $a>m$ and then $x>a$. By the assumptions on the derivative ${\psi }^{\prime }$, there is a unique ${\theta }_x>0$ with ${\psi }^{\prime }({\theta }_x)=x$. Apply exponential tilting to each summand:Under the product tilted law, the variables remain independent and identically distributed random variables and have mean $x$. Hence the strong law of large numbers implies that, for every $0<\delta <x-a$,Changing measure on this event givesThereforeFirst let $\delta \downarrow 0$ and then $x\downarrow a$. The continuity of a convex function gives the lower bound $-{\psi }^{\ast }(a)$. The endpoint $a=m$ follows by letting $a\downarrow m$, while for $a<m$ the strong law of large numbers makes the probability tend to one. This proves the required lower bound.

Let $p_i>0$ be a probability mass function on a finite alphabet with numerical outcomes $a_i$. Prescribe a mean $m$ strictly between the smallest and largest outcomes. Exponential tilting giveswith $\theta$ chosen so that ${\sum }_i{q}_i^{\ast }{a}_i=m$. For any other feasible law $r$, direct substitution yieldsRelative entropy nonnegativity therefore proves optimality and uniqueness. At an extreme prescribed mean, feasible laws may have zero probabilities; a finite tilting parameter need not realize them, although a limit can.

For a regular two-parameter exponential family, condition on the sample mean of the nuisance canonical statistic. Exponential tilting to its observed value and the saddlepoint density approximation give a nuisance-statistic density proportional to $d_{\lambda \lambda }^{-1/2}$ times its large-deviation exponential. Dividing the joint likelihood by this density gives the profile log-likelihood plus the displayed adjustment, up to a data-only constant. The sign is positive. This construction assumes an interior saddle with positive curvature and a sufficiently regular nonlattice density; conditioning on a lattice statistic requires the corresponding mass approximation.

For independent identically distributed continuous random variables, let $K(t)=\log \mathbb{E}{e}^{tY}$ be finite near zero. At a mean value $x$ reached by $K^{\prime }(\hat{t})=x$ in the interior of that domain, the displayed leading approximation describes the density of the sum $S_n$. Exponential tilting by $\hat{t}$ puts the sum at its tilted mean; a local normal approximation gives the denominator, and undoing the tilt gives the exponential factor. Under suitable smooth nonlattice regularity, at fixed interior $x$ the exact density is the displayed quantity times $1+O(n^{-1})$. This statistical approximation is an application of the saddle-point approximation to a transform inversion integral. It requires more than the existence of a few ordinary moments.

For a random walk with independent and identically distributed random variables as increments, let $M_n=e^{\lambda S_n}/\varphi (\lambda {)}^n$ define its finite-time exponential tilting. Under the tilted law, $L_n=M_n^{-1}$ is a martingale: the conditional mean of $\varphi (\lambda )e^{-\lambda X_{n+1}}$ is one. For a stopping time $T$, partition $\{T\le n\}$ into $\{T=j\}$ and apply the tower property of conditional expectation to obtain the displayed identity. Only finite-time absolute continuity of measures is needed; there need not be an infinite-horizon likelihood-ratio density.