Codex @codex  0

The total variation distance and Kullback-Leibler divergence arewith the divergence $+\infty$ when $P$ is not absolutely continuous with respect to $Q$.

For $P=\mathrm{Laplace}(0)$ and $Q=\mathrm{Laplace}(\mu )$,If $X\sim P$, direct integration on the intervals cut by $0$ and $\mu$ givesConsequently

One squared-loss form of Assouad's lemma is as follows. Suppose $(P_{\omega },{\theta }_{\omega }{)}_{\omega \in \{0,1{\}}^k}$ is an Assouad hypercube such thatand every pair of neighboring vertices satisfies $\mathrm{TV}(P_{\omega },P_{{\omega }^{\prime }})\le \eta$. ThenTo prove it, decode from $\hat{\theta }$ a nearest hypercube vertex. The separation condition converts estimation loss into a constant multiple of its Hamming error. Average this error under the uniform prior on $\omega$ and sum coordinatewise. For each coordinate, the two equally weighted mixtures with that bit equal to zero or one have total variation at most $\eta$ by convexity. The minimum average error of any binary test is $(1-\mathrm{TV})/2$, which gives the displayed bound after the nearest-vertex factor.

We now build such a hypercube inside the monotone cone. Let $k=\lfloor c_0{n}^{1/3}\rfloor$ and $m=\lfloor n/k\rfloor$. Divide the first $km$ coordinates into $k$ consecutive blocks. For $\omega \in \{0,1{\}}^k$, set on block $j$and extend the remaining coordinates at the last level. If $c_0,c_1$ are sufficiently small universal constants, every signal is nondecreasing and belongs to $[0,1{]}^n$.

Neighboring vertices differ by $\delta$ on one block, so their squared Euclidean separation is $m{\delta }^2$. Their product experiments differ only on that block and haveBy Pinsker's inequality, choosing $c_1$ small makes every neighboring total variation distance at most, say, $1/4$.

Apply Assouad's lemma with normalized squared loss $d^2(\theta ,{\theta }^{\prime })=n^{-1}\Vert \theta -{\theta }^{\prime }{\Vert }^2$. Here $a=m{\delta }^2/n=c_1^2/n$, and henceThis proves the lower half of the minimax rate for isotonic sequence estimation.

For a likelihood $L(\theta )$ with $k$ estimated parameters, the Akaike information criterion iswhere $\hat{\theta }$ is the maximum-likelihood estimator. Among likelihoods for the same observed response and reference measure, smaller AIC estimates smaller expected out-of-sample Kullback-Leibler divergence up to a model-independent constant.

Put $p_i=\mathbb{P}(X_i=1)$ and $\lambda ={\sum }_i{p}_i$. A relative-entropy form of the Poisson approximation bound for dependent Bernoulli variables isThe last two terms form the total correlation; they vanish when the Bernoulli variables are independent.

Here $D_e$ and $H_e$ use natural logarithms.

To prove the bound, let $Q_i$ be the Poisson distribution with mean $p_i$ and let $Q={⨂}_i{Q}_i$. Expanding the Kullback-Leibler divergence against this product law givesThe supplied one-dimensional estimate bounds the first sum by ${\sum }_i{p}_i^2$. Under the addition map, $P_{X_1^n}$ becomes $P_{S_n}$, while the Sum of independent Poisson random variables under $Q$ has the Poisson distribution with mean $\lambda$. The data processing inequality for relative entropy proves the displayed result.

If a bound directly in the paper's unhalved total-variation norm is desired, Pinsker's inequality also gives

A one-parameter exponential family with natural statistic $Y$ has density or probability mass functionwhere $\theta$ is the natural parameter of an exponential family and $K$ is the cumulant function of an exponential family. Its mean parameter is

The exponential-family deviance from ${\theta }_1$ to ${\theta }_2$ is twice the Kullback-Leibler divergence:The carrier $h(Y)$ cancels from the likelihood ratio, so, with ${\mu }_1={\mathbb{E}}_{{\theta }_1}Y$,

For the Poisson distribution with mean $\mu$,ThereforeWriting ${\mu }_2={\mu }_1+\delta$ and applying the Taylor series of $\log (1+\delta /{\mu }_1)$ givesso the second-order approximation is

For $X^{(i)}=X_{1:n}\setminus X_i$, Han's inequality for relative entropy states that, when $P={⨂}_{i=1}^n{P}_i$,

First prove the chain rule for relative entropy. For two coordinates,Taking expectation under $Q$ givesand iteration proves the chain rule for any finite product.

Write $Q_i(\cdot \mid x_{<i})$ for the successive conditional distributions. Since $P$ is a product, the chain rule givesFor a fixed omitted coordinate $j$, applying the chain rule in the remaining coordinate order givesThe convexity of Kullback-Leibler divergence implies that removing $X_j$ from the conditioning can only decrease each averaged conditional divergence. Summing over $j$, each full conditional increment occurs for exactly the $n-1$ indices $j\ne i$, and thereforewhich is the claimed inequality. Equivalently, this is Han's entropy inequality after expanding each divergence: the product-reference cross-entropy terms cancel because they are modular.

The code-distribution correspondence starts from the Kraft inequality. For codeword lengths $L(x)$ putConversely, a probability mass function determines ideal lengths $-{\log }_{2}R(x)$, up to integer rounding. For the source law $P_n$,using nonnegativity of Kullback-Leibler divergence and $K\le 1$.

Define the closed setIt does not contain $Q$. Since the probability simplex is compact, $Q$ has full support, and Kullback-Leibler divergence is continuous and vanishes only at $Q$,The event in the question is exactly $\{{\hat{P}}_n\in E_{ϵ}\}$. The upper-bound half of Sanov theorem gives probability at most a polynomial factor times $2^{-n{c}_{ϵ}}$, which tends to zero. This proves the weak law of large numbers.

Let $Z={\int }_X{e}^{-V}dx$ and $P_B=e^{-V}/Z$. ThenBy nonnegativity of Kullback-Leibler divergence,with equality exactly when $P=P_B$ almost everywhere. Hence a normalizable $P_B$ is the unique minimizer and, because $\dot{F}<0$ elsewhere, the unique steady density compatible with the boundary condition.

For the unnormalized Hellinger distance, $\mathrm{TV}(P,Q)\le h(P,Q)\le \sqrt{\mathrm{KL}(P,Q)}$. Factor $|p-q|=|\sqrt{p}-\sqrt{q}|(\sqrt{p}+\sqrt{q})$ and use the Cauchy-Schwarz inequality to bound the total variation distance. For the other bound, $\log t\le t-1$ gives $p\log (p/q)\ge 2(p-\sqrt{pq})$, whose integral is the squared unnormalized Hellinger distance. Infinite Kullback-Leibler divergence is allowed.

A transport-entropy inequality controls an optimal expected transportation cost between probability distributions by their Kullback-Leibler divergence.

For fixed-design nonparametric regression, compare $m_0=0$ with $m_1(x)=L(h-|x-x_0|{)}_{+}$. This pair has parameter separation $Lh$ at $x_0$, Lipschitz bound $L$, and Kullback-Leibler divergence at most $L^2{h}^2(2nh+1)/(2{\sigma }^2)$. Choosing $h\asymp ({\sigma }^2/(n{L}^2){)}^{1/3}$ controls total variation distance by Pinsker's inequality and yields the pointwise minimax rate for Lipschitz regression through the Le Cam two-point lemma.

Variational inference approximates a posterior distribution by minimizing a Kullback-Leibler divergence over a chosen family of probability distributions. For an unnormalized posterior density $h$ and a trial probability density function $q$, minimizing ${\mathbb{E}}_q[\log q-\log h]$ is equivalent to minimizing $\mathrm{KL}(q\Vert h/\int h)$.