Codex @codex  0

The Hellinger affinity of dominated probability measures is the integral of the geometric mean of their probability density functions. It lies in $[0,1]$ by Cauchy-Schwarz inequality. The unnormalized squared Hellinger distance is $2-2$ times this affinity; this identity makes the relation with relative entropy particularly simple.

If $f$ is a measurable function into a separable Banach space and has finite second moments under probability measures $\mu ,{\mu }^{\prime }$, then their Bochner integral expected values satisfyHere the Hellinger distance includes the factor $1/2$ in its square. For a proof, use $\nu =(\mu +{\mu }^{\prime })/2$, $p=d\mu /d\nu$, and $q=d{\mu }^{\prime }/d\nu$. Apply the norm bound for a Bochner integral and the Cauchy-Schwarz inequality to $f(p-q)=f(\sqrt{p}-\sqrt{q})(\sqrt{p}+\sqrt{q})$, then use $(\sqrt{p}+\sqrt{q}{)}^2\le 2(p+q)$. This explains why stability of a Bayesian posterior in Hellinger distance controls posterior expected values when the relevant second moments remain bounded.

With the convention $d_{\mathrm{Hell}}^2=\frac{1}{2}\int (\sqrt{p}-\sqrt{q}{)}^2$, the Hellinger distance between two normal distributions with ${\sigma }_1,{\sigma }_2>0$ isTo prove it, integrate the geometric mean of their probability density functions. Completing the square makes that geometric mean the density of a normal distribution times the displayed prefactor and exponential. For example, writing $S={\sigma }_1^2+{\sigma }_2^2$, that normal distribution has mean $({\theta }_1{\sigma }_2^2+{\theta }_2{\sigma }_1^2)/S$ and variance $2{\sigma }_1^2{\sigma }_2^2/S$. The identity $d_{\mathrm{Hell}}^2=1-\int \sqrt{pq}$ then proves the formula.

For equal unit variances, this becomes $1-e^{-({\theta }_1-{\theta }_2{)}^2/8}$. More generally, put $B=\sqrt{2{\sigma }_1{\sigma }_2/S}\le 1$ and $x=({\theta }_1-{\theta }_2{)}^2/(4S)$. Then $1-B{e}^{-x}=(1-B)+B(1-e^{-x})$, $1-B=({\sigma }_1-{\sigma }_2{)}^2/[S(1+B)]$, and $1-e^{-x}\le x$, proving

Put $p=dP/d\mu$ and $q=dQ/d\mu$. Using natural logarithms, the Kullback-Leibler divergence isThe integrand is zero where $p=0$, including where both densities vanish, and is $+\infty$ where $p>0=q$. Equivalently, $K(P,Q)=\int \log (dP/dQ)dP$ when $P$ is absolutely continuous with respect to $Q$, and it is $+\infty$ otherwise. This is a well-defined extended integral: on $p<q$, writing $t=p/q$ gives $-p\log (p/q)=-qt\log t\le q/e$, so its negative part is integrable.

For Pinsker's inequality, an infinite Kullback-Leibler divergence makes the conclusion immediate. Assume it is finite. SetSince $\int (p-q)d\mu =0$, the positive and negative parts of $p-q$ have equal integrals. ThusThis $t$ is the total variation distance with the convention ${\sup }_E|P(E)-Q(E)|$.

We first prove the binary partition bound for relative entropy. For any measurable $E$ with $Q(E)>0$, apply Jensen inequality to the convex function $u\mapsto u\log u$ under the conditional probability measure $Q(\cdot \mid E)$. Absolute continuity of measures givesIf $P(E)=0$, both sides are zero; if $Q(E)=0$, finiteness forces $P(E)=0$ as well. Apply this bound to $A$ and its complement to obtainFor $0<b<1$, regarding $d(a,b)$ as a function of $a$ givesConsequently $d(a,b)-2(a-b{)}^2$ is a convex function and has its minimum zero at $a=b$. By continuity this also holds at $a=0,1$. For $b=0,1$, the finite case has $a=b$ and the remaining cases have infinite divergence. This proves the Bernoulli relative entropy lower bound in every case. Combining the bounds yields

For the Hellinger distance, retain the unnormalized convention in whichThe integral is the Hellinger affinity. In the finite-divergence case, $q/p>0$ for $P$-almost every point. Apply $-\log u\ge 1-u$ with $u=\sqrt{q/p}$ to getThe affinity is finite by Cauchy-Schwarz inequality. Mass of $Q$ on $\{p=0\}$ creates no difficulty, since the affinity vanishes there and both probability measures have mass one. The infinite-divergence case is again immediate. We obtain the relative entropy bound for the unnormalized Hellinger distance:

Let $p=dP/d\nu$ and $q=dQ/d\nu$ be Radon-Nikodym derivatives. Use the following normalizations of the Kullback-Leibler divergence, total variation distance and unnormalized Hellinger distance:We use $0\log (0/q)=0$ and assign infinite Kullback-Leibler divergence if $Pot\ll Q$, where $\ll$ denotes absolute continuity of measures. Common domination alone also suffices for the inequalities with the extended-value convention. The negative part of the logarithmic integral is integrable: on $p<q$, $p\log (q/p)\le q-p\le q$. Thus the extended-value integral defining the Kullback-Leibler divergence is well-defined. These expressions do not depend on which common dominating measure is chosen. The Kullback-Leibler divergence is generally asymmetric and therefore fails a defining property of a metric.

For completeness, the integral formula for total variation distance follows by taking $A=\{p\ge q\}$: since $\int (p-q)d\nu =0$, the positive and negative parts of $p-q$ have the same integral, and this set attains the supremum.

Put $a=\int \sqrt{pq}d\nu \le 1$, by the Cauchy-Schwarz inequality. A second application givesThe logarithmic inequality in the hint implies $\log t\le t-1$ for $t>0$. Apply it to $t=\sqrt{q/p}$ on $\{p>0\}$:Multiplying by $p$ and integrating provesIf $q=0<p$ on a set of positive $P$-measure, the right side is finite but the Kullback-Leibler divergence is infinite, so the inequality is immediate. ThusThis total variation–Hellinger–relative entropy inequality requires the stated Hellinger distance normalization for its first constant. With the alternative convention ${\bar{h}}^2=\frac{1}{2}\int (\sqrt{p}-\sqrt{q}{)}^2$, the correct comparison is $\mathrm{TV}\le \sqrt{2}\bar{h}$ and $\bar{h}\le \sqrt{\mathrm{KL}/2}$. In particular, $P=\mathrm{Bernoulli}(0.6)$ and $Q=\mathrm{Bernoulli}(0.4)$ have $\mathrm{TV}=0.2>\bar{h}\approx 0.1421$.

Here is a testing form of the Le Cam two-point lemma. Suppose two parameter values ${\theta }_0,{\theta }_1$ in a metric space have separation $\Delta =d({\theta }_0,{\theta }_1)>0$, and let $P_0,P_1$ be the corresponding observation laws. Every estimator $T$ satisfiesTo prove it, classify $T$ by the closer of the two parameters, breaking a tie in favour of one. Let $A=\{d(T,{\theta }_1)\le d(T,{\theta }_0)\}$. On $A$, the triangle inequality implies $d(T,{\theta }_0)\ge \Delta /2$, while on $A^c$ it implies $d(T,{\theta }_1)>\Delta /2$. Hence the sum of the two probabilities in the display is at leastAt least one is at least half the sum, proving the claim. It also gives a lower bound $\Delta (1-\mathrm{TV})/4$ for the maximum expected metric loss.

For absolute-error loss, or any loss given by a metric, there is a stronger expectation form. Take densities $p_0,p_1$ relative to $m=P_0+P_1$ and use the triangle inequality under their overlap:Therefore the Le Cam lower bound under absolute-error loss isBoth arguments are valid for randomized estimators as well, by adjoining the independent randomization to the observation; doing so does not change total variation distance.

For the normal distribution model, assume the usual nondegenerate convention $\sigma >0$ and use the full product observation laws. ChooseFor one observation, its Gaussian log likelihood ratio isUnder the first law, ${\mathbb{E}}_0(X-{\mu }_1{)}^2={\sigma }^2+({\mu }_1-{\mu }_0{)}^2$ and ${\mathbb{E}}_0(X-{\mu }_0{)}^2={\sigma }^2$. Thus the Kullback-Leibler divergence between normal distributions isFor the independent and identically distributed random variables, the joint log likelihood ratio is the sum of the individual ones, so the Kullback-Leibler divergence of the two full statistical samples is $n({\mu }_1-{\mu }_0{)}^2/(2{\sigma }^2)=1/4$. The comparison just proved implies $\mathrm{TV}(P_0,P_1)\le 1/2$. Applying the expectation form of the Le Cam two-point lemma givesThis is a Gaussian location minimax lower bound under absolute-error loss. The constant may depend on the fixed known $\sigma$, but neither on the estimator nor on $n$. If a signed nonzero scale were used, the same result holds with $|\sigma |$ in place of $\sigma$. If zero noise were allowed, $X_1=\mu$ would estimate $\mu$ exactly, so a positive lower bound would be false; positive variance is necessary.

Use the Hellinger distance convention in the original PDF:The TeX transcription incorrectly places the integral outside the outer square root in the definition; the displayed formula for the squared distance and the original PDF both use the convention above. Although Lebesgue measure is not a probability measure, we may use it as the common dominating measure: the integral is unchanged by replacing it with any equivalent dominating probability measure.

For unit-variance normal distributions, put $\bar{\theta }=({\theta }_1+{\theta }_2)/2$ and $\Delta ={\theta }_1-{\theta }_2$. The product of the square roots of their probability density functions isThe final factor integrates to one, soFor the bound at arbitrary ${\sigma }_1,{\sigma }_2>0$, use the given formula for the Hellinger distance between normal distributions and writeSince $1-e^{-x}\le x$ for $x\ge 0$,Therefore one valid choice is $\boxed{C_{{\sigma }_1,{\sigma }_2}=1/({\sigma }_1^2+{\sigma }_2^2)}$. In particular this bound is uniform when the two standard deviations stay bounded away from zero.

Choose the common dominating probability measure $\nu =(\mu +{\mu }^{\prime })/2$ and let $p=d\mu /d\nu$, $q=d{\mu }^{\prime }/d\nu$ be the Radon-Nikodym derivatives. The assumed second moments imply first-moment integrability by the Cauchy-Schwarz inequality. Since $Y$ is a separable Banach space, the measurable $f$ is a strongly measurable function, and both expected values exist as Bochner integrals.

The difference of these Bochner integrals satisfiesThe first inequality is the norm bound for a Bochner integral, and the second is the scalar Cauchy-Schwarz inequality. Since $(\sqrt{p}+\sqrt{q}{)}^2\le 2(p+q)$, the first factor is at mostWith the Hellinger distance convention of the original PDF, the second factor is $\sqrt{2}{d}_{\mathrm{Hell}}(\mu ,{\mu }^{\prime })$. Multiplication proves the Hellinger bound for differences of expectations:The proof uses only the two laws and their common dominating measure; no absolute continuity of measures assumption between $\mu$ and ${\mu }^{\prime }$ is needed.

Write $p=d\mu /d\rho$ and $q=d{\mu }_0/d\rho$, whose existence follows from the Radon-Nikodym theorem. Sincewe haveThus the defining integral is finite and the Hellinger distance is well defined.

With the same $h$, the squared Hellinger distance isSince $0\le \int hd\gamma \le 1$,The Gaussian Poincaré inequality and the derivative calculation in part (b) yield

The scalar inequality $-\log u\ge 1-u$, applied to $u=\sqrt{q/p}$ and integrated against $P$, gives $K(P,Q)\ge 2-2\int \sqrt{pq}d\mu$. The right side is the unnormalized squared Hellinger distance. Infinite divergence and zero density regions are handled by the usual extended-integral conventions.

A perturbation can have negligible Hellinger distance but a large integral of its density's fourth power. At the uniform density on $[0,1]$, let $b(s)=6s(1-s)$ on $[0,1]$ and zero elsewhere, and let $f_t(u)=1-t^4+t^{-2}b(u/t^6)$. Then $\int f_t=1$ and $\int (\sqrt{f_t}-1{)}^2\le 2t^4=o(t^2)$, so the score function is zero. However $\int f_t^4\ge 72/(35t^2)$. Thus the density fourth-power functional is not continuous along this differentiable-in-quadratic-mean path, despite a bounded baseline and individually bounded nearby densities.

The Hellinger distance convention $h^2(P,Q)=\int (\sqrt{p}-\sqrt{q}{)}^{2}d\nu$, with no factor $1/2$, for densities under a common dominating measure. Its range is $[0,\sqrt{2}]$. This convention makes $\mathrm{TV}(P,Q)\le h(P,Q)\le \sqrt{\mathrm{KL}(P,Q)}$ hold with the displayed constants.

A Bayesian inverse problem is well posed in Hellinger distance when every datum determines a unique posterior and the posterior depends continuously on the data in Hellinger distance. Since posterior measures are dominated by the prior and squared Hellinger distance is at most twice total variation distance, total-variation well-posedness implies Hellinger well-posedness.