Hellinger distance ($d_H(P,Q)$)

The Hellinger distance is the square root of a constant multiple of $\int (\sqrt{p}-\sqrt{q}{)}^{2}d\lambda$. Common conventions use either the factor $1$ or $1/2$.

The scalar inequality $-\log u\ge 1-u$, applied to $u=\sqrt{q/p}$ and integrated against $P$, gives $K(P,Q)\ge 2-2\int \sqrt{pq}d\mu$. The right side is the unnormalized squared Hellinger distance. Infinite divergence and zero density regions are handled by the usual extended-integral conventions.

The Hellinger affinity of dominated probability measures is the integral of the geometric mean of their probability density functions. It lies in $[0,1]$ by Cauchy-Schwarz inequality. The unnormalized squared Hellinger distance is $2-2$ times this affinity; this identity makes the relation with relative entropy particularly simple.

For the unnormalized Hellinger distance, $\mathrm{TV}(P,Q)\le h(P,Q)\le \sqrt{\mathrm{KL}(P,Q)}$. Factor $|p-q|=|\sqrt{p}-\sqrt{q}|(\sqrt{p}+\sqrt{q})$ and use the Cauchy-Schwarz inequality to bound the total variation distance. For the other bound, $\log t\le t-1$ gives $p\log (p/q)\ge 2(p-\sqrt{pq})$, whose integral is the squared unnormalized Hellinger distance. Infinite Kullback-Leibler divergence is allowed.

The Hellinger distance convention $h^2(P,Q)=\int (\sqrt{p}-\sqrt{q}{)}^{2}d\nu$, with no factor $1/2$, for densities under a common dominating measure. Its range is $[0,\sqrt{2}]$. This convention makes $\mathrm{TV}(P,Q)\le h(P,Q)\le \sqrt{\mathrm{KL}(P,Q)}$ hold with the displayed constants.

If $f$ is a measurable function into a separable Banach space and has finite second moments under probability measures $\mu ,{\mu }^{\prime }$, then their Bochner integral expected values satisfyHere the Hellinger distance includes the factor $1/2$ in its square. For a proof, use $\nu =(\mu +{\mu }^{\prime })/2$, $p=d\mu /d\nu$, and $q=d{\mu }^{\prime }/d\nu$. Apply the norm bound for a Bochner integral and the Cauchy-Schwarz inequality to $f(p-q)=f(\sqrt{p}-\sqrt{q})(\sqrt{p}+\sqrt{q})$, then use $(\sqrt{p}+\sqrt{q}{)}^2\le 2(p+q)$. This explains why stability of a Bayesian posterior in Hellinger distance controls posterior expected values when the relevant second moments remain bounded.

With the convention $d_{\mathrm{Hell}}^2=\frac{1}{2}\int (\sqrt{p}-\sqrt{q}{)}^2$, the Hellinger distance between two normal distributions with ${\sigma }_1,{\sigma }_2>0$ isTo prove it, integrate the geometric mean of their probability density functions. Completing the square makes that geometric mean the density of a normal distribution times the displayed prefactor and exponential. For example, writing $S={\sigma }_1^2+{\sigma }_2^2$, that normal distribution has mean $({\theta }_1{\sigma }_2^2+{\theta }_2{\sigma }_1^2)/S$ and variance $2{\sigma }_1^2{\sigma }_2^2/S$. The identity $d_{\mathrm{Hell}}^2=1-\int \sqrt{pq}$ then proves the formula.

For equal unit variances, this becomes $1-e^{-({\theta }_1-{\theta }_2{)}^2/8}$. More generally, put $B=\sqrt{2{\sigma }_1{\sigma }_2/S}\le 1$ and $x=({\theta }_1-{\theta }_2{)}^2/(4S)$. Then $1-B{e}^{-x}=(1-B)+B(1-e^{-x})$, $1-B=({\sigma }_1-{\sigma }_2{)}^2/[S(1+B)]$, and $1-e^{-x}\le x$, proving