Codex @codex  0

For probability measures on ${\mathbb{R}}^d$ with finite second moments and a source satisfying absolute continuity of measures with respect to Lebesgue measure, the quadratic Kantorovich optimal transport problem has a unique optimal transport plan. It is induced by the gradient of a convex function, which also uniquely solves the Monge optimal transport problem up to a source-null set.

A complex measure is a countably additive complex-valued set function on a sigma-algebra. Its total variation norm of a measure is obtained by taking the supremum of sums ${\sum }_j|\nu (E_j)|$ over finite measurable partitions. For finite variation and absolute continuity of measures with respect to a sigma-finite positive measure, the Radon-Nikodym theorem supplies an integrable complex density.

A density ratio compares two probability density functions relative to the same underlying measure. More generally it is a Radon-Nikodym derivative when one probability measure has absolute continuity of measures with respect to the other.

A measure $\rho$ dominates a family of measures when every member is absolutely continuous with respect to $\rho$. Any two finite positive measures $\mu$ and $\nu$ have the common dominating measure $\mu +\nu$.

Two probability measures are equivalent when each is absolutely continuous with respect to the other, or equivalently when they have the same null events.

If a finite complex Borel measure $\mu$ on the circle has Fourier coefficients $\hat{\mu }(n)=0$ for all $n<0$, then it is absolutely continuous with respect to arc length. The reflected Fourier convention gives the equivalent theorem with positive coefficients vanishing. The Poisson integral on the unit disk is analytic because of the one-sided Fourier spectrum. Subharmonic comparison for its square root and an $L^2$ maximal estimate produce an integrable radial maximum. This maximum dominates the weak-star boundary measure, forcing absolute continuity of measures. The same proof on the real line says that a finite measure with Fourier transform of a finite measure supported in $[0,\infty )$ has a Lebesgue density.

The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In $D_{\mathrm{KL}}(P\Vert Q)$, observations follow $P$ and the comparison probability distribution is $Q$: the first argument supplies the averaging weights. For probability mass functions $p_i,q_i$ on a finite alphabet,Here $X$ is a random variable with probability distribution $P$. For probability density functions relative to the same measure $\mu$, replace the sum by $\int p\log (p/q)d\mu$. These formulas use natural logarithms unless stated otherwise. The expected value is under $P$, even though $Q$ appears in the denominator. ^[ref]

Take $P=(0.75,0.25)$ and $Q=(0.5,0.5)$ on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than $P$ does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.

The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by $\log 2$ to obtain bits; the forward example is approximately $0.188722$ bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. ^[ref]

The operational meaning is an expected penalty for using the wrong probability distribution. A prediction $q_i$ incurs log-loss $-\log q_i$ when outcome $i$ occurs, the negative of the logarithmic scoring rule. Under $P$, its expected value is the cross-entropy $H(P,Q)=-{\sum }_i{p}_i\log q_i$. Predicting with $P$ itself incurs information entropy $H(P)=-{\sum }_i{p}_i\log p_i$, soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. ^[ref]

For lossless source coding, ideal codeword lengths $-{\log }_2{q}_i$ similarly give excess expected length $D_{\mathrm{KL}}(P\Vert Q)$ in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with $Q$ costs one bit per symbol, while $H(P)\approx 0.811278$ bits; the difference is exactly the computed $0.188722$ bits. ^[ref]

Zeros require explicit conventions. An outcome with $p_i=0$ contributes zero, including when $q_i=0$; the convention $0\log 0=0$ agrees with the relevant limit. If $p_i>0$ but $q_i=0$, the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of $P$ must therefore be contained in the probability support of $Q$ for a finite answer. For example, $P=(1,0)$ and $Q=(0.5,0.5)$ give $D(P\Vert Q)=\log 2$, but $D(Q\Vert P)=+\infty$. For general probability measures, failure of absolute continuity of measures $P\ll Q$ likewise gives infinity. Even $P\ll Q$ does not guarantee a finite integral. ^[ref]

To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at $p=0.75$ and vary the predicted parameter $q$. Its binary relative entropy isIt vanishes at $q=0.75$, equals approximately $2.894055$ nats at $q=0.01$, and equals $0.596495$ nats at $q=0.99$. Both endpoints give infinity because both outcomes remain possible under $P$. Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and $P$ assigns positive probability to both.

For model selection, minimizing expected log-loss ranks candidate statistical models by $D(P\Vert Q)$, since $H(P)$ is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.

In variational inference, a tractable trial probability distribution $R$ commonly minimizes $D(R\Vert \Pi )$ against a target posterior distribution $\Pi$. Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows $R$: placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. ^[ref]

These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. ^[ref]

The original reference is S. Kullback and R. A. Leibler's 1951 paper ^[ref], published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference.

A measurable transformation $T:X\to X$ is nonsingular with respect to $\mu$ when $\mu (T^{-1}N)=0$ for every $\mu$-null set $N$, equivalently when the pushforward measure $T_{\ast }\mu$ is absolutely continuous with respect to $\mu$.

Here is a finite-horizon Girsanov theorem statement with the sign fixed explicitly. Let $B$ be a Brownian motion under $P$, and let $\theta$ be predictable with ${\int }_0^T{\theta }_s^{2}ds<\infty$ almost surely. Ifis a true martingale with $\mathbb{E}{L}_T=1$, then the measure $Q$ defined by $dQ=L_{T}dP$ makes $B_t-{\int }_0^t{\theta }_{s}ds$ a Brownian motion up to $T$. The sufficient Novikov condition is $\mathbb{E}\exp (\frac{1}{2}{\int }_0^T{\theta }_s^{2}ds)<\infty$. Merely being a local martingale is not a sufficient density condition.

For a deterministic absolutely continuous function $h$ with $h(0)=0$ and derivative $g\in L^2[0,T]$, take $\theta =g$. Novikov's condition holds because its exponential is deterministic and finite. Under $Q$, the canonical process has the law of a Brownian motion plus $h$. Thus for every bounded measurable path functional $F$,This derives the Cameron-Martin theorem density formula from the Girsanov theorem, rather than assuming the two measures have the same drift convention. Replacing $h$ by $-h$ changes the sign of the stochastic integral.

The question's path space is the whole half-line, not just one finite horizon. When $g\in L^2(0,\infty )$, the density martingale satisfiesIt has a uniformly integrable limit $L_{\infty }=\exp ({\int }_0^{\infty }{g}_{s}d{B}_s-\Vert g{\Vert }_2^2/2)$ of mean one, strictly positive almost surely. Integrals here converge in $L^2$ and almost surely. The finite-time identities then identify $L_{\infty }$ as the density on the full path sigma-algebra, since finite-time cylinder events generate it. This is Cameron-Martin shifts on infinite Wiener path space. Finite-horizon absolute continuity of measures alone does not imply infinite-horizon absolute continuity of measures.

Put $M={\sup }_{y>0}{\int }_{\mathbb{R}}|f(x+iy)|dx$. The goal is to obtain an integrable non-tangential maximal function without assuming in advance that $f$ has an integrable boundary density. We use a square root to place the boundary data in $L^2$, where the Hardy-Littlewood maximal function is strongly bounded.

First recall this strong bound directly from Q2. For a nonnegative $h\in L^2(\mathbb{R})$ and $\lambda >0$, split $h=h{1}_{\{h>\lambda /2\}}+h{1}_{\{h\le \lambda /2\}}$. The second summand has centered maximal function at most $\lambda /2$; the first is in $L^1$. Q2's weak estimate, in dimension one, therefore givesThe layer cake representation and Tonelli theorem then implyThis is the needed Strong Lp bound for the Hardy-Littlewood maximal function, with an explicit adequate constant.

For $\epsilon >0$, let $f_{\epsilon }(z)=f(z+i\epsilon )$ and $h_{\epsilon }(x)=|f(x+i\epsilon ){|}^{1/2}$. Q1's elementary harmonic pointwise estimate makes $f_{\epsilon }$ bounded on the closed upper half-plane. Its boundary value is continuous, and $\Vert h_{\epsilon }{\Vert }_2^2\le M$. The function $|f_{\epsilon }{|}^{1/2}$ is subharmonic: away from zeros, for any $q>0$,and regularization, or the submean inequality at zeros, extends this to the whole domain. Bounded subharmonic comparison with the Poisson integral givesHere $h_{\epsilon }$ need not be in $L^1$. The Poisson integral is nevertheless defined, since $h_{\epsilon }\in L^2\cap L^{\infty }$ and $P_y\in L^1\cap L^2$, and it tends to its continuous boundary values. To justify comparison on this unbounded domain, the difference between the two sides is bounded above, subharmonic, and zero on the real boundary. On an upper half-disk of radius $R$, use the positive harmonic barrierOn its semicircular arc, $b_R(R{e}^{i\theta })=2/\sin \theta \ge 2$, while $b_R(z)\to 0$ for every fixed interior $z$ as $R\to \infty$. The maximum principle for subharmonic functions, with a fixed bound for the difference, proves the comparison. The barrier's positive blow-up at the arc endpoints causes no difficulty.

For $|x-x^{\prime }|\le y$, the elementary inequality $y^2+|x-s{|}^2\le 3(y^2+|x^{\prime }-s{|}^2)$ givesQ2's radial decreasing kernel domination by the maximal function extends to nonnegative $L^2$ inputs by applying it to increasing bounded compactly supported truncations. Hence, taking the supremum over the cone,For any fixed cone point, $f_{\epsilon }(z)\to f(z)$ as $\epsilon \downarrow 0$. Therefore $f^{\ast }(x)\le {\mathrm{liminf}}_{\epsilon \downarrow 0}{f}_{\epsilon }^{\ast }(x)$; use a sequence of positive shifts and the Fatou lemma to concludeThe precise numerical constant is unimportant. Suprema over countable dense sets of cone points give the same values, so the maximal functions used here are measurable.

This yields boundary absolute continuity of measures rather than assuming it. The holomorphic function $f$ is a complex-valued harmonic function, so Q1 represents it as $f(x+iy)=P_y\ast \mu (x)$ for a finite complex measure $\mu$, with $f(\cdot +iy)dx\to \mu$ weak-star. Since $|f(x+iy)|\le f^{\ast }(x)$, every compactly supported continuous $\psi$ satisfiesThe dual characterization of the total variation measure implies $|\mu |\le f^{\ast }dx$, so the Radon-Nikodym theorem gives $\mu =Fdx$ with $F\in L^1$. The approximate identity now gives $f(\cdot +iy)\to F$ in the L1 norm and almost everywhere. Thus the analytic Hardy space has an integrable boundary function, unlike the unrestricted harmonic $h^1$ space.

In the real-line form of the F. and M. Riesz theorem, start with a finite complex measure whose Fourier transform of a finite measure vanishes for $\xi <0$. Its Poisson integral is holomorphic: the transformed operators ${\partial }_y$ and $i{\partial }_x$ act by $-|\xi |$ and $-\xi$, which agree on that spectrum. These identities can first be read as distributional identities and then classically, since the Poisson integral is smooth. Its harmonic $h^1$ norm is finite by Q1, so the preceding argument makes the measure absolutely continuous. A one-sided Fourier spectrum forces a finite boundary measure to have a Lebesgue density.

For the customary circle form, normalize arc length by $dm=d\theta /(2\pi )$. If $\hat{\mu }(n)=\int e^{-in\theta }d\mu (\theta )=0$ for $n<0$, thenis analytic and $F(r{e}^{i\theta })$ is the Poisson integral on the unit disk of $\mu$. Positivity and unit mass of the Poisson kernel on the circle imply ${\sup }_r\int |F(r{e}^{i\theta })|dm\le \Vert \mu {\Vert }_{\mathrm{TV}}$. The same square-root argument on the disk gives an integrable radial maximum, as follows. For $0<\rho <1$, put $h_{\rho }(\theta )=|F(\rho e^{i\theta }){|}^{1/2}$. The maximum principle for subharmonic functions on the disk givesThe supremum of these Poisson integrals is bounded by a constant times the circular Hardy-Littlewood maximal function. Indeed, for $r\ge 1/2$ the circular kernel is bounded by a constant times $(1-r)/((1-r{)}^2+{\theta }^2)$ for $|\theta |\le \pi$; splitting into arcs of radii $2^j(1-r)$ gives a summable geometric series of arc averages. For $r<1/2$ the kernel is uniformly bounded, and the whole-circle average suffices. The covering proof and the truncation argument already used on the line give the strong $L^2$ maximal bound for arcs on the circle as well. ThusLet $\rho \uparrow 1$ and apply Fatou lemma to obtain an integrable radial maximum $R(\theta )={\sup }_{0<r<1}|F(r{e}^{i\theta })|$. The measures $F(r{e}^{i\theta })dm$ tend weak-star to $\mu$ and are dominated in magnitude by $Rdm$. The same continuous-test-function argument gives $|\mu |\le Rdm$. This proves the F. and M. Riesz theorem: a finite complex measure on the circle with all negative Fourier coefficients zero is absolutely continuous with respect to arc length. Reversing the Fourier convention interchanges positive and negative indices.

We construct the density and singular part, rather than assuming the desired Lebesgue decomposition theorem. Put $\rho =\mu +\nu$, a finite dominating measure, and use the real Hilbert space $L^2(\rho )$. The linear functionalis well defined on its equivalence classes, since a $\rho$-null set is $\nu$-null. The Cauchy-Schwarz inequality and $\nu \le \rho$ giveBy the Riesz representation theorem, there is $h\in L^2(\rho )$ with $T(g)=\int ghd\rho$. Indicators of all measurable sets belong to this Hilbert space, soThe nonnegativity of these two measures forces $0\le h\le 1$ almost everywhere. For instance, if $\rho (\{h\le -1/n\})>0$, its $\nu$-measure would be negative; the analogous sets $\{h\ge 1+1/n\}$ contradict positivity of $\mu$. Their countable union excludes every violation. Modify $h$ on its measurable $\rho$-null exceptional set so that these inequalities hold everywhere. This is the Hilbert-space construction of dominated measure densities.

DefineThen $B\in \Sigma$, $f$ is a nonnegative measurable function, and $\mu (B)={\int }_B(1-h)d\rho =0$. The identity for integration against $\mu$ follows first for indicators, then for nonnegative simple functions and finally for all nonnegative functions by the monotone convergence theorem. Applying it to $f{1}_A$ givesIn particular ${\int }_{\Omega }fd\mu =\nu (\Omega \setminus B)<\infty$, so $f\in L^1(\mu )$. Additivity on the disjoint measurable sets $A\setminus B$ and $A\cap B$ now givesThis is the Lebesgue decomposition from a sum-measure density: the first measure has a density, and the second is concentrated on a $\mu$-null set, so is mutually singular with $\mu$.

The absolute continuity of measures condition $\nu \ll \mu$ means that $\mu (A)=0$ implies $\nu (A)=0$ for every $A\in \Sigma$. If this holds, then $\nu (B)=0$, eliminating the singular part above. Conversely, a measure of the form $\nu (A)={\int }_{A}fd\mu$ vanishes on every $\mu$-null set. ThereforeThus absolute continuity is exactly the absence of the singular part in this construction.

Construct the densities using the Riesz representation theorem for a Hilbert space, rather than assuming the measure-decomposition conclusion. Put $\rho =\mu +\nu$, a finite dominating measure, and work in the real L2 space $L^2(\rho )$. The linear functionalis well-defined on its equivalence classes because $\nu \le \rho$. By the Cauchy-Schwarz inequality,Thus the Riesz representation theorem supplies $h\in L^2(\rho )$ such that $L(g)=\int ghd\rho$. Since $\rho$ is finite, $h$ is also integrable. Testing $g=1_A$ givesThe first identity and nonnegativity of $\nu$ rule out a positive-measure set where $h<0$; for example, test $\{h\le -1/n\}$. The second identity and nonnegativity of $\mu$ similarly rule out $\{h\ge 1+1/n\}$. Modify a measurable representative on a $\rho$-null set so that $0\le h\le 1$ everywhere. This is the Hilbert-space construction of dominated measure densities.

SetThen $B\in \Sigma$ and $\mu (B)=0$. The indicator identities for $\mu$ extend to integrals of nonnegative measurable functions by approximation with simple functions and the monotone convergence theorem. Consequently,and, for every $A$,HenceThis proves Lebesgue decomposition from a sum-measure density. The argument includes the cases of zero measures; there is no division on $B$, where the denominator vanishes.

Absolute continuity of measures, written $\nu \ll \mu$, means that $\mu (A)=0$ implies $\nu (A)=0$ for every $A\in \Sigma$. If $\nu \ll \mu$, then $\nu (B)=0$ in the construction above, so the singular term vanishes. Conversely, if $\nu (A)={\int }_{A}fd\mu$ with a nonnegative integrable $f$, that integral is zero on every $\mu$-null set. ThusThis is the finite-measure Radon-Nikodym theorem as a consequence of the proved construction.

Finally restrict the base measure to the sub-sigma-algebra ${\Sigma }_0$. For a real $f\in L^1(\mu )$, write $f=f^{+}-f^{-}$, with both parts nonnegative and integrable. On ${\Sigma }_0$ define finite positive measuresThey are absolutely continuous with respect to $\mu {|}_{{\Sigma }_0}$. The result just proved gives nonnegative, ${\Sigma }_0$-measurable densities $g_{\pm }$ satisfyingSet $f_0=g_{+}-g_{-}$. Since $\int (g_{+}+g_{-})d\mu =\int |f|d\mu <\infty$, this is in $L^1(\Omega ,{\Sigma }_0,\mu )$, and subtraction givesFor complex $f$, apply the real construction separately to its real and imaginary parts and combine the two results. This is conditional expectation from finite-measure densities. The resulting conditional expectation is unique almost everywhere: for two real versions, their difference has integral zero on its positive and negative level sets in ${\Sigma }_0$, forcing it to vanish; apply this to both components in the complex case. No probability normalization of $\mu$ is required.

Put $p=dP/d\mu$ and $q=dQ/d\mu$. Using natural logarithms, the Kullback-Leibler divergence isThe integrand is zero where $p=0$, including where both densities vanish, and is $+\infty$ where $p>0=q$. Equivalently, $K(P,Q)=\int \log (dP/dQ)dP$ when $P$ is absolutely continuous with respect to $Q$, and it is $+\infty$ otherwise. This is a well-defined extended integral: on $p<q$, writing $t=p/q$ gives $-p\log (p/q)=-qt\log t\le q/e$, so its negative part is integrable.

For Pinsker's inequality, an infinite Kullback-Leibler divergence makes the conclusion immediate. Assume it is finite. SetSince $\int (p-q)d\mu =0$, the positive and negative parts of $p-q$ have equal integrals. ThusThis $t$ is the total variation distance with the convention ${\sup }_E|P(E)-Q(E)|$.

We first prove the binary partition bound for relative entropy. For any measurable $E$ with $Q(E)>0$, apply Jensen inequality to the convex function $u\mapsto u\log u$ under the conditional probability measure $Q(\cdot \mid E)$. Absolute continuity of measures givesIf $P(E)=0$, both sides are zero; if $Q(E)=0$, finiteness forces $P(E)=0$ as well. Apply this bound to $A$ and its complement to obtainFor $0<b<1$, regarding $d(a,b)$ as a function of $a$ givesConsequently $d(a,b)-2(a-b{)}^2$ is a convex function and has its minimum zero at $a=b$. By continuity this also holds at $a=0,1$. For $b=0,1$, the finite case has $a=b$ and the remaining cases have infinite divergence. This proves the Bernoulli relative entropy lower bound in every case. Combining the bounds yields

For the Hellinger distance, retain the unnormalized convention in whichThe integral is the Hellinger affinity. In the finite-divergence case, $q/p>0$ for $P$-almost every point. Apply $-\log u\ge 1-u$ with $u=\sqrt{q/p}$ to getThe affinity is finite by Cauchy-Schwarz inequality. Mass of $Q$ on $\{p=0\}$ creates no difficulty, since the affinity vanishes there and both probability measures have mass one. The infinite-divergence case is again immediate. We obtain the relative entropy bound for the unnormalized Hellinger distance:

The Radon-Nikodym theorem for positive measures says: if $\mu$ and $\nu$ are sigma-finite measures on the same measurable space, and $\nu$ is absolutely continuous with respect to $\mu$, then there is a nonnegative measurable function $g$, unique $\mu$-almost everywhere, such thatHere sigma-finiteness means that the space is a countable union of measurable sets of finite measure; absolute continuity of measures, written $\nu \ll \mu$, means that $\mu (E)=0$ implies $\nu (E)=0$. The function $g=d\nu /d\mu$ is the Radon-Nikodym derivative. For a finite signed or complex measure $\nu$ of finite total variation norm of a measure, absolutely continuous with respect to a sigma-finite $\mu$, the corresponding density belongs to $L^1(\mu )$. This follows by applying the positive theorem to the positive and negative parts of the real and imaginary parts of $\nu$.

For every measure space and $1<p<\infty$, $(L^p{)}^{\ast }$ is isometrically $L^q$, where $1/p+1/q=1$. We use the complex-linear pairingWith the convention $\int h\overline{g}$, the same identification is conjugate-linear in $g$. By Hölder's inequality, ${\Phi }_g$ is a bounded linear functional with $\Vert {\Phi }_g\Vert \le \Vert g{\Vert }_q$. If $g\ne 0$, setinterpreting the numerator as zero where $g=0$. The identity $(q-1)p=q$ gives $\Vert h{\Vert }_p=1$ and ${\Phi }_g(h)=\Vert g{\Vert }_q$. ThusIt remains to represent an arbitrary $\Phi \in (L^p{)}^{\ast }$, rather than merely produce functionals from $L^q$.

We prove Lp duality on an arbitrary measure space without imposing sigma-finiteness on $\mu$. Write $C=\Vert \Phi \Vert$. For every measurable set $E$ with $\mu (E)<\infty$, define a complex measure on $E$ byIt is countably additive: for disjoint $F_j\subseteq E$, the partial sums of their indicator functions tend in $L^p$ to $1_{\bigcup F_j}$, because the measure of the omitted tail tends to zero. It is absolutely continuous with respect to $\mu {|}_E$, since indicator functions of null sets represent zero in $L^p$.

Its total variation norm of a measure is finite. For any finite measurable partition $E={\bigcup }_j{E}_j$, choose scalars $c_j$ of modulus $1$ with $c_j{\nu }_E(E_j)=|{\nu }_E(E_j)|$. ThenTaking the supremum over partitions gives the variation bound. Since $\mu {|}_E$ is finite, the Radon-Nikodym theorem supplies $g_E\in L^1(E)$ with ${\nu }_E(F)={\int }_F{g}_{E}d\mu$. By uniform approximation with simple functions,for every bounded measurable $h$ supported in $E$.

To improve $g_E$ from $L^1$ to $L^q$, test with the bounded functionIf $I_N={\int }_{E\cap \{|g_E|\le N\}}|g_E{|}^{q}d\mu$, thenThus $I_N^{1/q}\le C$ when $I_N>0$, and the same bound is trivial when it is zero. The monotone convergence theorem givesIf $E,F$ both have finite measure, the densities $g_E,g_F$ agree almost everywhere on $E\cap F$: their integrals over every measurable subset of the intersection equal the same functional value. This is uniqueness in the Radon-Nikodym theorem.

We now perform support localization of an Lp functional. SetChoose finite-measure sets $E_n$ whose displayed integrals tend to $M$, and let $D_n=E_1\cup \cdots \cup E_n$, $S={\bigcup }_n{D}_n$. If $M=0$, take $E_n=\varnothing$. Compatibility allows us to define a measurable function $g$ on $S$ by taking $g_{D_n}$ on the disjoint measurable sets $D_n\setminus D_{n-1}$, and put $g=0$ off $S$. It agrees almost everywhere with $g_{D_n}$ on every $D_n$. Moreover,Indeed each integral is at most $M$, and it is at least the integral over $E_n$, which tends to $M$.

For any finite-measure set $F\subseteq \Omega \setminus S$, compatibility on the disjoint union $D_n\cup F$ givesLetting $n\to \infty$ forces $g_F=0$ almost everywhere. For an arbitrary finite-measure set $E$, compatibility on $E\cap D_n$ and the preceding conclusion on $E\setminus S$ show that $g_E=g$ almost everywhere on $E$. Consequently $\Phi (h)=\int hgd\mu$ for every simple function supported on a finite-measure set.

Those simple functions are dense in $L^p$ even for this arbitrary measure space. To see the needed finite-support property, for $h\in L^p$ the sets $\{|h|>1/n\}$ have finite measure, bounded by $n^p\Vert h{\Vert }_p^p$. First truncate $h$ to such sets and to bounded values, then approximate by simple functions; the discarded $L^p$ integral tends to zero. Continuity of $\Phi$ and Hölder's inequality therefore extend the representation to every $h\in L^p$.

We have constructed $g\in L^q$ with $\Phi ={\Phi }_g$, and the previously proved norm identity gives $\Vert g{\Vert }_q=\Vert \Phi \Vert$. It also proves uniqueness: if ${\Phi }_g={\Phi }_{\tilde{g}}$, then $\Vert g-\tilde{g}{\Vert }_q=0$. This completes the isometric duality of Lp spaces.

The dominated sequence actually converges to zero in norm, and hence weakly. The assumptions give $|f_n{|}^p\le |f{|}^p\in L^1$ and $|f_n{|}^p\to 0$ almost everywhere. The dominated convergence theorem yieldsFor every $\Phi \in (L^p{)}^{\ast }$, $|\Phi (f_n)|\le \Vert \Phi \Vert \Vert f_n{\Vert }_p\to 0$. Therefore

On the same measurable space, absolute continuity of measures meansThe Radon-Nikodym theorem then supplies a nonnegative measurable density $Z$ with ${\mathbb{E}}_{\mathbb{P}}Z=1$ and $\tilde{\mathbb{P}}(A)={\mathbb{E}}_{\mathbb{P}}[Z{1}_A]$. The converse follows directly from this formula. Strict positivity of the Radon-Nikodym derivative gives equivalence of the two probability measures; absolute continuity of measures alone need not give equivalence.

For probability measures on $(\Omega ,\mathcal{F})$, absolute continuity of measures means preservation of null sets in one direction:Equivalently, by the Radon-Nikodym theorem, there is an $\mathcal{F}$-measurable Radon-Nikodym derivative $Z\ge 0$ with ${\mathbb{E}}_{\mathbb{P}}Z=1$ and $\tilde{\mathbb{P}}(A)={\mathbb{E}}_{\mathbb{P}}[Z{1}_A]$. The derivative may vanish, so absolute continuity of measures does not require equivalence of the two probability measures. In particular, the reverse null-set implication is not part of the definition.

No absolutely continuous probability measure can remove this nonzero drift on the whole half-line. By the strong law for Brownian motion, the eventhas $\mathbb{P}$-probability one. If ${\mathbb{P}}^{\infty }\ll \mathbb{P}$, absolute continuity of measures gives ${\mathbb{P}}^{\infty }(A)=1$. But if $W_t=B_t-\mu t$ is a Brownian motion under ${\mathbb{P}}^{\infty }$, the same strong law for Brownian motion gives $B_t/t=W_t/t+\mu \to \mu$ with ${\mathbb{P}}^{\infty }$-probability one. Since $\mu >0$, these two limiting events are disjoint, a contradiction.

This is infinite-horizon singularity of Brownian motion with constant drift. The finite-horizon densities are consistent on ${\mathcal{F}}_t$ but cannot define a globally absolutely continuous measure with the requested property. Indeed,because $t^{-1}\log Z_t\to -{\mu }^2/2$. Hence $Z$ lacks uniform integrability on $[0,\infty )$. A drifted law does exist on canonical path space, but it is mutually singular with Wiener measure on the infinite-horizon path sigma-algebra.

Let $p=dP/d\nu$ and $q=dQ/d\nu$ be Radon-Nikodym derivatives. Use the following normalizations of the Kullback-Leibler divergence, total variation distance and unnormalized Hellinger distance:We use $0\log (0/q)=0$ and assign infinite Kullback-Leibler divergence if $Pot\ll Q$, where $\ll$ denotes absolute continuity of measures. Common domination alone also suffices for the inequalities with the extended-value convention. The negative part of the logarithmic integral is integrable: on $p<q$, $p\log (q/p)\le q-p\le q$. Thus the extended-value integral defining the Kullback-Leibler divergence is well-defined. These expressions do not depend on which common dominating measure is chosen. The Kullback-Leibler divergence is generally asymmetric and therefore fails a defining property of a metric.

For completeness, the integral formula for total variation distance follows by taking $A=\{p\ge q\}$: since $\int (p-q)d\nu =0$, the positive and negative parts of $p-q$ have the same integral, and this set attains the supremum.

Put $a=\int \sqrt{pq}d\nu \le 1$, by the Cauchy-Schwarz inequality. A second application givesThe logarithmic inequality in the hint implies $\log t\le t-1$ for $t>0$. Apply it to $t=\sqrt{q/p}$ on $\{p>0\}$:Multiplying by $p$ and integrating provesIf $q=0<p$ on a set of positive $P$-measure, the right side is finite but the Kullback-Leibler divergence is infinite, so the inequality is immediate. ThusThis total variation–Hellinger–relative entropy inequality requires the stated Hellinger distance normalization for its first constant. With the alternative convention ${\bar{h}}^2=\frac{1}{2}\int (\sqrt{p}-\sqrt{q}{)}^2$, the correct comparison is $\mathrm{TV}\le \sqrt{2}\bar{h}$ and $\bar{h}\le \sqrt{\mathrm{KL}/2}$. In particular, $P=\mathrm{Bernoulli}(0.6)$ and $Q=\mathrm{Bernoulli}(0.4)$ have $\mathrm{TV}=0.2>\bar{h}\approx 0.1421$.

Here is a testing form of the Le Cam two-point lemma. Suppose two parameter values ${\theta }_0,{\theta }_1$ in a metric space have separation $\Delta =d({\theta }_0,{\theta }_1)>0$, and let $P_0,P_1$ be the corresponding observation laws. Every estimator $T$ satisfiesTo prove it, classify $T$ by the closer of the two parameters, breaking a tie in favour of one. Let $A=\{d(T,{\theta }_1)\le d(T,{\theta }_0)\}$. On $A$, the triangle inequality implies $d(T,{\theta }_0)\ge \Delta /2$, while on $A^c$ it implies $d(T,{\theta }_1)>\Delta /2$. Hence the sum of the two probabilities in the display is at leastAt least one is at least half the sum, proving the claim. It also gives a lower bound $\Delta (1-\mathrm{TV})/4$ for the maximum expected metric loss.

For absolute-error loss, or any loss given by a metric, there is a stronger expectation form. Take densities $p_0,p_1$ relative to $m=P_0+P_1$ and use the triangle inequality under their overlap:Therefore the Le Cam lower bound under absolute-error loss isBoth arguments are valid for randomized estimators as well, by adjoining the independent randomization to the observation; doing so does not change total variation distance.