Past exam of the mathematics course of the University of Cambridge / 2015 / iii / Paper 36 (Topics in Statistical Theory)

Put $p=dP/d\mu$ and $q=dQ/d\mu$. Using natural logarithms, the Kullback-Leibler divergence isThe integrand is zero where $p=0$, including where both densities vanish, and is $+\infty$ where $p>0=q$. Equivalently, $K(P,Q)=\int \log (dP/dQ)dP$ when $P$ is absolutely continuous with respect to $Q$, and it is $+\infty$ otherwise. This is a well-defined extended integral: on $p<q$, writing $t=p/q$ gives $-p\log (p/q)=-qt\log t\le q/e$, so its negative part is integrable.

For Pinsker's inequality, an infinite Kullback-Leibler divergence makes the conclusion immediate. Assume it is finite. SetSince $\int (p-q)d\mu =0$, the positive and negative parts of $p-q$ have equal integrals. ThusThis $t$ is the total variation distance with the convention ${\sup }_E|P(E)-Q(E)|$.

We first prove the binary partition bound for relative entropy. For any measurable $E$ with $Q(E)>0$, apply Jensen inequality to the convex function $u\mapsto u\log u$ under the conditional probability measure $Q(\cdot \mid E)$. Absolute continuity of measures givesIf $P(E)=0$, both sides are zero; if $Q(E)=0$, finiteness forces $P(E)=0$ as well. Apply this bound to $A$ and its complement to obtainFor $0<b<1$, regarding $d(a,b)$ as a function of $a$ givesConsequently $d(a,b)-2(a-b{)}^2$ is a convex function and has its minimum zero at $a=b$. By continuity this also holds at $a=0,1$. For $b=0,1$, the finite case has $a=b$ and the remaining cases have infinite divergence. This proves the Bernoulli relative entropy lower bound in every case. Combining the bounds yields

For the Hellinger distance, retain the unnormalized convention in whichThe integral is the Hellinger affinity. In the finite-divergence case, $q/p>0$ for $P$-almost every point. Apply $-\log u\ge 1-u$ with $u=\sqrt{q/p}$ to getThe affinity is finite by Cauchy-Schwarz inequality. Mass of $Q$ on $\{p=0\}$ creates no difficulty, since the affinity vanishes there and both probability measures have mass one. The infinite-divergence case is again immediate. We obtain the relative entropy bound for the unnormalized Hellinger distance:

In the Gaussian white noise model, observe the entire pathwhere $W$ is standard Brownian motion and the deterministic drift belongs to L2 space on $[0,1]$. Equivalently, $dY(t)=f(t)dt+n^{-1/2}dW(t)$. For every deterministic $h\in L^2[0,1]$, the observed stochastic integral satisfiesThe noise $W(h)=\int hdW$ is an isonormal Gaussian process. In particular its variance is $\Vert h{\Vert }_2^2$. Gaussian white noise is interpreted through these integrals, rather than as an ordinary random function with a pointwise value at every time. No smoothness of $f$ is needed.

Here is the Gaussian maximum bound without independence. Put $M={\max }_{1\le i\le N}|g_i|$. It is integrable since it is bounded by ${\sum }_i|g_i|$. For every $\lambda >0$,The second inequality uses only the moment-generating function of the standard normal distribution, so independence is unnecessary. Jensen inequality gives $e^{\lambda EM}\le E{e}^{\lambda M}$, henceMinimizing at $\lambda =\sqrt{2\log (2N)}$ proves

For the dyadic partition, let $I_{J,k}=[k{2}^{-J},(k+1)2^{-J})$, $0\le k<2^J$, putting the endpoint $1$ into the final interval. The Haar scaling functionsform an orthonormal basis of the space $V_J$ of functions constant on each interval. The Haar wavelets can be written asFor $J\ge 1$, the constant function ${\varphi }_{0,0}=1$ together with ${\psi }_{j,k}$, $0\le j<J$, is another orthonormal basis of $V_J$. Indeed, each successive level splits a cell's two constants into their sum and difference; the number of basis elements is $1+{\sum }_{j=0}^{J-1}{2}^j=2^J$. Thus the Haar projection is the orthogonal projectionOn cell $I_{J,k}$ its value is $2^J{\int }_{I_{J,k}}f(t)dt$. The first formula also applies to $J=0$.

Estimate each coefficient by its observed stochastic integral:This is the Haar projection estimator in Gaussian white noise. The noise integrals over disjoint intervals form a Gaussian vector with zero off-diagonal covariance. The principle that uncorrelated jointly Gaussian variables are independent then makes these coefficients independent. ThereforeTaking expectations shows that ${\hat{\Pi }}_{V_J}f$ is an unbiased estimator of ${\Pi }_{V_J}f$, both coefficientwise and pointwise for the stated step-function representatives.

Only one scaling function is nonzero in each cell, so the supremum norm of the error has the exact formThe endpoint convention ensures the same identity at $1$. Apply the Gaussian maximum bound without independence with $N=2^J$ to obtain the supremum norm risk of a Haar projection estimator:The factor $2^{J/2}$ comes from cellwise scaling, while the extra square root of $J+1$ comes from taking the maximum over $2^J$ Gaussian errors.

The Hoeffding inequality states that if $X_1,\dots ,X_n$ are independent random variables with $a_i\le X_i\le b_i$ almost surely, then for $t>0$The same bound holds for the lower tail. Consequently the two-sided tail is at most twice this bound. When all ranges have length zero, the sum is deterministic and every positive tail probability is zero.

We construct an exponential packing of a Hamming cube. Choose an inclusion-maximal separated set $S$ in $\{-1,1{\}}^n$ with separation at least $r=n/8$ for the Hamming distance. It exists by a finite greedy procedure: keep adding any vertex at distance at least $r$ from all selected vertices until none remains. By maximality, every vertex is within distance strictly less than $r$ of some member of $S$. This is the principle that maximal separated sets give covers.

For a uniformly random vertex $U$ and any fixed vertex $s$, the mismatch indicators in different coordinates are independent Bernoulli random variables with success probability $1/2$. Hence $\rho (U,s)$ has the binomial distribution with parameters $n,1/2$. The lower-tail Hoeffding inequality givesThus the Hoeffding lower-tail bound for Hamming balls shows that every strict-radius ball $B_{<r}(s)$ has at most $2^n{e}^{-9n/32}$ vertices. Using a strict ball handles noninteger $n/8$ without changing the required separation.

The strict balls centered at $S$ cover the entire Hamming cube. Counting their union by the sum of their sizes yieldsThereforeIn particular this proves the assertion for every $n\ge 8$. Maximality here means that no further vertex can be added; finding a largest separated set is unnecessary.

The relevant fixed coordinate subspace is $\{(u,0):u\in {\mathbb{R}}^k\}$. Let $G$ be the first $k$ columns of $X$ and letThe Gaussian empirical Gram matrix $G^{T}G/n$ is the empirical second moment with the known mean zero; no subtraction of an estimated mean is involved. For $\theta =(u,0)$, the ratio under consideration is $|u^{T}Du|/\Vert u{\Vert }_2^2$. Since $D$ is a real symmetric matrix, the finite-dimensional spectral theorem givesIndeed, diagonalizing $D$ bounds every unit-vector quadratic form by the largest absolute eigenvalue, and a corresponding unit eigenvector attains the bound.

We first construct a unit sphere net from ball covering. Enlarge the numerical covering constant, if necessary, to $A_0\ge 1$. Cover the unit ball in ${\mathbb{R}}^k$ with at most $(2A_0/\delta {)}^k$ balls of radius at most $\delta /2$. For each such ball meeting the unit sphere, choose a point of the sphere in it and discard the others. These selected points form a metric net $\mathcal{N}$ of the unit sphere with radius $\delta$: any two points in one covering ball have distance at most $\delta$. This argument ensures that the net points have unit length even if the original covering centers did not.

Take $\delta =1/4$ and put $B=8A_0$, so $|\mathcal{N}|\le B^k$. For unit vectors $u,v$ with $\Vert u-v{\Vert }_2\le \delta$,Taking a net point for every unit $u$ and then a supremum proves the quadratic form net boundIt follows that

Fix $v\in \mathcal{N}$. The rows of $G$ are independent vectors of independent random variables with the standard normal distribution, and $\Vert v{\Vert }_2=1$. Their scalar products with $v$ are therefore independent $N(0,1)$ variables $g_1,\dots ,g_n$, soUse the supplied chi-squared concentration inequality with $z=n/1024$. Its threshold isConsequently $P(|v^{T}Dv|>1/4)\le 2e^{-n/1024}$. The union bound gives the stronger fixed-subspace estimateThis is Gaussian Gram matrix concentration on a fixed subspace.

Since $1\le k<p$, we have $p\ge 2$. Under $n\ge Ck\log p$,Choose the numerical constant $C\ge 1024(1+\log B/\log 2)$. Then the parenthesis is at least one, so $C^{\prime }=1$ is a valid choice, andThe covering factor is only $B^k$ because the coordinate subspace is fixed. The ambient dimension enters through the assumed sample-size bound.