Past exam of the mathematics course of the University of Cambridge / 2017 / iii / Paper 217 (Gaussian Processes)

Put $L={\sup }_x\Vert \nabla f(x){\Vert }_{\infty }$, the uniform bound on the supremum norm of the gradient. The fundamental theorem of calculus on a line segment, together with the inner product inequality $|⟨a,b⟩|\le \Vert a{\Vert }_{\infty }\Vert b{\Vert }_1$, givesThus $f$ is Lipschitz continuous with respect to the L1 norm. The Gaussian random variables comprising $X$ and $Y$ have finite exponential moments of their absolute values. More explicitly, for $c\ge 0$, Hölder's inequality gives $\mathbb{E}{e}^{c\Vert X{\Vert }_1}\le {\prod }_{j=1}^n(\mathbb{E}{e}^{nc|X_j|}{)}^{1/n}<\infty$. Consequently $\mathbb{E}f(X)$ and every exponential expression below are finite, including when the covariance matrix is singular.

Use the convex function $\Psi (v)=e^{\lambda |v|}$. Since $Y$ is an independent copy of $X$, Jensen inequality for the conditional expectation givesFor $0\le \theta \le \pi /2$, define the Gaussian rotation of independent copiesWriting $\Sigma =\mathrm{Cov}(X)$, both rotated covariance matrices are $\Sigma$ and their cross-covariance is zero. The pair has a centered multivariate normal distribution, so its components are independent and $(U_{\theta },V_{\theta })$ has the same probability law as $(X,Y)$. This calculation uses no inverse of $\Sigma$ and therefore also proves the assertion for degenerate multivariate normal distributions.

The path $\theta \mapsto U_{\theta }$ runs from $Y$ to $X$. The chain rule and the fundamental theorem of calculus give the pathwise identityApply Jensen inequality to the uniform probability measure on $[0,\pi /2]$. Then use Tonelli theorem and the Gaussian rotation of independent copies to obtainThis proves the Gaussian rotation interpolation inequality in the required form:The stated almost-sure domination now gives an exponential moment of an absolute standard normal variable bound. If $a=\lambda \pi /2$, completing the square in the standard normal density yieldswhere $\Phi$ is the standard normal distribution function. Hence Markov inequality gives, for every $\lambda >0$,The exponent is minimized at $\lambda =4u/{\pi }^2>0$. Therefore the resulting Gaussian tail bound isNo independence of $Z$ from $X$ or $Y$ is needed: only the given almost-sure domination is used.

Construct a mean-square derivative of a Gaussian process on the original probability space before choosing its continuous modification. The real covariance function satisfies $K(h)=K(-h)$, because covariance is symmetric. Three-times differentiability implies that $K^{\prime \prime }$ is continuous everywhere and differentiable at zero. Since $K^{\prime \prime }$ is an even function,This uses differentiability of $K^{\prime \prime }$ at zero, not a bound on $K^{\prime \prime \prime }$ throughout a neighborhood.

Let $D_{h}X(t)=(X(t+h)-X(t))/h$ for $h\ne 0$. Twice applying the fundamental theorem of calculus to the covariance function givesOriented integrals make this identity valid for negative as well as positive $h,k$. For a fixed $t$, the right side tends to $-K^{\prime \prime }(0)$ as $h,k\to 0$. In particular,By completeness of L2 space, the mean-square derivative of a Gaussian processexists. Finite vectors consisting of values of $X$ and $Y$ have a centered multivariate normal distribution: approximate the $Y$ coordinates by difference quotients and use mean-square convergence and characteristic functions. Thus $Y$ is a centered Gaussian process jointly constructed with $X$, withIn particular $-K^{\prime \prime }(0)\ge 0$, and $-K^{\prime \prime }$ is a positive-semidefinite kernel because it is an actual covariance function. Merely constructing an unrelated Gaussian process with that covariance function would not yet construct a modification of a stochastic process of $X$.

The increment variance of $Y$ isFor small $|t-s|$, differentiability of $K^{\prime \prime }$ at zero bounds this by $C|t-s|$. For the remaining distances in $[0,1]$, continuity of $K^{\prime \prime }$ enlarges $C$ to give the same bound. The fourth moment of a centered normal distribution is three times its variance squared, soThe one-dimensional Kolmogorov continuity theorem states that $\mathbb{E}|U(t)-U(s){|}^p\le C|t-s{|}^{1+\eta }$ with $p,\eta >0$ gives a continuous modification, with sample-path Hölder continuity of every order less than $\eta /p$. Apply it here with $p=4,\eta =1$. We obtain a continuous modification $\tilde{Y}$ on $[0,1]$. On its null exceptional event, set the entire path equal to zero.

Define, on this same probability space,The path integral exists because its integrand is a continuous function. It agrees in L2 space with the integral of $Y$: Riemann sums converge in L2 space because $Y$ is mean-square continuous; sums of $\tilde{Y}$ have the same values almost surely at each finite collection of mesh points and converge pathwise to the integral. These two limits therefore agree almost surely for each fixed $t$. Equivalently, the integral is the Bochner integral of the L2 space-valued map $s\mapsto Y(s)$.

To verify the modification of a stochastic process property directly, put $A_t=X(t)-X(0)$ and $I_t={\int }_0^{t}Y(s)ds$. The preceding covariance identities and the fundamental theorem of calculus giveTherefore $\mathbb{E}(A_t-I_t{)}^2=0$ and $\tilde{X}(t)=X(t)$ almost surely for every fixed $t$. This is also the mean-square fundamental theorem of calculus in this stationary setting. If a version on all of $\mathbb{R}$ is desired, retain the original $X(t)$ outside $[0,1]$; equality at every fixed parameter still holds.

The ordinary fundamental theorem of calculus now gives the simultaneous pathwise conclusionThus the requested differentiable modification of a stationary Gaussian process exists. Integrability of $K$ is compatible with the hypotheses but is unnecessary for this direct argument. It also avoids any assumption that a third absolute moment of a spectral density follows from three-times differentiability.

The Cameron-Martin space of a Gaussian random variable in a Banach space is most conveniently defined using the first Gaussian chaos. Work over the real separable Banach space $B$, with continuous dual space $B^{\ast }$. For a centered Gaussian random variable in a Banach space, defineThe set inside the closure is already a linear subspace. Every $g\in \mathcal{G}$ is a centered Gaussian random variable. The Bochner integral defining $S$ exists: Fernique's theorem states that for some $a>0$, $\mathbb{E}{e}^{a\Vert X{\Vert }_B^2}<\infty$, and then Cauchy-Schwarz inequality gives $\mathbb{E}\Vert Xg{\Vert }_B<\infty$.

The map $S$ is injective. Indeed, $Sg=0$ implies $\mathbb{E}[g\ell (X)]=\ell (Sg)=0$ for all $\ell \in B^{\ast }$. Density of these observations in $\mathcal{G}$ then implies $\mathbb{E}{g}^2=0$. Define the Reproducing kernel Hilbert space byThe injectivity makes the inner product unambiguous, and the completeness of $\mathcal{G}$ makes $H$ a Hilbert space. The embedding into $B$ is continuous sinceFor $\ell \in B^{\ast }$ put $Q\ell =S(\ell (X))$. The reproducing property isThis defines the Banach-space Gaussian RKHS even for a degenerate Gaussian measure.

For $h=Sg\in H$, write $\hat{h}(X)=g$. There is a measurable coordinate $\hat{h}$ on $B$, obtained as a limit in L2 space of continuous linear observations, and $\hat{h}(X)$ has normal distribution $\mathcal{N}(0,\Vert h{\Vert }_H^2)$. The Cameron-Martin theorem for a Gaussian measure, in its real separable Banach space form, states that if $\mu$ is the probability law of $X$, then the probability law ${\mu }_h$ of $X+h$ is an equivalent probability measure to $\mu$ precisely when $h\in H$. For such a shift,For $h\notin H$ the two laws are mutually singular measures. The coordinate $\hat{h}$ need not be a continuous functional on $B$, and the sample $X$ need not belong to $H$.

Apply this theorem to the shift $-h$. With $a=\Vert h{\Vert }_H^2/2$, it givesChoose the measurable coordinate so that $\hat{h}(-x)=-\hat{h}(x)$ on a symmetric set of full $\mu$-measure. This is possible by taking an almost-surely convergent subsequence of the approximating linear observations and intersecting its convergence set with its negative. Alternatively the joint law of $(X,\hat{h}(X))$ is invariant under simultaneous negation, directly from those observations and their mean-square convergence. Since $C=-C$ and the centered Gaussian measure is symmetric, the two exponential integrals with signs $+$ and $-$ agree. Consequentlybecause $\cosh v\ge 1$. The symmetric Gaussian translation lower bound is thereforeOnly symmetry and Borel measurability of $C$ are required; convexity is not needed.

There is one degenerate flaw in the last printed claim. The separable Banach space $B=\{0\}$ with $X=0$ has $H=\{0\}$ dense in $B$, and all positive-radius balls have probability one. Nevertheless $\Phi (t)=1$ for every $t>0$. Thus the statement needs the additional hypothesis $B\ne \{0\}$. If the course convention excludes the zero space, this hypothesis is already implicit.

Under this necessary hypothesis, density of $H$ implies that $H$ contains a nonzero vector. Given $0<a<b$, scale that vector to obtain $h\in H$ with $\Vert h{\Vert }_B=(a+b)/2$, and choose $0<\delta <(b-a)/2$. The triangle inequality givesThe centered closed ball $C_{\delta }=\{x:\Vert x{\Vert }_B\le \delta \}$ is a symmetric Borel set. Apply the symmetric Gaussian translation lower bound to getHence the Gaussian norm distribution is strictly increasing when $B$ is nonzero. This proves the strict increase of a Gaussian norm distribution without assuming that its distribution has a density or that spheres have zero probability.

The equal-variance form of Slepian's lemma is the following. If $U,V$ are centered vectors with multivariate normal distributions, $\mathbb{E}{U}_i^2=\mathbb{E}{V}_i^2$ for every $i$, and $\mathbb{E}{U}_i{U}_j\ge \mathbb{E}{V}_i{V}_j$ whenever $i\ne j$, thenfor every threshold vector $(u_i)$. In particular $\mathbb{P}({\max }_i{U}_i\le u)\ge \mathbb{P}({\max }_i{V}_i\le u)$. No nonsingularity of the covariance matrices is required.

Here the pointwise variances $2t^{2H}$ and $2t^{2H^{\prime }}$ differ. To make the comparison rigorous, use the increment form, namely the Sudakov-Fernique inequality: if centered multivariate normal distributions satisfy $\mathbb{E}(U_i-U_j{)}^2\le \mathbb{E}(V_i-V_j{)}^2$ for every pair, then $\mathbb{E}{\max }_i{U}_i\le \mathbb{E}{\max }_i{V}_i$. Unlike the preceding form of Slepian's lemma, this statement has no equal-pointwise-variance hypothesis.

For completeness, the increment form follows by Gaussian integration by parts. Make $U,V$ independent, define $Z_r=\sqrt{1-r}U+\sqrt{r}V$ for $0<r<1$, and letThe $p_i$ are softmax function weights, with $p_i\ge 0$ and ${\sum }_i{p}_i=1$. Direct differentiation gives ${\partial }_i{F}_{\tau }=p_i$ and ${\partial }_{ij}{F}_{\tau }=\tau (p_i{1}_{i=j}-p_i{p}_j)$. The Gaussian integration by parts identity therefore yieldsThe last summands are nonnegative because $D_{ii}+D_{jj}-2D_{ij}=\mathbb{E}(V_i-V_j{)}^2-\mathbb{E}(U_i-U_j{)}^2$. Sinceintegration in $r$ and the limit $\tau \to \infty$ prove the Sudakov-Fernique inequality. All derivatives needed here are bounded for a fixed $\tau$. Singular covariance matrices can be handled by adding small independent centered normal noise with the same variance to both vectors, proving the identity for the nonsingular vectors, and letting the noise variance decrease to zero.

Use the printed normalization of fractional Brownian motion, which omits the conventional factor $1/2$. Its increment variance isFor $s,t\in [0,1]$ and $H>H^{\prime }$, $|t-s{|}^{2H}\le |t-s{|}^{2H^{\prime }}$. Enumerate $T$ by increasing finite subsets $T_n$ whose union is $T$, with $0\in T_n$. The Sudakov-Fernique inequality givesBoth Gaussian processes vanish at zero almost surely. Thus these maxima are nonnegative and increase with $n$; monotone convergence theorem givesTheir continuous modifications agree with the original Gaussian processes simultaneously on the countable set $T$, after discarding a single null event. Continuity and density of the rational parameters make each rational supremum equal to the supremum over $[0,1]$ for these modifications.

The expected suprema are finite. Indeed, the canonical distance of $Y_{H^{\prime }}$ is $d(s,t)=\sqrt{2}|t-s{|}^{H^{\prime }}$. A Euclidean grid supplies a closed-ball metric covering number bound $N(\epsilon )\le 1+\lceil (\sqrt{2}/\epsilon {)}^{1/H^{\prime }}\rceil$ for $0<\epsilon \le \sqrt{2}$. The Dudley entropy integral bound for an anchored centered separable Gaussian process isThe integral is finite since its integrand grows at most like a constant times $\sqrt{1+\log (1/\epsilon )}$ near zero. The same argument applies to $X_H$.

For any real path vanishing at zero,A centered Gaussian process has the same probability law as its negative. Combining this symmetry with the increment comparison gives the stronger boundIn particular the requested comparison isThe common normalization factor $\sqrt{2}$ affects both sides equally. The comparison is valid for every $0<H^{\prime }<H<1$, including the equality cases $s=t$ or $|s-t|=1$ in the increment bounds.