Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 3 / c

Use only the first $n-1$ observations to defineThus the full noisy covariance matrix is $C_n=\left(\begin{array}{cc}A& b\\ b^T& c\end{array}\right)$ and the full cross-covariance vector is $(a^T,d{)}^T$. SetThe scalar $s$ is the Schur complement of $A$. Substituting the block inverse into the Gaussian process regression posterior variance givesConsequently,andFor interpretation, after the first $n-1$ observations the remaining covariance between $f_{\ast }$ and $f(x_n)$ is $r=d-b^T{A}^{-1}a$, while the conditional variance of the new noisy observation is $s$. Thus the reduction is the squared remaining covariance divided by the remaining observation variance.

The Schur complement can be writtenbecause the first term is a conditional variance and ${\sigma }^2>0$. Therefore $c_1=s^{-1}>0$ and the squared term is nonnegative, soEquality holds exactly when the remaining conditional covariance $k(x_n,x_{\ast })-b^T{A}^{-1}a$ is zero. In particular, the reduction need not be strict even at a previously unseen input.

Additional observations weakly decrease the posterior variance at every fixed prediction input. Apply part (ii) repeatedly to add observations one at a time. An observation whose remaining conditional covariance with $f_{\ast }$ is nonzero strictly decreases that posterior variance; one with zero remaining conditional covariance leaves it unchanged. Large observation noise reduces the information gained through the denominator $v_{n-1}(x_n)+{\sigma }^2$.

For fixed Gaussian process hyperparameters and a Gaussian likelihood, the posterior variance depends on the observation locations and noise level, but not on the observed numerical outputs. Hence the reduction holds for each possible new output, rather than only on average. The predictive variance of a future noisy observation also decreases by the same amount because its extra independent noise variance is fixed. Refitting the hyperparameters after adding data can change the covariance function; the monotonicity statement compares conditioning under one fixed model.

For $x_n=x_{\ast }$, the notation of part (i) gives $b=a$, $d=k_{\ast \ast }$, and $c=k_{\ast \ast }+{\sigma }^2$. Write $v={\mathrm{Var}}_{n-1}(f_{\ast })=k_{\ast \ast }-a^T{A}^{-1}a$. The remaining covariance is then $v$ and the new observation's remaining variance is $v+{\sigma }^2$. ThusEquivalently,The expression is valid also when $v=0$ and ${\sigma }^2>0$. For $v>0$, a noiseless observation, interpreted as the limit ${\sigma }^2\downarrow 0$, removes all remaining variance at the observed target.

The source's assertion that the target input always gives the greatest reduction needs an additional hypothesis. The reduction formula above does not require that assertion. For a counterexample, use the linear kernel $k(x,x^{\prime })=x{x}^{\prime }$ with ${\sigma }^2=1$ and target $x_{\ast }=1$. With no informative previous observations, $v=1$. This can also be arranged with $n-1=1$ by observing first at $x_1=0$, which has zero covariance with all latent values. Observing at the target gives reduction $1/2$, whereas observing at $x_n=2$ givesIndeed this Gaussian process is $f(x)=xZ$ for a standard normal random variable $Z$, so the measurement at $2$ observes the same unknown coefficient with a larger signal relative to its noise.

One sufficient extra condition is that the existing latent posterior variance be maximal at $x_{\ast }$ among all candidate inputs. To prove it, let $w=v_{n-1}(x)$ and $r=\mathrm{Cov}(f(x),f_{\ast }\mid {\mathbf{y}}_{1:n-1},\theta )$. The Cauchy-Schwarz inequality gives $r^2\le vw$. If $w\le v$, thenbecause $w\mapsto w/(w+{\sigma }^2)$ is increasing. Thus target sampling is optimal under this extra condition, but it is not implied by the paper's arbitrary-covariance function assumptions.