Mathematics and statistics course of the University of Oxford (BA, MMath)

Official course description.

Common first-year syllabus and assessment with Year 1 of the mathematics course of the University of Oxford.

Common mathematics core; Probability course of the University of Oxford and Statistics course of the University of Oxford are compulsory. A12 simulation and statistical programming is optional.

SB1 applied and computational statistics is compulsory; students must also take at least two units from SB2 and SB3.

Compulsory double unit: SB1.1 applied statistics and SB1.2 computational statistics.

Part B option listed for 2026–27; applications of probability.

Part B option listed for 2026–27; a course in machine learning.

Statistics options and a compulsory statistics-project dissertation, with mathematics options available under the course rules.

Source: Oxford Trinity Term 2025, SC4, A16047H1, parent PDF pages 11–16. The six-page paper was examined on 11 June 2025. Solutions cover all three questions; question statements are omitted.

The marginal likelihood integrates the likelihood function against the prior density:It is the normalizing constant in Bayes' theorem. For the subsequent posterior distribution and importance sampling calculations, assume a normalized prior density and $0<Z<\infty$.

Write $h(\theta )=\log p(\mathcal{D}\mid \theta )+\log p(\theta )$. Choose an interior maximum a posteriori estimate, a globally maximizing posterior modeSuppose $h$ is twice continuously differentiable near $\mu$ and its negative Hessian matrix $A=-{\nabla }^{2}h(\mu )$ is a positive-definite matrix. The second-order Taylor expansion isExponentiating and normalizing this local approximation gives the Laplace approximationThus the approximate mean is the posterior mode, and the approximate covariance matrix is the inverse local curvature. This normal approximation is useful when most posterior probability is concentrated around this mode. A boundary mode, singular curvature, or several substantial modes requires a modified approximation.

Evaluate Bayes' theorem at the posterior mode $\mu$:The multivariate normal density in the Laplace approximation has height $(2\pi {)}^{-d/2}|\Sigma {|}^{-1/2}$ at its mean. ThereforeEquivalently, the approximate log marginal likelihood isThe determinant factor accounts for the volume of the locally plausible parameter region, which the peak likelihood alone does not measure.

Put $r(\theta )=p(\mathcal{D}\mid \theta )p(\theta )$ and $Z=\int r(\theta )d\theta$. The support condition for importance sampling is essential: $q(\theta )>0$ almost everywhere that $r(\theta )>0$. For one draw $\theta \sim q$, the importance weight $W=r(\theta )/q(\theta )$ satisfiesBy linearity of expectation,This proves that it is an unbiased estimator. If the proposal omits a region of positive target mass, the expected value instead misses that region's contribution; the unconditional unbiasedness claim needs the stated support condition for importance sampling.

With $r$, $Z$, and $W$ as above, independence of the draws gives the variance of a sample mean:This formula allows infinite variance. To identify the optimum, write $\pi (\theta )=p(\theta \mid \mathcal{D})=r(\theta )/Z$. ThenThe last equality uses $\int \pi =\int q=1$. Its nonnegative integrand provesIndeed, all importance weights then equal $Z$. This is a formal optimum: evaluating the normalized posterior density generally already requires the unknown marginal likelihood.

Poor coverage of regions with substantial posterior mass produces large weights. In particular, excessively light proposal tails, a proposal that is too narrow, or a missed mode can make $\int r^2/q$ enormous or infinite. A proposal that is unnecessarily diffuse can also waste nearly all draws on negligible posterior probability. Omitting target support altogether violates unbiasedness, rather than merely increasing variance.

For reciprocal importance sampling, use the same $r$ and $Z$ but draw from the posterior density $\pi =r/Z$. Here the support requirement is reversed: the normalized density $q$ must vanish almost everywhere that $r=0$. Thus $q$ may be confined to a subset of the posterior distribution's support. For $V=q(\theta )/r(\theta )$,Consequently,It is the estimate of the reciprocal that is an unbiased estimator. Taking its reciprocal does not generally produce an unbiased estimator of $Z$: inversion is nonlinear. In fact, when the average is strictly positive and the relevant expectation exists, Jensen's inequality gives $\mathbb{E}{\hat{p}}_2\ge Z$.

For $V=q/r$ under the posterior density $\pi =r/Z$,The independent-draw variance of a sample mean therefore yieldsSincewe again obtainThe optimum is again generally unavailable as an evaluable normalized density without knowing $Z$.

A proposal with too much mass where the posterior density is tiny makes reciprocal weights unstable. In particular, proposal tails that are too heavy relative to the posterior distribution can make $\int q^2/\pi$ infinite. This contrasts with ordinary importance sampling, whose dangerous denominator is $q$. Choosing a normalized $q$ restricted to a well-covered posterior region is allowed, although a very small region can make most draws contribute zero and increase variance. Taking $q=p(\theta )$ gives the Harmonic mean estimator of Bayesian model evidence, illustrating why very small likelihood values can cause severe instability.

Use the factorization $p(\theta ,\mathcal{D})=p(\theta \mid \mathcal{D})p(\mathcal{D})$ inside the evidence lower bound:HenceThe marginal likelihood is independent of the variational parameter $\varphi$, so maximizing the evidence lower bound is equivalent to minimizing the Kullback-Leibler divergence. Nonnegativity of the Kullback-Leibler divergence also gives $\mathcal{L}(\varphi )\le \log p(\mathcal{D})$.

The independent coordinates make the logarithm of the probability density ratio a sum. For coordinate $i$, its contribution to the Kullback-Leibler divergence isbecause ${\mathbb{E}}_q{\theta }_i^2={\varphi }^2$. Thus, retaining all constants,Here ${\sigma }_1,{\sigma }_2>0$ and $\varphi >0$, so both normal densities are nondegenerate.

The printed minimization of the evidence lower bound has no solution. To distinguish that literal result from the usual variational inference optimum, put $S={\sigma }_1^{-2}+{\sigma }_2^{-2}>0$. The preceding Kullback-Leibler divergence satisfiesIts unique global minimum occurs at ${\varphi }^2=2/S$. Since $\mathcal{L}=\log p(\mathcal{D})-\mathcal{J}$, this is the unique global maximum of the evidence lower bound:The optimal common variance is the harmonic mean of the two posterior variances. However, $\mathcal{J}(\varphi )\to +\infty$ as either $\varphi \downarrow 0$ or $\varphi \to \infty$. ThereforeThe source PDF confirms the minimization wording; replacing it by maximization would give the finite value above, consistently with part (i).

In this setting a positive-semidefinite kernel is an inner-product similarity: there exist a real Hilbert space $\mathcal{H}$ and a feature map $\phi :\mathcal{X}\to \mathcal{H}$ such thatThe feature map may be infinite-dimensional. Equivalently, $k$ is symmetric and every finite kernel matrix is a positive semidefinite matrix. The equivalence follows by forming the canonical feature space of a positive-semidefinite kernel and taking its Hilbert space completion.

For every $m\ge 1$, every choice $x_1,\dots ,x_m\in \mathcal{X}$, and every coefficient vector $\alpha \in {\mathbb{R}}^m$, the required condition isThus ${\alpha }^{T}K\alpha \ge 0$ for the kernel matrix $K$. For a feature map, this follows fromIn kernel methods, the term positive-definite kernel commonly uses this non-strict inequality. Strict positive definiteness is the stronger requirement that the expression be positive for distinct inputs and every nonzero $\alpha$; it is not required here.

The closure property of positive-semidefinite kernels gives three constructions, each seen directly from the feature maps ${\phi }_1,{\phi }_2$.

Another useful consequence is pullback: if $h:\mathcal{X}\to \mathcal{Z}$ and $k_0$ is a positive-semidefinite kernel on $\mathcal{Z}$, then $k_0(h(x),h(x^{\prime }))$ has feature map ${\phi }_0\circ h$.

Take the real L2 space $\mathcal{H}=L^2((0,\infty ),dt)$ and the feature mapEach indicator function belongs to $\mathcal{H}$ because its squared L2 norm is $x<\infty$. Their inner product is the length of the overlapping intervals:This realizes $k_1$ as a positive-semidefinite kernel; it is also the Brownian covariance kernel.

Choose two inputs $0<a<b$. Their proposed kernel matrix isThe coefficient vector $\alpha =(1,-1{)}^T$ givesThus $K$ is not a positive semidefinite matrix, andEquivalently, its determinant $ab-b^2=b(a-b)$ is negative.

Reciprocation reverses the order of positive numbers, soThis is the pullback of the positive-semidefinite kernel from part (i) by $x\mapsto 1/x$. Explicitly, the feature map into the same L2 space isHence $k_3$ is a positive-semidefinite kernel, with $\Vert \psi (x){\Vert }_2^2=1/x<\infty$ for every allowed input.

Let $J(t)$ be the $n\times d$ Jacobian matrix with row $J_i(t)={\nabla }_{\theta }f(\theta (t),x_i{)}^T$. Differentiating the squared-error loss function givesThe parameter gradient flow is consequently $\dot{\theta }=J^T(y-u)$. The chain rule then giveswhere, explicitly,It is a Gram matrix and therefore a positive semidefinite matrix: $v^{T}H(t)v=\Vert J(t{)}^{T}v{\Vert }_2^2\ge 0$. These derivatives are understood wherever the network is differentiable; for piecewise differentiable activations the same calculation applies along differentiable portions of the trajectory.

Under the stated infinite-width scaling and freezing assumptions, the deterministic limiting kernel matrix has entriesFor a sequence of finite-width networks, this expectation is interpreted in the stated width limit. The resulting neural tangent kernel is a positive-semidefinite kernel, since every finite coefficient vector $a$ gives $a^T{H}^{\ast }a=\mathbb{E}\Vert {\sum }_i{a}_i{\nabla }_{\theta }f(\theta (0),x_i){\Vert }_2^2\ge 0$.

The prediction gradient flow is linear:The matrix exponential solves the constant-coefficient system because the residual $u-y$ satisfies $\frac{d}{dt}(u-y)=-H^{\ast }(u-y)$. Each positive-eigenvalue component decays exponentially; a component in the null space of $H^{\ast }$ stays at its initial residual. Thus $u(t)\to y$ for every initial prediction if $H^{\ast }$ is a positive-definite matrix. The limiting kernel matrix being deterministic does not require $u(0)$ to be deterministic.

For $h$ in the Reproducing kernel Hilbert space ${\mathcal{H}}_k$, take the directional derivative:The reproducing property says $h(x_i)=⟨k(x_i,\cdot ),h{⟩}_{{\mathcal{H}}_k}$. HenceEach kernel section belongs to the Reproducing kernel Hilbert space, so this finite sum is an admissible element. Uniqueness of a vector with these inner products gives the functional gradient

Evaluating the negative functional gradient at $x_j$ yieldsThus the evaluation vector $v(t)=(g_t(x_1),\dots ,g_t(x_n){)}^T$ followsThis is exactly the limiting prediction dynamics in part (ii). If $v(0)=u(0)$, subtracting the two systems gives $\frac{d}{dt}(v-u)=-H^{\ast }(v-u)$ with zero initial value, so $v(t)=u(t)$ for every $t$.

An infinitely wide neural network in this regime follows the same training-prediction dynamics as kernel regression with its neural tangent kernel and matching initial predictions. For this squared-error loss function, the relevant algorithm is gradient flow in the Reproducing kernel Hilbert space. Matching predictions at arbitrary additional inputs also requires the corresponding cross-input neural tangent kernel to freeze and the initial predictions there to match. A finite-width neural network can change its feature map during training, so the equivalence depends on the assumptions rather than holding for every architecture and parameterization.

A transformer block has two principal sublayers: multi-head attention and a positionwise feedforward neural network. Each sublayer has a residual connection and layer normalization. To make the components explicit, let $X\in {\mathbb{R}}^{L\times d_{\mathrm{model}}}$ contain the representations of $L$ sequence positions. For head $h$, learned projections formThe scaled dot-product attention output iswhere the softmax function acts along each row and $B$ is an optional attention mask. For causal self-attention, disallowed future positions receive $-\infty$ before the softmax function, giving zero attention weights. In self-attention, queries, keys, and values all come from the same input sequence. The multi-head attention output concatenates the heads and projects them:A standard post-normalization transformer block can then be writtenHere layer normalization normalizes the feature coordinates of each position separately, within each example, and applies learned scales and shifts. The positionwise feedforward neural network applies the same two affine maps and activation function at every position, for exampleThus attention mixes information between positions, while the feedforward sublayer transforms each position's features. The residual connections support propagation through a deep stack, and dropout is often used during training. Pre-normalization variants move layer normalization before the sublayer. A model's positional encoding supplies order information to the sequence representation or attention mechanism; a decoder in an encoder-decoder model also has a cross-attention sublayer between its self-attention and feedforward neural network sublayers.

A recurrent neural network processes positions through a recurrence such as $h_t=F(h_{t-1},x_t)$. Computing $h_t$ requires $h_{t-1}$, so the standard computation is sequential across sequence positions, although different examples in a batch can be processed in parallel. Information from position $s$ to $t$ must pass through $t-s$ recurrent steps, and backpropagated gradients involve products of that many step Jacobian matrices. This creates vanishing gradients or exploding gradients and can make long-range dependencies difficult to learn. Gating can mitigate these difficulties while retaining the recurrent dependence.

In a transformer block, all queries, keys, and values for an available input sequence can be formed in parallel. The self-attention weights are computed by matrix multiplication, so positions within one layer can be processed together; layers themselves remain sequential. With unrestricted self-attention, one position can directly receive information from any other position in one layer, giving a constant-length route across distant positions rather than one proportional to their separation. With causal self-attention, there is the same direct route to any permitted earlier position.

A transformer model supports parallel training across known sequence positions and short routes for long-range dependencies. Even a causal model can compute all training positions together when the entire training sequence is supplied and the attention mask excludes future positions. However, ordinary autoregressive generation still generates one new token at a time. Full self-attention also constructs $L^2$ pairwise scores, with quadratic time and storage in sequence length for the direct implementation; a standard recurrent neural network has linear sequential work in sequence length at fixed hidden width. Neither architecture guarantees that a useful long-range dependency will actually be learned.

Gaussian process regression supplies a predictive distribution and uncertainty estimates. A Gaussian process regression posterior gives both a posterior mean and a posterior variance, allowing credible intervals for the latent function and a posterior predictive distribution for future noisy observations. The kernel ridge regression estimator by itself specifies a point estimate.

With the same positive-semidefinite kernel and matching regularization, the Gaussian process regression posterior mean equals the kernel ridge regression prediction. Thus the main additional benefit is the explicit probability distribution of predictions, conditional on the chosen Gaussian process model and its hyperparameters, rather than an automatic improvement in point-prediction accuracy.

Condition throughout on the training inputs and $x_{\ast }$. Interpret $\theta$ here as the fixed covariance function and noise hyperparameters. Assume observation errors ${ϵ}_i$ are independent standard normal random variables, independent of the Gaussian process $f$, and writeThe independent-error assumption is the standard Gaussian process regression interpretation of the observation model; the single-observation marginal noise law alone would not specify cross-observation noise correlations. We take ${\sigma }^2>0$ so that $C$ is a positive-definite matrix. More generally the inverse formulas below apply whenever the indicated noisy covariance matrices are invertible.

The Gaussian process prior makes $(f_{\ast },\mathbf{f})$ a Gaussian random vector. Adding the independent Gaussian noise vector to $\mathbf{f}$ preserves a multivariate normal distribution. Its mean is zero, its observation covariance matrix is $C$, and $\mathrm{Cov}(f_{\ast },y_i)=k(x_{\ast },x_i)$ because the noise is independent of $f_{\ast }$. ThereforeA degenerate latent variance is allowed; the displayed multivariate normal distribution then denotes a Gaussian law even if it has no full-dimensional probability density.

The conditional multivariate normal distribution giveswhereFor a direct justification, form $R=f_{\ast }-k_{\ast }^T{C}^{-1}\mathbf{y}$. It is jointly Gaussian with $\mathbf{y}$, has zero mean, and satisfiesThe independence of uncorrelated jointly normal variables therefore makes $R$ independent of $\mathbf{y}$. Conditional on $\mathbf{y}$, only the displayed affine shift remains, proving the result. This is the latent-function posterior variance; a new observation $y_{\ast }=f_{\ast }+\sigma {ϵ}_{\ast }$ with fresh independent noise has predictive variance $v_n(x_{\ast })+{\sigma }^2$.

Use only the first $n-1$ observations to defineThus the full noisy covariance matrix is $C_n=\left(\begin{array}{cc}A& b\\ b^T& c\end{array}\right)$ and the full cross-covariance vector is $(a^T,d{)}^T$. SetThe scalar $s$ is the Schur complement of $A$. Substituting the block inverse into the Gaussian process regression posterior variance givesConsequently,andFor interpretation, after the first $n-1$ observations the remaining covariance between $f_{\ast }$ and $f(x_n)$ is $r=d-b^T{A}^{-1}a$, while the conditional variance of the new noisy observation is $s$. Thus the reduction is the squared remaining covariance divided by the remaining observation variance.

The Schur complement can be writtenbecause the first term is a conditional variance and ${\sigma }^2>0$. Therefore $c_1=s^{-1}>0$ and the squared term is nonnegative, soEquality holds exactly when the remaining conditional covariance $k(x_n,x_{\ast })-b^T{A}^{-1}a$ is zero. In particular, the reduction need not be strict even at a previously unseen input.

Additional observations weakly decrease the posterior variance at every fixed prediction input. Apply part (ii) repeatedly to add observations one at a time. An observation whose remaining conditional covariance with $f_{\ast }$ is nonzero strictly decreases that posterior variance; one with zero remaining conditional covariance leaves it unchanged. Large observation noise reduces the information gained through the denominator $v_{n-1}(x_n)+{\sigma }^2$.

For fixed Gaussian process hyperparameters and a Gaussian likelihood, the posterior variance depends on the observation locations and noise level, but not on the observed numerical outputs. Hence the reduction holds for each possible new output, rather than only on average. The predictive variance of a future noisy observation also decreases by the same amount because its extra independent noise variance is fixed. Refitting the hyperparameters after adding data can change the covariance function; the monotonicity statement compares conditioning under one fixed model.

For $x_n=x_{\ast }$, the notation of part (i) gives $b=a$, $d=k_{\ast \ast }$, and $c=k_{\ast \ast }+{\sigma }^2$. Write $v={\mathrm{Var}}_{n-1}(f_{\ast })=k_{\ast \ast }-a^T{A}^{-1}a$. The remaining covariance is then $v$ and the new observation's remaining variance is $v+{\sigma }^2$. ThusEquivalently,The expression is valid also when $v=0$ and ${\sigma }^2>0$. For $v>0$, a noiseless observation, interpreted as the limit ${\sigma }^2\downarrow 0$, removes all remaining variance at the observed target.

The source's assertion that the target input always gives the greatest reduction needs an additional hypothesis. The reduction formula above does not require that assertion. For a counterexample, use the linear kernel $k(x,x^{\prime })=x{x}^{\prime }$ with ${\sigma }^2=1$ and target $x_{\ast }=1$. With no informative previous observations, $v=1$. This can also be arranged with $n-1=1$ by observing first at $x_1=0$, which has zero covariance with all latent values. Observing at the target gives reduction $1/2$, whereas observing at $x_n=2$ givesIndeed this Gaussian process is $f(x)=xZ$ for a standard normal random variable $Z$, so the measurement at $2$ observes the same unknown coefficient with a larger signal relative to its noise.

One sufficient extra condition is that the existing latent posterior variance be maximal at $x_{\ast }$ among all candidate inputs. To prove it, let $w=v_{n-1}(x)$ and $r=\mathrm{Cov}(f(x),f_{\ast }\mid {\mathbf{y}}_{1:n-1},\theta )$. The Cauchy-Schwarz inequality gives $r^2\le vw$. If $w\le v$, thenbecause $w\mapsto w/(w+{\sigma }^2)$ is increasing. Thus target sampling is optimal under this extra condition, but it is not implied by the paper's arbitrary-covariance function assumptions.