Past exam of the mathematics course of the University of Cambridge / 2024 / iii / Paper 207 / 3

Let $K\in {\mathbb{R}}^{n\times n}$ have entries $K_{ij}=\kappa (x_i,x_j)$, let $k_{\ast }\in {\mathbb{R}}^n$ have entries $(k_{\ast }{)}_i=\kappa (x_i,x_{\ast })$, and let $k_{\ast \ast }=\kappa (x_{\ast },x_{\ast })$. The Gaussian process prior and independent Gaussian noise implyApplying the conditional multivariate normal distribution gives the Gaussian process regression posteriorwhere

Set $x_{\ast }=0.5$ in the Gaussian process regression posterior and compute $m_{\ast }$ and $v_{\ast }$ as in part a. Standardization of a normal random variable then giveswhere $\Phi$ is the standard normal cumulative distribution function.

Let $N_m=|\{i:c_i=m\}|$. The label likelihood for the mixture weights is proportional to ${\prod }_{m=1}^3{\pi }_m^{N_m}$. Multiplication by the $\mathrm{Dirichlet}({\alpha }_1,{\alpha }_2,{\alpha }_3)$ density and Dirichlet-multinomial conjugacy givesConditional on the labels, the observations and component means provide no further information about $\pi$.

For component $m$, stack its $N_m$ assigned observations as $y_1^{(m)},\dots ,y_{N_m}^{(m)}\in {\mathbb{R}}^n$. The prior is $f_m\sim N_n(0,K)$ and each assigned vector is conditionally $N_n(f_m,{\sigma }^2{I}_n)$. Normal-normal conjugacy giveswhereIf $N_m=0$, this reduces to the prior $N_n(0,K)$.

Bayes theorem turns the categorical distribution prior probabilities and the component multivariate normal densities intoThese probabilities define the label update in the Gibbs sampler.

A Dirichlet process mixture model avoids fixing the number of occupied functions. Let $G_0=N_n(0,K)$ be the finite-dimensional Gaussian process law on the common input grid and specifyA draw from a Dirichlet process is almost surely discrete, so several $f_i$ coincide and thereby form clusters. The number of occupied clusters is random and can grow with the data.