Past exam of the mathematics course of the University of Cambridge / 2014 / iii / Paper 36 / 3

Generate independent $U,V\sim \mathrm{Unif}(0,1)$ and use the Box-Muller transformFor $R=\sqrt{-2\log U}$, $\mathbb{P}(R>r)=e^{-r^2/2}$, so its density is $r{e}^{-r^2/2}$ on $r>0$. The angle $2\pi V$ is uniform on $[0,2\pi )$ and independent of $R$. Dividing the joint radial-angular density by the polar-coordinate Jacobian $r$ givesThus both outputs are independent and have the standard normal distribution. Independent pairs of uniforms give further independent normal observations. The null event $U=0$ is excluded in implementation so the logarithm is finite.

For fixed parameters $(a,b)$, every new error is independent of the previous observations. Put ${\phi }_0(z)=(2\pi {)}^{-1/2}{e}^{-z^2/2}$. Since $X_0=X_1=0$, the first three conditional densities areandIn general,Its density is ${\phi }_0(x_{t+2}-a{x}_{t+1}-b{x}_t)$. The recursion is interpreted from $t=0$, as required to define $X_2$ from the two given initial values. These are parameter-conditional sampling distributions; integrating over the prior would instead give predictive mixtures.

Use the chain rule for conditional densities. With $x_0=x_1=0$, the likelihood function of the nondegenerate observations isThe first residual is $x_2$ and carries no parameter information. There is no ordinary joint Lebesgue density for $X_1,\dots ,X_n$ because $X_1=0$ deterministically; this expression is the conditional likelihood function given the fixed initial values, or the likelihood function on $x_2,\dots ,x_n$. The initial point mass is parameter independent and has no effect on inference. This is the conditional likelihood of an initialized Gaussian AR(2) process.

Define the sufficient data sums, all over $0\le t\le n-2$,Multiplying the likelihood function by the independent standard-normal priors and collecting the quadratic terms givesLet $\Lambda =\left(\begin{array}{cc}A& C\\ C& D\end{array}\right)$ and $\Delta =AD-C^2$. This precision matrix is $I+\sum v_t{v}_t^T$, where $v_t=(x_{t+1},x_t{)}^T$, so it is positive definite even for a short or singular design. Completing the square provesThe fully normalized posterior density isThis is Gaussian conjugacy for an initialized AR(2) regression.

Completing each one-dimensional square, or using the supplied conditional-normal identity, givesWhen $n=2$, all regressor sums vanish and the posterior remains the independent standard-normal prior. No stationary-parameter restriction is imposed: the specified prior is on all of ${\mathbb{R}}^2$, and the finite initialized chain is defined for all $(a,b)$.

Start from any $(a^{(0)},b^{(0)})$. For independent draws $G_{1,k},G_{2,k}$ from the standard normal distribution at each sweep, the Gibbs sampler can be implemented asThe second update must use the newly drawn first coordinate. The transform in part (a) supplies the needed independent Gaussian draws. Each conditional update preserves the joint posterior, so their composition does too.

Convergence is particularly transparent here. Centering at the posterior mean givesPositive definiteness gives $C^2/(AD)<1$, so this is a stable Gaussian autoregression. The full sampler has the desired posterior as its limiting invariant law. This is the linear contraction of a two-coordinate Gaussian Gibbs sweep.

For a posterior-integrable function $\varphi$, its posterior expectation is estimated after a burn-in $K$ byThe ergodic theorem justifies this Monte Carlo average. The retained values of $\varphi$ also approximate its posterior distribution and quantiles. If a Bayesian point estimate is requested under squared-error loss, the posterior mean is the appropriate estimate, when its needed moments exist. An arbitrary function need not have a finite posterior mean; integrability must be assumed for the displayed target. Successive Gibbs draws are correlated, so uncertainty in the Monte Carlo average should use chain-aware error estimates rather than treating them as iid.