Past exam of the mathematics course of the University of Cambridge / 2018 / iii / Paper 216 / 6

Let $h(\beta )=\Phi (x_{\mathrm{test}}^T\beta )$. To distinguish the number of posterior draws from the training-sample size, write it as $N$; in the question's sample notation $N=n$. The Monte Carlo estimator of the requested posterior mean isFor independent exact posterior draws it is unbiased, andThe features are conditioned on as fixed data. The estimated probability concerns a future genuine banknote under the posterior predictive probability, not an indicator sampled from that predictive distribution.

Put $t_i=x_i^T\beta$ and $s_i=2Y_i-1\in \{-1,1\}$. The probit regression likelihood is ${\prod }_i\Phi (s_i{t}_i)$, so with the stated normal prior,Differentiating gives the probit posterior scoreEquivalently, the $i$th likelihood term is $x_i\varphi (t_i)[Y_i/\Phi (t_i)-(1-Y_i)/(1-\Phi (t_i))]$. The prior's gradient is essential.

This posterior density is smooth, positive everywhere, and bounded above by a constant times the Gaussian prior, since the likelihood is at most one and has positive normalizing constant. Its derivatives are integrable: differentiating a likelihood factor gives a bounded normal density, and differentiating the prior gives a linear factor times a Gaussian. Integration by parts therefore yieldsThus each component has a known zero expected value and can be used as a posterior score control variate. This identity differentiates the posterior density in the random parameter; it is distinct from the usual mean-zero score identity for a likelihood under repeated sampling.

The requisite second moments are finite. For $t\ge 0$, $\varphi (t)/\Phi (t)$ is bounded. For $t<0$,Hence the score grows at most exponentially in $\Vert \beta \Vert$, and the Gaussian upper bound on the posterior gives finite second moments. For a fixed vector $a$, averaging $h(\beta )-a^{T}g(\beta )$ is therefore an unbiased Monte Carlo estimator of the same mean; suitable coefficients can reduce its variance.

All moments below are under the proper posterior distribution of part (b). Write the known covariance matrix in block formThe score covariance $S$ is positive definite in this model. Indeed, a further integration by parts gives $\mathbb{E}[g(\beta ){\beta }^T]=-I_p$. If $a^{T}Sa=0$, then $a^{T}g=0$ almost surely, because $\mathbb{E}[g]=0$. Multiplying by $a^T\beta$ would give $0=-\Vert a{\Vert }^2$, so $a=0$.

For a fixed coefficient vector $a$, the control variate summand has varianceThus the unique minimum is attained at $a_{\ast }=S^{-1}c$, givingThe coefficient is fixed because the covariance is assumed known, and the posterior score control variate has mean zero, so this estimator remains unbiased.

We can identify exactly when the improvement is strict. Integration by parts with the bounded smooth function $h$ givesThe scalar expectation is strictly positive. Therefore $c\ne 0$ whenever $x_{\mathrm{test}}\ne 0$, and positive definiteness gives a strictly smaller variance in that case. If $x_{\mathrm{test}}=0$, $h\equiv 1/2$ and both estimators already have variance zero. The printed request for a smaller variance therefore needs this nondegeneracy qualification; a non-increasing variance always holds.