Past exam of the mathematics course of the University of Cambridge / 2002 / iii / Paper 40 / 4 / a

Put $v={\sigma }^2>0$, $\bar{x}=n^{-1}{\sum }_i{x}_i$ and $S={\sum }_i(x_i-\bar{x}{)}^2$. The likelihood function and log-likelihood areFor every fixed $v$, the unique maximum in $\mu$ is $\bar{x}$. At that value, $\partial \ell /\partial v=-n/(2v)+S/(2v^2)$ changes sign from positive to negative at $v=S/n$. Thus, for $S>0$,This is the global maximum-likelihood estimate, since the first maximization was global for each $v$ and the second has a unique peak. If $S=0$, the likelihood function is unbounded at $\mu =\bar{x}$, $v\downarrow 0$; there is no maximizer in the parameter space $v>0$. The freezing argument below uses the nondegenerate case $S>0$.

At temperature $T>0$, target a probability density function proportional to $\exp (f(\theta )/T)$ on a parameter region where it is normalizable, possibly with a specified proper base measure. A Metropolis–Hastings algorithm proposal $q(\theta ,{\theta }^{\prime })$ has acceptance probabilityFor a symmetric proposal, improving moves are accepted and a loss $\Delta$ is accepted with probability $e^{-\Delta /T}$. Run updates while gradually lowering $T$ toward zero. This simulated annealing scheme initially permits exploration and increasingly favors high objective values. Appropriate cooling and mixing conditions are needed for a general global convergence theorem; a finite arbitrary schedule does not guarantee finding a global maximum.

To maximize a positive likelihood function, take $f=\log L$, which has the same maximizers. The resulting likelihood-power annealing target is $L^{1/T}$, and available Gibbs sampler conditional distributions can replace Metropolis proposals.

Use the Gaussian likelihood Gibbs annealing target ${\pi }_T(\mu ,v)\propto L(\mu ,v{)}^{1/T}$ relative to $d\mu dv$, with $S>0$. Completing the normal square and recognizing an inverse-gamma kernel giveThe shape is $n/(2T)-1$, because the probability density function in $v$ is $v^{-n/(2T)}{e}^{-[S+n(\mu -\bar{x}{)}^2]/(2Tv)}$. The base measure matters: inserting a prior factor $1/v$ would change this shape.

Integrating out $\mu$ contributes a factor proportional to $v^{1/2}$, so the marginal is $v\sim \mathrm{IG}(n/(2T)-3/2,S/(2T))$. Thus the joint target is proper for $T<n/3$. Choose a decreasing sequence $0<T_m\le n/10$ tending to zero and a finite starting value $v_0>0$. At iteration $m$, sample ${\mu }_m$ from the first conditional using $v_{m-1}$, then sample $v_m$ from the second using ${\mu }_m$. This is an explicit annealing algorithm entirely using Gibbs sampling, with all its inverse-gamma moments finite for the convergence proof.

We prove convergence of the actual sequential updates above, not just concentration of ideal equilibrium distributions. Write $M_m=\mathbb{E}{v}_m$ and $N_m=\mathbb{E}{v}_m^2$. Given $v_{m-1}$, the centered normal draw satisfiesThe conditional inverse-gamma distribution moments therefore giveFor $T_m\le n/10$, the coefficient multiplying $M_{m-1}$ is at most $1/6$, so $M_m$ stays bounded by a geometric-recursion argument. The coefficient multiplying $N_{m-1}$ is at most $1/8$, and the other terms are bounded using the first-moment bound, so $N_m$ also stays bounded. These bounds start from the finite deterministic $v_0$.

Since $T_m\to 0$, the boundedness makes every term containing $T_m{M}_{m-1}$ or $T_m^2{N}_{m-1}$ vanish. Hence $M_m\to S/n$ and $N_m\to S^2/n^2$. ConsequentlyThusThis establishes mean-square convergence for the specified Gaussian annealer. It does not assert that every implementation or cooling rule for every objective has the same guarantee, nor that mean-square convergence alone is almost-sure convergence.