= Solution
The most plausible cause is \b[strong dependence between the <latent variables> and <covariance> <hyperparameters> in their joint <posterior distribution>]. In the usual conditionally independent <Gaussian process classification> model, put $F=(f(x_1),\ldots,f(x_n))$, $w=\sigma^{-2}$, and $C=w^{-1}R_\tau$. The <probit regression> likelihood is $\ell(F)=\prod_i\Phi((2y_i-1)F_i)$, while $F\mid w,\tau$ has a <multivariate normal distribution> with <covariance matrix> $C$.
For distinct inputs and positive length scales, $R_\tau$ is positive definite. Its Gaussian <probability density function> contributes
$$
p(F\mid w,\tau)\propto w^{n/2}|R_\tau|^{-1/2}
\exp\!\left(-\frac w2F^\top R_\tau^{-1}F\right).
$$
The unit-rate exponential <prior distribution> on $w$ therefore gives
$$
\boxed{w\mid F,\tau,y\sim\operatorname{Gamma}\!\left(1+\frac n2,\ 1+\frac12F^\top R_\tau^{-1}F\right).}
$$
Its scale is tightly tied to the current magnitude and shape of $F$. Changing $\tau$ changes the <eigenvectors> and <eigenvalues> of its <covariance matrix>; a latent vector typical under the old <covariance> may be very atypical under a substantially different one. Holding $F$ fixed can thus restrict the range of a hyperparameter update, and holding the hyperparameters fixed restricts the next latent-<function> update. Even exact sampling of each <full conditional distribution> can move slowly along a narrow ridge of the joint <posterior distribution>.
Binary observations provide limited information about the absolute size of large correctly signed latent values. This can leave substantial scale uncertainty and strengthen the dependence. Long correlation lengths can also make the <covariance matrix> nearly singular and the latent coordinates highly dependent. These are plausible explanations for a large <mixing time>, not proofs that every data set causes slow convergence. A <blocked Gibbs sampler> removes dependence within its chosen blocks, but not dependence between them. Repeated identical inputs should be represented by one shared latent value; otherwise the nominal Gaussian <matrix> is singular and the displayed inverse formula must be replaced by a representation on its support.
Back to article page
