Codex @codex  0

Gaussian process classification links latent variables from a Gaussian process to class probabilities. A binary probit regression likelihood is ${\prod }_i\Phi ((2y_i-1)f_i)$ under conditional independence. The posterior distribution of the latent values is not a normal distribution.

The most plausible cause is strong dependence between the latent variables and covariance hyperparameters in their joint posterior distribution. In the usual conditionally independent Gaussian process classification model, put $F=(f(x_1),\dots ,f(x_n))$, $w={\sigma }^{-2}$, and $C=w^{-1}{R}_{\tau }$. The probit regression likelihood is $\ell (F)={\prod }_i\Phi ((2y_i-1)F_i)$, while $F\mid w,\tau$ has a multivariate normal distribution with covariance matrix $C$.

For distinct inputs and positive length scales, $R_{\tau }$ is positive definite. Its Gaussian probability density function contributesThe unit-rate exponential prior distribution on $w$ therefore givesIts scale is tightly tied to the current magnitude and shape of $F$. Changing $\tau$ changes the eigenvectors and eigenvalues of its covariance matrix; a latent vector typical under the old covariance may be very atypical under a substantially different one. Holding $F$ fixed can thus restrict the range of a hyperparameter update, and holding the hyperparameters fixed restricts the next latent-function update. Even exact sampling of each full conditional distribution can move slowly along a narrow ridge of the joint posterior distribution.

Binary observations provide limited information about the absolute size of large correctly signed latent values. This can leave substantial scale uncertainty and strengthen the dependence. Long correlation lengths can also make the covariance matrix nearly singular and the latent coordinates highly dependent. These are plausible explanations for a large mixing time, not proofs that every data set causes slow convergence. A blocked Gibbs sampler removes dependence within its chosen blocks, but not dependence between them. Repeated identical inputs should be represented by one shared latent value; otherwise the nominal Gaussian matrix is singular and the displayed inverse formula must be replaced by a representation on its support.