Gaussian conjugacy for a normal linear model

If $Y\mid \beta \sim N(X\beta ,{\Sigma }_e)$ and $\beta \sim N(m_0,{\Sigma }_0)$, then the posterior is normal with precision matrix $X^T{\Sigma }_e^{-1}X+{\Sigma }_0^{-1}$ and mean equal to the inverse precision times $X^T{\Sigma }_e^{-1}Y+{\Sigma }_0^{-1}{m}_0$.

In a normal linear model $Y\mid b,s\sim N(Xb,sI)$, assign the coefficient vector $b$ a proper normal distribution with fixed positive-definite covariance matrix $D$, independently of the variance $s$, whose prior distribution is inverse-gamma distribution. Completing the square givesMultiplying the residual likelihood function by the inverse-gamma distribution density also givesThese full conditional distributions permit a Gibbs sampler. The marginal coefficient posterior distribution is obtained by marginalization over $s$. Its conditional mean and covariance matrix both depend on $s$, so it is generally not a Student's t-distribution. Even if the columns of $X$ and the prior coordinates make the conditional coefficient components independent, averaging over a shared unknown $s$ need not preserve that independence.

A normal-inverse-gamma prior instead scales the conditional coefficient prior covariance matrix by $s$, producing a multivariate Student's t-distribution on marginalization. It is a different joint prior distribution.

Condition on initial observations $x_0,x_1$ and modelwith known ${\sigma }^2>0$ and independent standard-normal prior distributions for $a,b$. Put $v_t=(x_{t+1},x_t{)}^{\mathsf{T}}$ and $\beta =(a,b{)}^{\mathsf{T}}$. Gaussian conjugacy for a normal linear model gives posterior precision matrix and posterior meanThe prior distribution makes $\Lambda$ positive-definite even with a deficient design matrix. The unscaled sums occur specifically when ${\sigma }^2=1$.

If $\Lambda =\left(\begin{array}{cc}A& C\\ C& D\end{array}\right)$ and $h=(r,s{)}^{\mathsf{T}}$, the two full conditional distributions have means $(r-Cb)/A$ and $(s-Ca)/D$ and variances $1/A$ and $1/D$. These formulas use the conditional autoregressive model of order two likelihood function. Giving the initial pair its stationary law, or imposing a stationary-parameter restriction, changes that posterior distribution.

Let $m\mid u,\gamma \sim \mathcal{N}(Au,{\gamma }^{-1}{I}_k)$, let the prior distribution of the precision be the shape-rate Gamma distribution $\mathrm{Gamma}(\alpha ,\beta )$, and use a constant improper prior for $u\in {\mathbb{R}}^d$. Assume $A$ has zero null space. Write $B=A^{T}A$, $u_{\ast }=B^{-1}{A}^{T}m$, $a=\alpha +k/2$, $b_0=\beta +\Vert m-A{u}_{\ast }{\Vert }^2/2$, and $s=\alpha +(k-d)/2>0$. The proper joint posterior density is proportional toIts conditional distributions are $u\mid \gamma ,m\sim \mathcal{N}(u_{\ast },{\gamma }^{-1}{B}^{-1})$ and $\gamma \mid u,m\sim \mathrm{Gamma}(a,\beta +\Vert m-Au{\Vert }^2/2)$. Its marginal precision law is $\mathrm{Gamma}(s,b_0)$. Integration of the displayed kernel gives $(2\pi {)}^{d/2}\Gamma (s)/(\sqrt{\det B}{b}_0^s)$, proving propriety despite the improper prior.

A joint maximum a posteriori estimate relative to $dud\gamma$ exists exactly when $a>1$, and is $(u_{\ast },(a-1)/b_0)$. If $a=1$, the supremum is approached at $\gamma =0$ but not attained; if $a<1$, the density is unbounded there. Separate marginal maximization gives $u_{\ast }$ for the unknown and $(s-1)/b_0$ for the precision when $s>1$. This illustrates that joint and marginal modes need not agree, and that a proper posterior distribution need not have a mode in its open parameter space.

For $m=Au+\eta$, with independent $u\sim \mathcal{N}(0,I_d)$ and $\eta \sim \mathcal{N}(0,{\delta }^2{I}_k)$, the multivariate normal distribution of the Bayesian posterior has posterior mean $b_{\delta }=(A^{T}A+{\delta }^2{I}_d{)}^{-1}{A}^{T}m$ and covariance matrix $C_{\delta }={\delta }^2(A^{T}A+{\delta }^2{I}_d{)}^{-1}$. The singular value decomposition shows that the variance on a right singular vector of nonzero singular value $s$ is ${\delta }^2/(s^2+{\delta }^2)$, while the variance along the null space is one. Thus, for fixed data, the mean tends to the minimum-norm least-squares solution and the covariance tends to the orthogonal projection onto the null space. Unobserved components retain their prior distribution even as the noise vanishes.

A Gaussian likelihood is a likelihood function obtained by modeling the observations conditionally with a normal distribution.