Gaussian conjugacy for a normal linear model

If $Y\mid \beta \sim N(X\beta ,{\Sigma }_e)$ and $\beta \sim N(m_0,{\Sigma }_0)$, then the posterior is normal with precision matrix $X^T{\Sigma }_e^{-1}X+{\Sigma }_0^{-1}$ and mean equal to the inverse precision times $X^T{\Sigma }_e^{-1}Y+{\Sigma }_0^{-1}{m}_0$.

With independent standard-normal priors on the two coefficients, the conditional AR(2) likelihood gives precision $\Lambda =I+{\sum }_t{v}_t{v}_t^T$, where $v_t=(x_{t+1},x_t{)}^T$, and mean ${\Lambda }^{-1}{\sum }_t{v}_t{x}_{t+2}$. Positive definiteness holds even for a deficient design. If $\Lambda =\left(\begin{array}{cc}A& C\\ C& D\end{array}\right)$ and the linear term is $(r,s)$, the conditional means are $(r-Cb)/A$ and $(s-Ca)/D$, with variances $1/A$ and $1/D$.

Let $m\mid u,\gamma \sim \mathcal{N}(Au,{\gamma }^{-1}{I}_k)$, let the prior distribution of the precision be the shape-rate Gamma distribution $\mathrm{Gamma}(\alpha ,\beta )$, and use a constant improper prior for $u\in {\mathbb{R}}^d$. Assume $A$ has zero null space. Write $B=A^{T}A$, $u_{\ast }=B^{-1}{A}^{T}m$, $a=\alpha +k/2$, $b_0=\beta +\Vert m-A{u}_{\ast }{\Vert }^2/2$, and $s=\alpha +(k-d)/2>0$. The proper joint posterior density is proportional toIts conditional distributions are $u\mid \gamma ,m\sim \mathcal{N}(u_{\ast },{\gamma }^{-1}{B}^{-1})$ and $\gamma \mid u,m\sim \mathrm{Gamma}(a,\beta +\Vert m-Au{\Vert }^2/2)$. Its marginal precision law is $\mathrm{Gamma}(s,b_0)$. Integration of the displayed kernel gives $(2\pi {)}^{d/2}\Gamma (s)/(\sqrt{\det B}{b}_0^s)$, proving propriety despite the improper prior.

A joint maximum a posteriori estimate relative to $dud\gamma$ exists exactly when $a>1$, and is $(u_{\ast },(a-1)/b_0)$. If $a=1$, the supremum is approached at $\gamma =0$ but not attained; if $a<1$, the density is unbounded there. Separate marginal maximization gives $u_{\ast }$ for the unknown and $(s-1)/b_0$ for the precision when $s>1$. This illustrates that joint and marginal modes need not agree, and that a proper posterior distribution need not have a mode in its open parameter space.

For $m=Au+\eta$, with independent $u\sim \mathcal{N}(0,I_d)$ and $\eta \sim \mathcal{N}(0,{\delta }^2{I}_k)$, the multivariate normal distribution of the Bayesian posterior has posterior mean $b_{\delta }=(A^{T}A+{\delta }^2{I}_d{)}^{-1}{A}^{T}m$ and covariance matrix $C_{\delta }={\delta }^2(A^{T}A+{\delta }^2{I}_d{)}^{-1}$. The singular value decomposition shows that the variance on a right singular vector of nonzero singular value $s$ is ${\delta }^2/(s^2+{\delta }^2)$, while the variance along the null space is one. Thus, for fixed data, the mean tends to the minimum-norm least-squares solution and the covariance tends to the orthogonal projection onto the null space. Unobserved components retain their prior distribution even as the noise vanishes.

A Gaussian likelihood is a likelihood function obtained by modeling the observations conditionally with a normal distribution.