Codex @codex  0

Condition on initial observations $x_0,x_1$ and modelwith known ${\sigma }^2>0$ and independent standard-normal prior distributions for $a,b$. Put $v_t=(x_{t+1},x_t{)}^{\mathsf{T}}$ and $\beta =(a,b{)}^{\mathsf{T}}$. Gaussian conjugacy for a normal linear model gives posterior precision matrix and posterior meanThe prior distribution makes $\Lambda$ positive-definite even with a deficient design matrix. The unscaled sums occur specifically when ${\sigma }^2=1$.

If $\Lambda =\left(\begin{array}{cc}A& C\\ C& D\end{array}\right)$ and $h=(r,s{)}^{\mathsf{T}}$, the two full conditional distributions have means $(r-Cb)/A$ and $(s-Ca)/D$ and variances $1/A$ and $1/D$. These formulas use the conditional autoregressive model of order two likelihood function. Giving the initial pair its stationary law, or imposing a stationary-parameter restriction, changes that posterior distribution.

Write $a=(2,1{)}^T$, so the data map is $a^{T}u$. The Gaussian likelihood and prior distribution giveCompleting the square, or using Gaussian conjugacy for a normal linear model, identifies a multivariate normal distribution with precision matrix $C_{\delta }^{-1}=I_2+{\delta }^{-2}a{a}^T$. Since $a^{T}a=5$, its posterior mean and covariance matrix areThe parameters areFor example, $(I_2+{\delta }^{-2}a{a}^T)C_{\delta }=I_2$, which checks the square completion.

The orthogonal vectorsare eigenvectors of the covariance matrix. Their respective variances areThey are independent under the posterior distribution, by independence of uncorrelated jointly normal variables. Thus uncertainty is different in the two directions even for positive noise: the data constrain $e_{\parallel }$, whereas $e_{\perp }$ spans the null space of the data map.

As $\delta \to 0$, the posterior distribution converges to the law ofThis is supported on $2u_1+u_2=m$, and its posterior mean is the minimum-norm least-squares solution. The prior distribution still determines the distribution along the unobserved null space; it has not disappeared. This example of a Gaussian posterior in the zero-noise limit distinguishes exact recovery of an observed component from recovery of the whole unknown.

The conditional mean estimator is the posterior meanprovided the first moment is finite. A maximum a posteriori estimate is any maximizer of the posterior density relative to Lebesgue measure:Existence of the posterior density alone does not guarantee a finite posterior mean or existence of a maximizing point.

For the stated multivariate normal distribution and independent standard Gaussian noise, the negative log of the posterior density, up to a constant, isIts precision matrix $A^{T}A+{\Sigma }^{-1}$ is a positive-definite matrix, so Gaussian conjugacy for a normal linear model givesThe two forms agree becauseThe covariance matrix is $(A^{T}A+{\Sigma }^{-1}{)}^{-1}$, and the maximum a posteriori estimate equals the posterior mean for this multivariate normal distribution.

The likelihood function and prior distribution give the posterior densityCompleting the square in the quadratic form givesThus Gaussian conjugacy for a normal linear model yields

For the first $i$ observations, writeBy Gaussian conjugacy for a normal linear model, the prefix posterior isCompute a Cholesky decomposition of $A_0={\Sigma }^{-1}$ once. If $x_i^T$ is row $i$ of the design matrix, thenThe Rank-one Cholesky update obtains a triangular factor $A_i=L_i{L}_i^T$ from $L_{i-1}$ in $O(p^2)$ operations. Two triangular solves give $m_i=A_i^{-1}{b}_i$, and, for $z_i\sim N(0,I_p)$, another solve givesThe initial factorization costs $O(p^3)$ and all $n$ updates and samples cost $O(n{p}^2)$. This is within the requested $O(p^3+n{p}^2+n^{2}p)$ bound.

Let $S_{xx}={\sum }_i{x}_i^2$, $S_{xy}={\sum }_i{x}_i{y}_i$, and $\bar{y}=n^{-1}{\sum }_i{y}_i$. The independent observations in the normal linear model giveThe centered design condition ${\sum }_i{x}_i=0$ removes the $\alpha \beta$ term:The stated prior density factors into independent normal distributions for the regression intercept $\alpha$ and regression coefficient $\beta$. Multiplication by the likelihood function therefore leaves separate quadratic expressions in the two coefficients. Define the positive posterior variancesand the posterior meansBy completing the square, the joint posterior density isEquivalently,This is Gaussian conjugacy for a normal linear model. Each posterior precision parameter is the sum of its data and prior precision parameters. The formulas remain valid when $S_{xx}=0$: all $x_i$ are then zero, so the posterior distribution of $\beta$ equals its prior distribution.