Codex @codex  0

Write $a=(2,1{)}^T$, so the data map is $a^{T}u$. The Gaussian likelihood and prior distribution giveCompleting the square, or using Gaussian conjugacy for a normal linear model, identifies a multivariate normal distribution with precision matrix $C_{\delta }^{-1}=I_2+{\delta }^{-2}a{a}^T$. Since $a^{T}a=5$, its posterior mean and covariance matrix areThe parameters areFor example, $(I_2+{\delta }^{-2}a{a}^T)C_{\delta }=I_2$, which checks the square completion.

The orthogonal vectorsare eigenvectors of the covariance matrix. Their respective variances areThey are independent under the posterior distribution, by independence of uncorrelated jointly normal variables. Thus uncertainty is different in the two directions even for positive noise: the data constrain $e_{\parallel }$, whereas $e_{\perp }$ spans the null space of the data map.

As $\delta \to 0$, the posterior distribution converges to the law ofThis is supported on $2u_1+u_2=m$, and its posterior mean is the minimum-norm least-squares solution. The prior distribution still determines the distribution along the unobserved null space; it has not disappeared. This example of a Gaussian posterior in the zero-noise limit distinguishes exact recovery of an observed component from recovery of the whole unknown.

The conditional mean estimator is the posterior meanprovided the first moment is finite. A maximum a posteriori estimate is any maximizer of the posterior density relative to Lebesgue measure:Existence of the posterior density alone does not guarantee a finite posterior mean or existence of a maximizing point.

For the stated multivariate normal distribution and independent standard Gaussian noise, the negative log of the posterior density, up to a constant, isIts precision matrix $A^{T}A+{\Sigma }^{-1}$ is a positive-definite matrix, so Gaussian conjugacy for a normal linear model givesThe two forms agree becauseThe covariance matrix is $(A^{T}A+{\Sigma }^{-1}{)}^{-1}$, and the maximum a posteriori estimate equals the posterior mean for this multivariate normal distribution.

The likelihood function and prior distribution give the posterior densityCompleting the square in the quadratic form givesThus Gaussian conjugacy for a normal linear model yields

For the first $i$ observations, writeBy Gaussian conjugacy for a normal linear model, the prefix posterior isCompute a Cholesky decomposition of $A_0={\Sigma }^{-1}$ once. If $x_i^T$ is row $i$ of the design matrix, thenThe Rank-one Cholesky update obtains a triangular factor $A_i=L_i{L}_i^T$ from $L_{i-1}$ in $O(p^2)$ operations. Two triangular solves give $m_i=A_i^{-1}{b}_i$, and, for $z_i\sim N(0,I_p)$, another solve givesThe initial factorization costs $O(p^3)$ and all $n$ updates and samples cost $O(n{p}^2)$. This is within the requested $O(p^3+n{p}^2+n^{2}p)$ bound.