In a Gaussian linear mixed model with , the conditional multivariate normal distribution of given has mean and covariance matrix . When nonsingular, its mean is also its mode. Estimated parameters yield empirical conditional modes and shrink group deviations toward zero. At a zero variance component the associated effect is degenerate at zero; the covariance formula still applies, whereas formulas involving require limits.
In a linear Gaussian state-space model, let the predicted state mean and covariance be . The observation innovation process has conditional mean zero and covariance , with innovation . The conditional multivariate normal distribution gives the predictive observation density. Multiplying these conditional densities by the chain rule gives the displayed likelihood function; equivalently the innovations are independent Gaussian vectors. The Kalman filter supplies recursively, so one evaluates the exact observed-data likelihood function without integrating all latent states jointly.
Gaussian process regression estimates an unknown function by assigning its values a Gaussian process law, then conditioning on observed values, often with independent Gaussian noise. The covariance function expresses the proposed relation between nearby inputs; the conditional multivariate normal distribution gives the resulting Gaussian process regression posterior. Its conditional mean provides a prediction and its conditional variance quantifies uncertainty under the chosen model. The index set can represent spatial locations or other inputs as well as time.
Take the original synthetic model and , mutually independent random variables, with the second argument denoting variance. The pair has a multivariate normal distribution with mean zero, unit marginal variances, and Pearson correlation coefficient . Its conditional multivariate normal distribution is
For a first reading , the second reading has conditional expectation and conditional variance , hence standard deviation . There is still conditional probability of , a movement farther from zero, where is the standard normal distribution function. The line describes a conditional mean, not an individual trajectory.
Figure 1.
An extreme first reading predicts a less extreme conditional mean, with substantial scatter
. Original synthetic scatter plot of 1,800 generated pairs, explicitly not actual measurements. Each pair has unit marginal variances and Pearson correlation coefficient ; orange points satisfy . The blue line is the conditional expectation , the band is one conditional standard deviation on either side, and the dashed line denotes . The band is not a confidence interval for an estimated line. The axes show a finite display window. Details.
A short derivation isolates . Its covariance with is zero and its variance is . Because has a multivariate normal distribution, zero covariance implies independence: independently of . This also explains why reversing the two measurements gives . The statistical association does not identify a direction of causal effect.
Kalman filter Created 2026-10-03 Updated 2026-10-14
The Kalman filter alternates a Kalman prediction step with a Kalman measurement update. It carries a state estimate forward using a state-space model, then adjusts it by the part of the new observation that was not predicted: the innovation. The Kalman gain determines the adjustment from the uncertainty of the prediction and the measurement noise. A noisy measurement should have less influence than an equally surprising precise one.
For a precise discrete-time statement, let the state and observation be vectors in real vector spaces obeying the linear state-space model
The matrices and control inputs are known. Assume a Gaussian random vector with mean and covariance matrix , and mutually independent centered Gaussian noises , independent of , with covariance matrices . Suppose is positive-definite, so the observation innovation covariance is invertible. Then the recursions below give the exact posterior mean and posterior covariance matrix , hence the full conditional multivariate normal distribution. They also minimize mean squared error. Without the Gaussian assumptions, suitable second-order assumptions still give best affine state estimates, but do not generally give an exact posterior distribution. [ref].
For an exact scalar example, suppose the predicted state has normal distribution and a measurement reports in the same units. With measurement noise variance , the Kalman gain is and the posterior distribution is
With the same prediction and observed value but measurement noise variance , the Kalman gain becomes and the posterior distribution is instead . Both updates move toward the observation; the more precise measurement moves the estimate further and reduces uncertainty more.
Figure 1.
The same observed value produces different Kalman measurement updates when its uncertainty changes
. Both panels start from and observe . The green curve is the likelihood function normalized over the scalar state for this unit-coefficient observation model. The orange curve is the posterior density; its narrower width reflects reduced variance. The vertical dotted line marks the observed value, rather than a known true state. Details.
The useful compression is exact under the stated model: the two objects contain everything from the past observations needed for the next update. Their size depends on the state dimension, not on the length of the observation history. This makes causal state filtering practical when observations arrive continually. It also gives Linear-quadratic-Gaussian control an estimate of the state on which to act.
Rudolf E. Kalman's 1960 paper organized linear filtering around state transitions, error covariance and recursive computation, connecting estimation with optimal regulation. Its introduction acknowledges earlier work by Wiener, Bode, Shannon and others; the useful historical claim is this particular formulation, rather than that estimation began in 1960. [ref].
Write the expected value as and assume the relevant covariance matrix is invertible. Remove the linear predictor from the first block by defining
A linear image of a multivariate normal vector is again multivariate normal. Direct covariance calculation gives
Thus and are independent: uncorrelated jointly Gaussian variables are independent. Expanding the covariance matrix of , using , gives
Conditioning on translates by the fixed vector and leaves its covariance matrix unchanged. Consequently the conditional multivariate normal distribution is
The conditional covariance is the Schur complement and does not depend on .
Applying the conditional multivariate normal distribution to the two differences gives, more generally,
At the result is
Both this law and the marginal law have mean zero, but the conditional variance is smaller. Specifically, with , the conditional variance is the marginal variance times . The assumptions imply , so the decrease is strict. Matching the breadth differences to zero gives information about the correlated length differences; it does not force the length differences themselves to be zero. The residual uncertainty remains positive. The original PDF has the between-son differences used here; the self-subtractions in the converted TeX are transcription errors.
Write the partitioned mean as and assume is nonsingular, as required by the displayed inverse. For the conditional multivariate normal distribution, form the residual
A linear transformation of a multivariate normal distribution is again normal. Direct calculation gives
The jointly normal blocks and are therefore independent: their joint characteristic function has no cross term in its quadratic exponent and factors into the two marginal characteristic functions. Conditioning leaves the distribution of unchanged. Reconstructing proves the full conditional law, including the requested Schur complement covariance:
This describes a regular conditional law; it does not require the equality event to have positive probability.
Fix . For , the residual observed path is
This Brownian bridge is independent of . Indeed it is independent of , while its covariance with is ; zero covariance implies independence in the joint Gaussian law. The same argument for every finite collection of bridge times, followed by continuity and generation from rational times, gives independence of the entire bridge. Since the full observed path is determined by its endpoint and this independent bridge, Brownian endpoint sufficiency for a constant drift gives
First suppose the prior covariance matrix is positive definite. The Bayes' theorem multiplies the prior Gaussian density by the conditional observation density, whose dependence on is . Thus the posterior exponent, up to terms not depending on , is
Completing the square gives the Gaussian Brownian drift filter:
For invertible these are equivalently and . At they reduce to the prior parameters.
The boxed formulas remain valid for a singular positive semidefinite matrix . To see this directly without inverse prior matrices, the joint Gaussian random vector has
The latter matrix is invertible for . The conditional multivariate normal distribution therefore has mean
and covariance matrix
Alternatively, subtract this mean from ; the resulting Gaussian random vector has zero covariance with and is therefore independent of it, which proves these conditional parameters.
Write the proposed innovation process in the observable form
It is adapted to the observation filtration and continuous. To prove the martingale property, take . Future increments of are independent of the joint information generated by and the past of , so
Also the tower property of conditional expectation gives for . Conditional integration is legitimate because . Subtracting the integral proves . Removing a continuous finite variation process does not change quadratic covariation, hence
The Lévy characterization of multidimensional Brownian motion now applies: a continuous zero-start martingale with this bracket matrix is a standard -dimensional Brownian motion relative to its given filtration. The observable innovation is a Wiener process in the observation filtration.
Differentiating the posterior formulas yields and
For the final expression, the printed inverse matrices require the extra assumption that is positive definite. Under that assumption put . The Itô formula gives
since and . On the other hand,
The trace terms cancel in the innovation exponential for Gaussian drift filtering, leaving
Thus and is a positive stochastic exponential, hence a nonnegative local martingale. To obtain the requested supermartingale conclusion explicitly, choose localizing stopping times for which is a true martingale. The Conditional Fatou lemma gives
The same argument at gives , supplying integrability. The process is a supermartingale in the observation filtration. No unproved exponential-integrability condition on the random drift is required.
For a singular prior the literal inverse-matrix expression in the PDF is undefined, but there is a natural extension with the same conclusion. Define, for fixed ,
For positive definite , a Gaussian integral shows this reciprocal is exactly the printed expression. For any positive semidefinite , the integral is positive and finite, , and , by the same posterior weighting. The Itô formula consequently gives . Its reciprocal satisfies the boxed stochastic differential equation and the same supermartingale proof. This completes the singular-covariance extension while explicitly separating it from the undefined printed formula.
The state-space model as printed supplies only uncorrelated white noise. That is insufficient for the claimed normal conditional distributions. For example, take normal, independent process noises taking values equally often, and independent Gaussian observation noises. With and no previous observations, is a two-component shifted mixture distribution, not normal for . Its characteristic function is , which has zeros, whereas a normal characteristic function has none. Similarly a non-Gaussian observation noise prevents the asserted conditional normal law for .
The intended scalar Gaussian Kalman recursion requires and independent of each other and of the previous state and observation history. Assume these Gaussian independence conditions, , , and the given normal filtering law. Conditional on the past, is a sum of independent normals, hence
The new observation noise is independent of and the past, so . Write and . The independent normal pair shows directly that
The covariance comes from the common term. Applying the conditional multivariate normal distribution formula with as the conditioned coordinate gives the Kalman filter update
The final expression verifies nonnegativity of the posterior variance.
For , define
The recursions become and . If , then and, with ,
The nonnegative solution is the steady-state local-level Kalman gain:
For the limit is zero; for it lies strictly between zero and one. These are exact conditional-law results under the additional Gaussian assumptions, not consequences of the weaker printed noise description alone.
Use the standard linear Gaussian state-space model convention: the initial state is independent of both mutually independent iid noise sequences. This independence from the initial state is needed for the usual forecast recursion. Let and write and . Taking conditional expectations in the state equation gives
To justify the covariance, the forecast error is . Its two terms have zero cross-covariance because is independent of the previous state and observations. Their covariances therefore add. In this Gaussian model the conditional error covariance depends on the parameters and time but not on the realized data, so it also equals the unconditional forecast-error covariance used in the notation of the question.
For completeness, derive the filtering step rather than leave the algorithm unspecified. Set and . Conditionally on the previous data, the observation has mean and covariance
Define the innovation process and gain by
The state forecast error and innovation are jointly normal, with cross-covariance . The residual is uncorrelated with by the definition of , hence independent of it by joint normality. Its mean is zero and its covariance is . This proves the conditional multivariate normal distribution update
Initialize . For , predict using the first boxed pair, store that forecast, form , and update using the second boxed pair. The updated state then starts the next iteration. This is the Kalman filter, and supplies the entire forecast sequence in one forward pass.
The requested unconditional innovation moments are
In fact , with deterministic for fixed parameters. Thus the innovation is independent of the past observations, and hence the innovations are mutually independent. Equivalently, orthogonality to the past together with joint normality gives this independence.
The chain rule for densities now yields the Gaussian innovation likelihood, treating the specified initial mean and covariance as known:
Both and are functions of the trial parameters, so the filtering recursion must be run again for each candidate . The ordinary density formula assumes positive definite; for example positive definite ensures this. Singular observation laws instead require densities on their Gaussian support.
Evaluate the log-likelihood numerically by the forward recursion, and maximize it over admissible , enforcing the covariance constraints. A Cholesky decomposition gives and the quadratic term , avoiding an explicit matrix inverse. Covariance parameterizations or constrained optimization keep positive semidefinite and the innovation covariance nonsingular. An attained maximizer is a maximum-likelihood estimator; in general this is a numerical optimization problem and neither uniqueness nor global convergence of a local optimizer is automatic.
Put , the precision matrix. The conditional independence graph has an edge between and exactly when and are not conditionally independent given all the remaining components. For the multivariate normal distribution,
Indeed, the conditional density of the pair given the others has its quadratic term determined by the corresponding two-by-two principal submatrix of ; its cross term vanishes exactly when , making the two conditional Gaussian variables independent. This is a Gaussian graphical model.
To derive the conditional law of one component, isolate in the multivariate normal density and complete the square. The terms involving it are
Therefore
Equivalently, its mean is and its variance is the Schur complement . These forms agree because the block of gives , and its block then identifies the variance. This proves the relevant conditional multivariate normal distribution formula.
In particular, the population regression of component on the other components has coefficients
intercept , and a Gaussian residual independent of the predictors with variance . These are the conditional regression coefficients from a Gaussian precision matrix. The coefficient is zero exactly when the corresponding edge is absent. Its counterpart has the same zero pattern and sign, but generally a different magnitude.
This motivates nodewise regression: for each , regress on , estimating an intercept as well. For sparse graph estimation, use the Nodewise Lasso criterion
The Lasso penalty encourages many fitted coefficients to be zero and remains usable when the number of predictors exceeds the sample size. Ordinary unpenalized sample regressions would usually give a dense estimated graph. Separate regressions can disagree about a pair, so an undirected graph can use the OR rule for nodewise graph selection, retaining an edge if either fitted direction is nonzero, or the more conservative AND rule for nodewise graph selection, retaining it only if both are nonzero. These are choices for combining estimated neighbourhoods, rather than identities forced by finite-sample fits.
The printed joint objective uses a Group Lasso penalty on the unordered pairs
This is paired group Lasso for Gaussian graph estimation. Each group represents a single undirected edge, coupling its two directional regressions. A sum of group norms encourages entire pairs to be zero; a squared group norm would give smooth ridge-type shrinkage instead of edge deletion. The constraint removes self-prediction, and the intercepts are not penalized.
The free symbols called in the objective are regression intercepts, not generally the marginal means of the original Gaussian vector. Profiling them out gives
and replaces all columns by their centred versions in the squared-error terms. This is profiling intercepts in nodewise regression. Scaling predictors to comparable units can also help interpret a common penalty, although the displayed objective itself is well defined on the given data.
The grouped KKT conditions make the sparsity mechanism precise. With centred residuals , define
The subdifferential of the Euclidean norm at zero is its closed unit ball, and away from zero is . Hence a minimizer satisfies
These are the paired group Lasso optimality conditions. They jointly threshold the two directional scores, matching the population fact that both directions represent the same edge.
Estimate the graph by including exactly when . This graph is undirected by construction. The penalty does not impose , nor does it mathematically require both coordinates of every selected pair to be nonzero. That distinction is appropriate because the population regression coefficients themselves need not have equal magnitudes. The fitted coefficient matrix is not automatically a symmetric positive definite precision matrix; its pair supports, rather than its entries as a precision estimate, supply the requested conditional independence graph.
For the Gaussian linear mixed model, temporarily regard as known, with here and . The joint multivariate normal distribution is
By the conditional multivariate normal distribution formula,
A nonsingular Gaussian density has its mode at its mean. Thus the conditional mode of Gaussian random effects equals the conditional expectation. Replacing the unknown parameters by fitted values gives
For positive-definite and , an equivalent derivation minimizes the negative conditional log density,
Differentiation yields
The inverse-covariance penalty shrinks furnace intercepts and slopes toward the population effects. For furnace , set and ; the same formula provides its two-component conditional mode from that furnace's observations, conditional on the fitted common parameters. If a variance component is zero, the covariance form remains meaningful and its corresponding random effect is zero, whereas the formula with needs a limiting interpretation. With known covariance parameters these are best linear unbiased predictors; plug-in variance estimates give their empirical version.
For one factor, write for the loading vector and . Let be its current value and . The E-step of the expectation-maximization algorithm uses the conditional multivariate normal distribution to obtain
Thus . The terms of the expected complete-data log-likelihood that depend on a candidate are
This is a strictly concave quadratic in , since . Its unique maximizer is
For an explicit expression involving only the current loading and data, put . Substitution gives the EM update for a single-factor Gaussian model
No empirical centering is applied because the model fixes the mean to zero. If initialized at zero, this update remains at zero; monotonicity alone does not guarantee that such a stationary point is a global maximum. This part is present in the PDF but missing from the TeX transcription.
At the fitted hyperparameters, define , , and , with the matrices formed by pairwise kernel evaluations. Conditional independence of observation noise and future latent values gives
The residual has zero covariance with , so the joint normal law makes it independent of . This derives the conditional multivariate normal distribution and the Gaussian process regression posterior:
The covariance is a Schur complement and is positive semidefinite. When nonsingular, its joint density is . Repeated phases can make it singular, in which case the displayed normal law is interpreted on its support.
This predicts the true latent light curve, so no future measurement variance is added. For noisy future observations, add their independent noise covariance to . Fixing the hyperparameters omits their estimation uncertainty, as requested; integrating over their posterior would add it. The periodic model extrapolates by phase, even though all prediction times follow the observed times.
Let . The Karhunen–Loève expansion and Gaussianity give
and
The conditional multivariate normal distribution formula yields
If is singular, the same formula uses its Moore-Penrose pseudoinverse.
For disjoint nonempty index sets and , partition the mean and covariance conformably. The conditional multivariate normal distribution is
If the sets overlap, the shared coordinates are fixed and this formula applies to .
Partition into the and blocks. The block-inverse formula identifies the displayed conditional covariance with
For , inversion of the two-by-two precision block gives
The denominator is positive, so the conditional covariance vanishes exactly when . A Gaussian pair is independent exactly when it is uncorrelated, proving
Profiling the Gaussian likelihood over gives . Apart from constants and a positive factor, the negative log-likelihood for the precision matrix is
Its derivative is , so the unpenalized minimizer is . The Graphical Lasso solves
with some conventions also penalizing the diagonal.
Let have entries , let have entries , and let . The Gaussian process prior and independent Gaussian noise imply
Applying the conditional multivariate normal distribution gives the Gaussian process regression posterior
where
The conditional multivariate normal distribution gives
For a jointly Gaussian vector, the residual is independent of the regressor, so with independent of .
The Square-root Lasso minimizes
At a nonzero residual , its KKT condition is
which proves the required inequality.
Writing the response vector as , the reverse triangle inequality gives
The strong law of large numbers gives , hence .
Under , Gaussianity makes independent of and hence of . Conditionally on , . The remaining numerator term obeys
Combining this with and Slutsky theorem proves the standard-normal limit.
Let be a real symmetric matrix with a positive-definite matrix. Its quadratic form satisfies the completing the square identity
For each fixed vector , the unique minimizing vector is , so the minimized value is . Also, is a positive-definite matrix exactly when is a positive-definite matrix, under the stated assumption on .[ref]
An original scalar example is
Here , , , and the Schur complement is . Holding would give , but letting relax to reduces the quadratic form to . At the respective values are and . Eliminating a variable therefore changes the effective cost of the retained variable.
Figure 1.
A Schur complement measures the quadratic form left after another variable relaxes
. Original contours of and the two resulting costs. The minimizing line picks the least value at each fixed ; the plotted profiles are exact model values, not experimental measurements. Details.