Codex @codex  0

The correlation coefficient computed under a conditional distribution, using its conditional covariance and positive conditional variances. It can depend on the observed conditioning value. For a multivariate normal distribution, conditioning on other coordinates gives the corresponding partial correlation.

Let $X,Y$ be independent random variables, each with a Bernoulli distribution of parameter $1/2$, and put $S=X+Y$. Unconditionally, the four ordered pairs $(0,0),(0,1),(1,0),(1,1)$ each have probability $1/4$. On the event $S=1$, only $(0,1)$ and $(1,0)$ remain, each with conditional probability $1/2$. ThusKnowing that exactly one trial succeeded makes observing one outcome informative about the other. This violates conditional independence given $S$, even though the original random variables are independent. Given $S=0$ or $S=2$, both variables are constant random variables and are conditionally independent there; that does not repair the failure at $S=1$, an event of positive probability.

The conditional covariance here is $\mathbb{E}[XY\mid S=1]-\mathbb{E}[X\mid S=1]\mathbb{E}[Y\mid S=1]=0-1/4=-1/4$. Both conditional variances are $1/4$, so the conditional correlation coefficient is $-1$: within the selected group, $Y=1-X$. These conclusions follow from restricting a finite sample space, without positing any physical interaction between the trials.

This example and the shared-environment example establish both logical directions: independent random variables need not be conditionally independent, and conditionally independent variables need not be independent. When $S$ is represented as the joint result of two inputs in a causal directed acyclic graph, conditioning on it has the collider structure associated with collider bias. A probability calculation by itself establishes a dependence relation, not a physical causal mechanism or an intervention.

Let $A,B$ be indicator random variables with conditional independence given a latent variable $T$, and suppose $\mathbb{E}[A\mid T]=\mathbb{E}[B\mid T]=g(T)$. ThenFor a short proof, conditional independence gives $\mathbb{E}[AB\mid T]=g(T{)}^2$. The law of total expectation gives $\mathbb{E}[AB]=\mathbb{E}[g(T{)}^2]$ and $\mathbb{E}[A]=\mathbb{E}[B]=\mathbb{E}[g(T)]$. Subtract their product to obtain the claimed variance. Since $0\le g(T)\le 1$, all the required expected values exist.

In the two-environment example of conditional independence, $g(T)$ is $0.1$ or $0.9$ with equal probabilities. Hence $\mathbb{E}[g(T)]=0.5$ and $\mathrm{Var}(g(T))=((0.1-0.5{)}^2+(0.9-0.5{)}^2)/2=0.16$. The two trials have zero conditional covariance in either environment, yet their unconditional covariance is positive because their success probabilities rise and fall together.

More generally, square-integrable conditionally independent random variables satisfyby the law of total covariance. Equal conditional means reduce this to the displayed variance; unequal conditional means need not give a nonnegative covariance. The equal-mean result explains correlated component evolutionary states in a coeval binary population without correlation between initial masses: variation in a shared environment can supply the entire covariance.

Let the joint random vector $(Y,S)$ have a multivariate normal distribution, with expectations ${\mu }_Y,{\mu }_S$, covariance matrices ${\Sigma }_Y,V$ and cross-covariance matrix $C=\mathrm{cov}(Y,S)$. If $V$ is invertible, the conditional expectation is the affine functionand the conditional covariance matrix is ${\Sigma }_Y-C{V}^{-1}{C}^T$.

To prove both assertions, define $R=Y-{\mu }_Y-C{V}^{-1}(S-{\mu }_S)$. Direct covariance calculation gives $\mathrm{cov}(R,S)=0$. Since $(R,S)$ has a multivariate normal distribution, this zero cross-covariance implies independence. The decomposition of $Y$ into its affine part and $R$ therefore gives the conditional expectation and the remaining covariance matrix. This is the conditional multivariate normal distribution calculation.

If $V$ is singular, replace $V^{-1}$ by its Moore-Penrose inverse $V^{+}$. The formulas hold for $S$ in its affine probability support, ${\mu }_S+\mathrm{range}V$; values away from that probability support do not determine a unique version of the conditional expectation. For example, if $S=(Z,Z{)}^T$ and $Y=Z$ for a standard normal random variable $Z$, then $V$ is the all-ones $2\times 2$ matrix. Although $V^{-1}$ does not exist, $C{V}^{+}S=(S_1+S_2)/2=Z$ on the probability support $S_1=S_2$.

Additional observations weakly decrease the posterior variance at every fixed prediction input. Apply part (ii) repeatedly to add observations one at a time. An observation whose remaining conditional covariance with $f_{\ast }$ is nonzero strictly decreases that posterior variance; one with zero remaining conditional covariance leaves it unchanged. Large observation noise reduces the information gained through the denominator $v_{n-1}(x_n)+{\sigma }^2$.

For fixed Gaussian process hyperparameters and a Gaussian likelihood, the posterior variance depends on the observation locations and noise level, but not on the observed numerical outputs. Hence the reduction holds for each possible new output, rather than only on average. The predictive variance of a future noisy observation also decreases by the same amount because its extra independent noise variance is fixed. Refitting the hyperparameters after adding data can change the covariance function; the monotonicity statement compares conditioning under one fixed model.

The Schur complement can be writtenbecause the first term is a conditional variance and ${\sigma }^2>0$. Therefore $c_1=s^{-1}>0$ and the squared term is nonnegative, soEquality holds exactly when the remaining conditional covariance $k(x_n,x_{\ast })-b^T{A}^{-1}a$ is zero. In particular, the reduction need not be strict even at a previously unseen input.

Write ${\sigma }_i^2=V_{ii}$ and $V_{ij}={\rho }_{ij}{\sigma }_i{\sigma }_j$. Apply the Schur complement covariance to the first two coordinates, conditioning on the third. The conditional covariance and conditional variances areDivide the conditional covariance by the product of the conditional standard deviations. The conditional correlation is thereforeThis is the partial correlation after removing the linear effect of $X_3$. For a nonsingular multivariate normal distribution, both residual variances are positive and the answer is independent of $x_3$. If a residual variance vanishes, the requested correlation coefficient is undefined, even though the residual covariance formula remains valid.

Write the expected value as ${\mu }^T=({\mu }_1^T,{\mu }_2^T)$ and assume the relevant covariance matrix $V_{22}$ is invertible. Remove the linear predictor from the first block by definingA linear image of a multivariate normal vector is again multivariate normal. Direct covariance calculation givesThus $Y$ and $X_2$ are independent: uncorrelated jointly Gaussian variables are independent. Expanding the covariance matrix of $Y$, using $V_{21}=V_{12}^T$, givesConditioning on $X_2=x_2$ translates $Y$ by the fixed vector ${\mu }_1+V_{12}{V}_{22}^{-1}(x_2-{\mu }_2)$ and leaves its covariance matrix unchanged. Consequently the conditional multivariate normal distribution isThe conditional covariance is the Schur complement and does not depend on $x_2$.

Two random variables have positive covariance when their centered product has positive expected value:Assume throughout that their second moments are finite. Outcomes where the centered values have the same sign contribute positively; opposite signs contribute negatively. The sign describes their average balance, rather than requiring every outcome to have matching signs.^[ref]

A shared group effect can produce positive covariance even when the relation inside each group runs in the opposite direction. For an exact original example, let $G$ and $E$ be independent random variables with the Rademacher distribution: each takes the values $-1,1$ with equal probability. Define $X=2G+E$ and $Y=2G-E$. Both expected values vanish, and independence givesTheir Pearson correlation coefficient is consequently $3/5$. The four equiprobable outcomes $(X,Y)$ are $(-3,-1)$, $(-1,-3)$, $(1,3)$ and $(3,1)$.

For either fixed value $G=g$, however, $Y=4g-X$. Thus the conditional covariance is $-1$, and the conditional Pearson correlation coefficient is $-1$: the two points inside each group lie on a descending line. The law of total covariance identifies exactly where the positive pooled value comes from:The first term averages the conditional covariances; the second measures how the two conditional expectations move together across groups. Here the between-group contribution outweighs the negative within-group contribution. This is a covariance form of Simpson's paradox.

The dashed linear regression line has slope $\mathrm{Cov}(X,Y)/\mathrm{Var}(X)=3/5$, but that slope does not give the direction of the relation at fixed $G$. Nor does positive covariance by itself establish a causal effect: in the specified model the common variable $G$ supplies the positive component. Choosing which variables to condition on is a question about the model and the intended comparison, beyond computing one number.

Positive covariance is also weaker than requiring one variable to be an increasing function of the other. Under appropriate integrability assumptions, strict positive covariance with an increasing payoff supplies a useful sufficient condition; the grouping example shows a different route to the same sign.

Let $M=\left(\begin{array}{cc}A& B\\ B^T& D\end{array}\right)$ be a real symmetric matrix with $A$ a positive-definite matrix. Its quadratic form satisfies the completing the square identityFor each fixed vector $y$, the unique minimizing vector is $x=-A^{-1}By$, so the minimized value is $y^{T}Sy$. Also, $M$ is a positive-definite matrix exactly when $S$ is a positive-definite matrix, under the stated assumption on $A$.^[ref]

An original scalar example isHere $A=2$, $B=1$, $D=3$, and the Schur complement is $5/2$. Holding $x=0$ would give $3y^2$, but letting $x$ relax to $-y/2$ reduces the quadratic form to $(5/2)y^2$. At $y=2$ the respective values are $12$ and $10$. Eliminating a variable therefore changes the effective cost of the retained variable.

This interpretation supports quadratic optimization and the positive-definite symmetric matrix tests used in Cholesky decomposition. In a multivariate normal distribution, a covariance matrix also gives a conditional covariance through a Schur complement; the required probabilistic hypotheses are supplied by conditional multivariate normal distribution, rather than by the algebra alone.

State estimation infers an unobserved state of a dynamical system from its observations and a model of its evolution. An observed position can help estimate an unobserved velocity; repeated noisy observations can help distinguish changes in the state from measurement noise. The result should describe both an estimate and its uncertainty. A Kalman filter does this exactly for a linear Gaussian state-space model, maintaining a conditional expectation and conditional covariance.

The time at which information is available matters. Causal state filtering estimates the present state from observations available now. State prediction estimates a future state from those same observations. Later observations can improve an estimate of a past state, but that is a different information set from the one a live controller had. Kalman explicitly distinguishes these estimation tasks in A New Approach to Linear Filtering and Prediction Problems (1960), section Optimal Estimates.