= Covariance matrix
{title2=$\Sigma=\mathbb E[(X-\mathbb EX)(X-\mathbb EX)^T]$}
{wiki}
= Variance-covariance matrix
{synonym}
For a real <random vector> $X=(X_1,\ldots,X_p)^T$ with finite <second moments>, its <covariance matrix> collects all pairwise <covariances>:
$$
\Sigma=\mathbb E[(X-\mathbb EX)(X-\mathbb EX)^T],\qquad
\Sigma_{ij}=\operatorname{Cov}(X_i,X_j).
$$
The diagonal entries are <variances>. The <covariance matrix> is a <symmetric matrix> and a <positive semidefinite matrix>, because for every real <vector> $a$,
$$
a^T\Sigma a=\operatorname{Var}(a^TX)\geq0.
$$
Thus one <matrix> describes the <variance> of every <linear combination> of the coordinates. It is a <positive-definite matrix> exactly when no nonzero <linear combination> is constant <almost surely>. A deterministic linear relation produces a zero <eigenvalue>.
For example,
$$
\Sigma=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}
$$
has unit coordinate <variances>, <covariance> $0.8$, and <eigenvalues> $1.8$ and $0.2$. Its unit <eigenvectors> $(1,1)^T/\sqrt2$ and $(1,-1)^T/\sqrt2$ describe directions of greatest and least projected <variance>. For a unit <vector> $a=(\cos\theta,\sin\theta)^T$, direct multiplication gives $a^T\Sigma a=1+0.8\sin(2\theta)$: the variation in the sum direction is nine times that in the difference direction.
\Image[/covariance-projection-geometry.png]
{title=A covariance matrix encodes projected variance in every direction}
{description=Left: the <covariance ellipse> has a <semi-major axis> of <length> $\sqrt{1.8}$ and a <semi-minor axis> of <length> $\sqrt{0.2}$ along the <eigenvectors> of $\Sigma$. Right: the <variance> of the unit-direction projection is $1+0.8\sin(2\theta)$. This is exact <matrix> geometry for the stipulated <covariance matrix>, with no sampled observations or claimed probability coverage.}
The <spectral theorem> supplies an <orthonormal basis> of these directions. This connects <covariance matrices> to <principal component analysis>: its leading <eigenvector> identifies the unit-direction projection with the largest <variance>. For a <linear transformation> $A$, the transformed <random vector> has <covariance matrix> $A\Sigma A^T$, so uncertainties propagate through a linear model by <matrix multiplication>.
Back to article page
