 Covariance matrix

For a real random vector $X=(X_1,\dots ,X_p{)}^T$ with finite second moments, its covariance matrix collects all pairwise covariances:The diagonal entries are variances. The covariance matrix is a symmetric matrix and a positive semidefinite matrix, because for every real vector $a$,Thus one matrix describes the variance of every linear combination of the coordinates. It is a positive-definite matrix exactly when no nonzero linear combination is constant almost surely. A deterministic linear relation produces a zero eigenvalue.

For example,has unit coordinate variances, covariance $0.8$, and eigenvalues $1.8$ and $0.2$. Its unit eigenvectors $(1,1{)}^T/\sqrt{2}$ and $(1,-1{)}^T/\sqrt{2}$ describe directions of greatest and least projected variance. For a unit vector $a=(\cos \theta ,\sin \theta {)}^T$, direct multiplication gives $a^T\Sigma a=1+0.8\sin (2\theta )$: the variation in the sum direction is nine times that in the difference direction.

The spectral theorem supplies an orthonormal basis of these directions. This connects covariance matrices to principal component analysis: its leading eigenvector identifies the unit-direction projection with the largest variance. For a linear transformation $A$, the transformed random vector has covariance matrix $A\Sigma A^T$, so uncertainties propagate through a linear model by matrix multiplication.

A covariance matrix is a square matrix that captures the covariance between multiple random variables. It is a key concept in statistics, probability theory, and multivariate data analysis. Each element in the covariance matrix represents the covariance between two variables.