Covariance matrix

For a random vector $X$ with finite second moments, its covariance matrix isIt is positive semidefinite, and the variance of $a^{T}X$ is $a^T\mathrm{cov}(X)a$.

A compound-symmetry covariance matrix has a common diagonal entry and a common off-diagonal entry: $\Sigma =aI+cJ$. For a group of size $k$, its within-group eigenvalue is $a$ and its constant-direction eigenvalue is $a+kc$. Thus $a\ge 0$ and $a+kc\ge 0$ characterize positive semidefiniteness. When $c\ge 0$, an independent shared random intercept plus individual noise realizes this covariance. A group mean has variance $c+a/k$; a difference of two disjoint subgroup means within the same group cancels $c$.

For a unit vector $u$ and $\theta >0$, a rank-one covariance spike is the covariance matrix $I+\theta u{u}^{\top }$. Its eigenvalue is $1+\theta$ in the direction $u$ and one on the orthogonal complement. It arises by adding an independent scalar normal distribution signal $Yu$ to a standard Gaussian random vector.

The sample mean and covariance estimate the first two moments of a random vector from repeated observations.

The uncentered empirical second-moment matrix averages the outer products of the observations. Its expected value is $\mathrm{Cov}(X)+\mathbb{E}[X]\mathbb{E}[X{]}^{\top }$. It is an unbiased estimator of the covariance matrix when the population expected value is known to be zero. It differs from a sample covariance matrix centered at the sample mean.

For observations $X_1,\dots ,X_n$, the sample mean is $\overline{X}=n^{-1}{\sum }_{i=1}^n{X}_i$.

For observations $x_1,\dots ,x_n$, the sample covariance matrix is $n^{-1}{\sum }_i(x_i-\bar{x})(x_i-\bar{x}{)}^T$, or the same sum divided by $n-1$ under the unbiased convention.

If $H=I_n-11^T/n$ and $S=X^{T}HX/n$, then $\mathrm{rank}S=\mathrm{rank}(HX)\le \min (p,n-1)$. In particular full column rank of raw $X$ does not ensure a nonsingular centered sample covariance matrix. When $S$ is singular, its Gaussian precision profile likelihood has no finite maximizer: increase precision along a nonzero null vector.

For paired scalar observations $(x_i,y_i)$, their sample covariance is $n^{-1}{\sum }_i(x_i-\bar{x})(y_i-\bar{y})$, or the same sum divided by $n-1$ under the unbiased convention. It is an off-diagonal entry of the sample covariance matrix of the paired vectors.

The effective rank measures the total variance of a covariance matrix relative to its largest directional variance.

For independent centered Gaussian vectors with covariance $\Sigma$ and uncentered sample covariance $\hat{\Sigma }=n^{-1}{\sum }_i{x}_i{x}_i^T$, with probability at least $1-e^{-t}$,

The precision matrix is the inverse of a nonsingular covariance matrix. For a multivariate normal distribution, a zero off-diagonal precision entry encodes conditional independence of the corresponding coordinates given all remaining coordinates.

In a Gaussian graphical model, regressing component $k$ on the others gives the displayed coefficients, residual variance $1/{\Omega }_{kk}$ and intercept ${\mu }_k-{\mu }_{-k}^{\mathsf{T}}{\theta }_{-k,k}$. Opposite directions have identical zero patterns and signs because $\Omega$ is symmetric with positive diagonal, but generally unequal coefficient magnitudes.

For independent multivariate normal observations with unknown mean and nonsingular covariance, profiling the mean at its sample value gives negative log-likelihood proportional to $-\log \det \Omega +\mathrm{tr}(S\Omega )$ over $\Omega \succ 0$. The sample covariance matrix uses divisor $n$. Its minimizer exists exactly when $S\succ 0$, and is then $S^{-1}$.

If a positive-definite matrix $Q=L{L}^T$ is the precision matrix of a multivariate normal distribution, draw independent $\xi \sim N(0,I)$ and solve $L^T\eta =\xi$. Then $m+\eta$ has covariance $L^{-T}{L}^{-1}=Q^{-1}$. This avoids forming the inverse and exploits a banded Cholesky decomposition when $Q$ is banded.

The constrained ${\ell }^1$ minimization estimator minimizes $\Vert \Omega {\Vert }_1$ subject toIts constraints and objective separate by columns. Feasibility of the true precision matrix and elementary matrix-norm inequalities yield entrywise error bounds under an entrywise covariance-estimation bound.

The graphical Lasso estimates a sparse precision matrix by minimizing a Gaussian negative log-likelihood plus an entrywise ${\ell }^1$ penalty.

For a Graphical Lasso solution with inverse $\hat{\Sigma }$, the unique minimizer of $\frac{1}{2}\Vert Wb-W^{-1}{S}_{-j,j}{\Vert }_2^2+\lambda \Vert b{\Vert }_1$, $W^2={\hat{\Sigma }}_{-j,-j}\succ 0$, is $b=-{\hat{\Omega }}_{-j,j}/{\hat{\Omega }}_{jj}$. The inverse identity gives ${\hat{\Sigma }}_{-j,j}={\hat{\Sigma }}_{-j,-j}b$, and the off-diagonal Graphical-Lasso Karush-Kuhn-Tucker conditions become the column Lasso conditions because the corresponding signs are opposite.

For objective $-\log \det \Omega +\mathrm{Tr}(S\Omega )+\lambda \Vert \Omega {\Vert }_{1,\mathrm{entry}}$, the KKT conditions are $-{\Omega }^{-1}+S+\lambda Z=0$ with $Z_{ij}\in \partial |{\Omega }_{ij}|$.

For random variables with finite positive variances,

For square-integrable variables of positive variance, correlation one is equivalent to $X-\mathbb{E}X=c(Y-\mathbb{E}Y)$ with $c>0$. The squared norm of the difference between the two standardized centered variables is $2(1-\rho )$, proving the equivalence. For uncentered variables the relationship can have a nonzero intercept; it need not be proportionality.

For two strictly positive zero-rate stock martingales with bounded volatilities and the same initial value, proportional terminal prices must be equal because their expectations coincide. Conditional expectation then makes the whole paths equal. Comparing their stochastic integrals by the Itô isometry gives equality of the volatility coefficients almost everywhere in time and probability. Perfect positive correlation alone does not provide the required proportionality.

The stocks $S_t=e^{B_t-t/2}$ and $S_t^{\prime }=(1+S_t)/2$ start at one, are true square-integrable zero-rate martingales, and have perfect positive correlation at each positive time. Their bounded multiplicative volatilities are $1$ and $S_t/(1+S_t)$, which differ throughout. The additive intercept is what allows the same correlation but different stochastic-exponential coefficients.

Responses observed on the same process or individual at different times may have nonzero correlation coefficients. Such serial correlation can arise from a shared random effect or from history-dependent conditional means. A scalar dispersion parameter adjustment alone does not represent this temporal dependence.

The intraclass correlation coefficient measures the correlation between observations in the same group. In a random-intercept model with group variance ${\sigma }_G^2$ and individual variance ${\sigma }_I^2$, it is ${\sigma }_G^2/({\sigma }_G^2+{\sigma }_I^2)$.