Sample mean and covariance ($\overline{X},S$)

For observations $X_1,\dots ,X_n\in {\mathbb{R}}^p$, the sample mean and empirical covariance matrix describe average location and joint variation:Each summand is an outer product, so $S_n$ is positive semidefinite. If the observations are independent and identically distributed random variables with covariance matrix $\Sigma$, then $\mathbb{E}{S}_n=(n-1)\Sigma /n$. For $n\ge 2$, replacing the divisor $n$ by $n-1$ gives an unbiased estimator.

For the three observations $(0,0),(1,2),(2,1)$, the sample mean is $(1,1)$. The sum of centered outer products isso the $n$-divisor sample covariance matrix is $\left(\begin{array}{cc}2/3& 1/3\\ 1/3& 2/3\end{array}\right)$, and the $n-1$ version is $\left(\begin{array}{cc}1& 1/2\\ 1/2& 1\end{array}\right)$. Stating the divisor avoids ambiguity when comparing formulas. The rank of a centered sample covariance matrix also explains why too few observations force singularity in high dimension.

The uncentered empirical second-moment matrix averages the outer products of the observations. Its expected value is $\mathrm{Cov}(X)+\mathbb{E}[X]\mathbb{E}[X{]}^{\top }$. It is an unbiased estimator of the covariance matrix when the population expected value is known to be zero. It differs from a sample covariance matrix centered at the sample mean.

For observations $X_1,\dots ,X_n$ with $n\ge 1$, the sample mean is $\overline{X}=n^{-1}{\sum }_i{X}_i$. It is a statistic summarizing location: values $(2,3,7)$ give $\overline{X}=4$. For an IID sample with mean $\mu$ and finite variance ${\sigma }^2$, linearity of expectation gives $\mathbb{E}\overline{X}=\mu$, while variance additivity for independent random variables gives $\mathrm{Var}(\overline{X})={\sigma }^2/n$. Dependence changes the variance through cross-covariances. A sample mean is sensitive to extreme observations, and a population such as the Cauchy distribution need not have the finite mean needed for ordinary averaging guarantees.

For independent samples of sizes $n,m$ from populations with means ${\mu }_X,{\mu }_Y$ and finite variances ${\sigma }_X^2,{\sigma }_Y^2$, the difference of their sample means has expected value ${\mu }_X-{\mu }_Y$ and variance ${\sigma }_X^2/n+{\sigma }_Y^2/m$. This uses independence between and within the samples. With dependent samples, a covariance term is required instead.

Partition a sample into two disjoint groups $A,B$ with fixed positive sizes $n_A,n_B$, and let $w=n_A/(n_A+n_B)$. The overall sample mean is the weighted meanSubtracting gives the displayed identity. The subgroup and overall sample means share observations, so they are not independent. If the two disjoint groups are sampled independently, thenThus treating the subgroup and the combined sample as two independent samples uses the wrong standard error. Equality of the subgroup and overall population means is equivalent to equality of subgroup and complement means when both groups have positive weight, but the sampling distribution must respect the shared observations.

For independent and identically distributed observations with finite positive variance, the central limit theorem, consistency of the sample variance and Slutsky's theorem give $T_n\stackrel{d}{\to }N(0,1)$. Under an independent normal distribution sample it instead has the exact finite-sample Student t-distribution with $n-1$ degrees of freedom, as described by the studentized normal sample mean. The asymptotic statement does not require normal individual observations.

For observations $x_1,\dots ,x_n$, the sample covariance matrix is $n^{-1}{\sum }_i(x_i-\bar{x})(x_i-\bar{x}{)}^T$, or the same sum divided by $n-1$ under the unbiased convention.

If $H=I_n-11^T/n$ and $S=X^{T}HX/n$, then $\mathrm{rank}S=\mathrm{rank}(HX)\le \min (p,n-1)$. In particular full column rank of raw $X$ does not ensure a nonsingular centered sample covariance matrix. When $S$ is singular, its Gaussian precision profile likelihood has no finite maximizer: increase precision along a nonzero null vector.

For paired scalar observations $(x_i,y_i)$, their sample covariance is $n^{-1}{\sum }_i(x_i-\bar{x})(y_i-\bar{y})$, or the same sum divided by $n-1$ under the unbiased convention. It is an off-diagonal entry of the sample covariance matrix of the paired vectors.

For a nonzero positive semidefinite covariance matrix $\Sigma$, its covariance effective rank iswhere the operator norm is induced by Euclidean norm and the ${\lambda }_j$ are the nonnegative eigenvalues. Each nonzero eigenvalue contributes at most one to this ratio, soA scalar multiple of $I_p$ has covariance effective rank $p$, while a matrix $\lambda u{u}^T$ with $\lambda >0$ and a unit vector $u$ has covariance effective rank one. For $\Sigma =\mathrm{diag}(100,1,1)$, the ratio is $102/100=1.02$, although the ordinary matrix rank is three.

Thus the quantity measures total variance relative to the largest directional variance, rather than merely counting nonzero directions. Multiplying $\Sigma$ by a positive scalar leaves the ratio unchanged. Its role in the Gaussian sample-covariance operator-norm bound makes it a useful measure of the dimension that matters for covariance estimation. The zero matrix requires a separate convention because the displayed ratio is $0/0$.

Let $X_1,\dots ,X_n\in {\mathbb{R}}^p$ be independent centered Gaussian random vectors with common nonzero covariance matrix $\Sigma$, and put $\hat{\Sigma }=n^{-1}{\sum }_i{X}_i{X}_i^T$. There is a universal constant $C>0$ such that for $t\ge 1$, with probability at least $1-e^{-t}$,where $r(\Sigma )=\mathrm{tr}(\Sigma )/\Vert \Sigma {\Vert }_{\mathrm{op}}$ is the effective rank of a covariance matrix.

The bound combines an expected operator norm error with Gaussian concentration inequality. In Euclidean space, the Gaussian effective-dimension parameter in the primary result is comparable to this trace ratio; replacing maxima by sums gives the displayed version.^[ref]

For $\Sigma =I_p$, the covariance effective rank is $p$, and relative error is controlled by $\sqrt{(p+t)/n}+(p+t)/n$. covariance concentrated in fewer directions can have much smaller covariance effective rank. The estimator here is the uncentered empirical second moment for a known zero mean; an unknown mean and empirical centering require the corresponding formulation. The bound remains meaningful when $\Sigma$ is singular but nonzero, since it does not invert $\Sigma$.