Past exam of the mathematics course of the University of Cambridge / 2004 / iii / Paper 43 / 2

Use the usual sample mean and the divisor-$n-1$ sample covariance matrix,The residual sum is zero, so expanding $x_j-\mu =(x_j-\bar{x})+(\bar{x}-\mu )$ givesMultiplying the independent multivariate normal densities and taking logarithms yields the log-likelihoodHere $|V|$ is the determinant and $\mathrm{tr}$ denotes the trace. The maximum-likelihood estimators areIf $S$ instead denotes the divisor-$n$ sample covariance matrix, replace $(n-1)S$ by $nS$ and then $\hat{V}=S$. These formulas describe a nonsingular maximum-likelihood estimator when the empirical scatter is positive-definite; if it is singular, there is no maximizer over positive-definite covariance matrices.

Let $n={\sum }_{\nu =1}^g{n}_{\nu }$, ${\bar{x}}_{\nu }=n_{\nu }^{-1}{\sum }_j{x}_j^{(\nu )}$ and $\bar{x}=n^{-1}{\sum }_{\nu }{n}_{\nu }{\bar{x}}_{\nu }$. Define the within-group and between-group scatter matrices byThe within-group and between-group scatter decomposition gives total scatter $T=W+B$: in each group the residuals about ${\bar{x}}_{\nu }$ sum to zero, so all cross terms vanish. Under unrestricted group means, the maximum-likelihood estimators are ${\hat{\mu }}_{\nu }={\bar{x}}_{\nu }$ and $\hat{V}=W/n$. Under the common-mean null hypothesis, they are $\hat{\mu }=\bar{x}$ and ${\hat{V}}_0=(W+B)/n$.

For either fitted scatter matrix $A$, substitution into the log-likelihood givessince $\mathrm{tr}[(A/n{)}^{-1}A]=np$. Thus the maximized likelihood ratio isA generalized likelihood-ratio test rejects for small ${\Lambda }_{\mathrm{LR}}$, equivalentlyThe cutoff $c$ is chosen for the desired significance level. The determinant ratio $|W|/|W+B|$ itself is the Wilks lambda statistic. The derivation assumes $W$ is positive-definite, which for continuous normal samples requires enough within-group degrees of freedom, $n-g\ge p$; singular $W$ does not support this ordinary determinant-based test without a modified model.

The within-group and between-group scatter decomposition separates noise from group separation: $W$ measures variation about each group sample mean, and $B$ measures separation of those means about the overall sample mean. In linear discriminant analysis, seek a projection $a^{T}x$ maximizing between-group relative to within-group variation,Normalize by $a^{T}Wa=1$; a Lagrange multiplier calculation gives the generalized eigenvalue problem $Ba=\lambda Wa$. The leading canonical discriminant directions have the largest eigenvalues, and subsequent directions are orthogonal in the $W$ inner product. There are at most $\min (p,g-1)$ nonzero separating directions. Projecting observations onto these directions gives a low-dimensional display of group separation.

For classification, estimate the common covariance matrix by $S_p=W/(n-g)$ and use the linear discriminant analysis scoreswhere ${\pi }_{\nu }$ are class prior probabilities. Assign to the largest score. The same within-group scatter sets the noise scale, while between-group scatter selects the directions in which the group means separate most strongly.