Past exam of the mathematics course of the University of Cambridge / 2006 / iii / Paper 46 (Applied Multivariate Analysis)

Let $Z=(Z_1,\dots ,Z_p{)}^T$. Independence of the standard normal distributions gives $\mathbb{E}Z=0$, $\mathrm{Cov}(Z)=I_p$, and joint densityChoose a matrix $C$ with $C{C}^T=\Sigma$, for example the lower-triangular factor from the Cholesky decomposition. It exists and is nonsingular because $\Sigma$ is a symmetric positive-definite matrix. Then the required simulation isIt has mean $\mu$ and covariance matrix $C{C}^T=\Sigma$, and every linear combination of its coordinates is Gaussian, so it has the required multivariate normal distribution.

For the multivariate normal density, apply the change of variables formula to $z=C^{-1}(x-\mu )$. Its absolute Jacobian determinant is $|\det C{|}^{-1}=|\Sigma {|}^{-1/2}$, andHenceThe nonsingularity assumption is needed for a density with respect to $p$-dimensional Lebesgue measure.

Take the coordinate-selection matrixThen $A^{T}X=X_1$, $A^T\mu ={\mu }_1$ and $A^T\Sigma A={\Sigma }_{11}$. The permitted result on a linear image of a multivariate normal vector therefore givesThis is the marginal distribution obtained by discarding the other coordinates. The conclusion holds also for a positive-semidefinite marginal covariance: in that case the Gaussian law may be degenerate and need not have a full-dimensional density. If the original covariance is positive definite, its principal block ${\Sigma }_{11}$ is positive definite as well.

Assume an independent Gaussian sample, positive-definite population covariance, and $n>p$. Define the sample mean and unbiased sample covariance matrix byThe standard test is based on Hotelling's T-squared statisticReject the null hypothesis for large values. Under the null its exact distribution isThe factor uses the unbiased, divisor-$n-1$ covariance convention. Normal-sample theory makes $\bar{X}$ independent of the residual covariance, whose whitened version has a Wishart distribution; this yields the stated F-distribution. The condition $n>p$ gives invertibility of $S$ almost surely and positive denominator degrees of freedom.

This is also the likelihood-ratio test. To see the connection, put $W=(n-1)S$ and $\delta =\bar{X}-{\mu }_0$. The maximized covariance estimates under the unrestricted and null models are $W/n$ and $W/n+\delta {\delta }^T$, respectively. The determinant identity for a rank-one update givesThus the likelihood ratio is $(1+T^2/(n-1){)}^{-n/2}$, decreasing in $T^2$. The term best is understood here as this standard likelihood-ratio, affine-invariant procedure; a two-sided multidimensional alternative does not specify a single direction to optimize power against.

For the transformed sample $Y_i=A{X}_i+b$, with $A$ the prescribed nonsingular coordinate transformation, we haveConsequentlyThis proves the affine invariance of Hotelling's statistic. The statistic, its p-value and its test decision are unchanged by a nonsingular affine transformation, provided the null mean is transformed with the data. Changes of units and rotations are included.

The inequality in the printed rule is reversed. With the stated definition of $L$, a larger score favors group one. For example, in one dimension take ${\mu }_1=1$, ${\mu }_2=-1$, $\Sigma =1$, equal priors and $x=1$. The score is $2$, and $f_1(1)>f_2(1)$, whereas the printed rule assigns group two. The correct Bayes classifier follows from minimizing the error cost as follows.

Let $R$ be the region assigned to group one, let $f_i$ be its multivariate normal density, and let each misclassification cost $c>0$, with zero cost for correct classification. The expected cost isThe permitted integral-minimization result therefore selects $R=\{x:{\pi }_1{f}_1(x)>{\pi }_2{f}_2(x)\}$. Points of equality may be assigned to either group without changing the risk. Equivalently, the Bayes classifier chooses the larger posterior probability.

For positive priors and a common positive-definite covariance matrix, the normalization constants cancel in the likelihood ratio. Expanding the two quadratic forms givesThus the correct Gaussian Bayes classifier, choosing group one on ties, isThis proves the intended classification result with the necessary correction. Equal error costs remove $c$ from the decision, while prior odds determine the threshold.

Use a multivariate normal distribution within each species, a common positive-definite covariance matrix, equal prior odds and equal misclassification costs. Assume the training observations are independent and representative of the populations. Substituting the sample means and pooled sample covariance matrix gives a fitted linear discriminant analysis rule, rather than a rule with known population parameters.

The determinant of the pooled covariance is $8$, soThe midpoint of the sample means is $(4,6{)}^T$, whose scalar product with $L$ is $-11/2$. Hence the fitted log likelihood ratio isAssign fattus when $q(x)>0$ and apathus when $q(x)<0$; either assignment is optimal on $q(x)=0$ under equal priors. The separating line is $x_2=0.4x_1+4.4$, with fattus below it. As a sign check, the scores at the two sample means are $19/4$ and $-19/4$, respectively. This uses the corrected Gaussian Bayes classifier from (a).

Evaluating the fitted linear discriminant analysis score givesThe fifth individual lies exactly on the discriminant boundary. Its two fitted posterior probabilities are equal, so the equal-prior Bayes classifier cannot prefer either species. A specified tie convention could assign it to one species, but such an assignment would not be evidence favoring that species.

In the sketch the solid line is the equal-prior boundary. The dashed line is the prior-dependent Gaussian discriminant boundary for the altered priors in (iii).

Now the prior odds are ${\pi }_f/{\pi }_a=2$. The fitted log posterior odds are $q(x)+\log 2$, so the Gaussian Bayes classifier assigns fattus when $q(x)>-\log 2$. The prior-dependent Gaussian discriminant boundary becomesIt moves upwards parallel to the original line by $0.8\log 2\simeq 0.5545$, expanding the fattus region. This is the dashed line in the sketch. The fifth individual's fitted log likelihood ratio remains zero, but its posterior odds become $2:1$, giving posterior fattus probability $2/3$.

The fifth individual is now classified as fattus. For completeness, the other four classifications remain apathus, fattus, apathus and fattus: their scores lie on the same side of the new threshold as before.

Principal component analysis replaces the original coordinates by orthogonal linear combinations, ordered so that the first few retain as much variance as possible. It gives a lower-dimensional summary for visualization or compression and reveals the main directions of variation. A direction of large variance need not be the direction most useful for classification.

Write $\mu =\mathbb{E}X$. For a unit vector $u$, the centered score $Y=u^T(X-\mu )$ has varianceMaximizing this Rayleigh quotient with a Lagrange multiplier givesThus a maximizing direction must be an eigenvector of the covariance matrix. To identify the maximum, use the spectral theorem for real symmetric matrices to choose an orthonormal basis of eigenvectors $v_1,\dots ,v_p$, with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. Writing $u={\sum }_j{c}_j{v}_j$ givesEquality is attained at $u=v_1$. Consequently the first population principal component isFor each subsequent principal component, maximize the same variance over unit coefficient vectors orthogonal to all the previously chosen ones. Restricting the expansion above to the remaining eigenvectors gives $v_2$, then $v_3$, and so on. ThusThe principal components are uncorrelated; they need not be independent unless additional distributional assumptions, such as joint Gaussianity, hold. Their total variance is $\mathrm{tr}\Sigma ={\sum }_j{\lambda }_j$. Signs of eigenvectors are arbitrary, and a repeated eigenvalue permits any orthonormal basis in its eigenspace. In sample principal component calculations, replace $\mu$ by the sample mean and $\Sigma$ by the sample covariance matrix.

For $v_1=(1,1,1,0{)}^T$, direct multiplication by the sample covariance matrix givesIt is therefore an eigenvector, with eigenvalue $2250$. Normalization gives $u_1=v_1/\sqrt{3}$, so the corresponding sample principal component isIts coefficients are proportional to the required vector. The remaining eigenvalues calculated in (ii) are smaller, confirming that this is the first principal component.

Let $v_2,v_3,v_4$ be the three supplied coefficient vectors, in their displayed order. Multiplication by the sample covariance matrix givesThe four vectors are pairwise orthogonal, with norms $\sqrt{3}$, $\sqrt{102}$, $\sqrt{6}$ and $\sqrt{204}$. Hence $u_j=v_j/\Vert v_j\Vert$ form an orthonormal basis of eigenvectors. The variance of the principal component with unit coefficient vector $u_j$ is its eigenvalue ${\lambda }_j$. Using unnormalized coefficient vectors would instead give variances ${\lambda }_j\Vert v_j{\Vert }^2$ and would not produce the desired proportions.

The total variance isTherefore the explained variance of a principal component isThe first two principal components together account for $2658/3162\simeq 84.06\%$, and the first three account for $2958/3162\simeq 93.55\%$ of the total variance.

The first principal component has equal positive coefficients on the three academic subjects and zero coefficient on physical education. It measures overall academic performance in those subjects and explains $71.16\%$ of the variance. The large positive academic covariances explain why these subjects vary predominantly together.

The second principal component has normalized coefficients approximately $(0.099,0,-0.099,0.990)$. It is mainly a physical-education score, with a small adjustment for the physics-versus-mathematics contrast, and explains $12.90\%$ of the variance. The small physical-education covariances with the academic subjects make this direction close to a separate physical-education axis. This is an interpretation of the observed covariance matrix, not proof of distinct causal abilities or statistical independence.

The third principal component compares physics and mathematics together against biology: its coefficient vector is $(1,-2,1,0)/\sqrt{6}$. It captures subject balance rather than overall academic performance, accounting for $9.49\%$ of the variance. The fourth principal component primarily contrasts physics with mathematics, with a smaller physical-education correction; its normalized coefficients are approximately $(0.700,0,-0.700,-0.140)$, and it accounts for the remaining $6.45\%$. Changing all signs in any component reverses its score without changing the interpretation of its contrast or its explained variance of a principal component.

A two-component summary separates broad academic performance from predominantly physical-education performance and retains about $84.06\%$ of the variation. It still discards about $15.94\%$, including differences between academic subjects. Whether that loss is acceptable depends on the purpose: an overall summary may use two principal components, whereas studying subject-specific strengths requires the later principal components too. A single principal component would lose most of the physical-education information.

If the change is purely a rescaling of the physics mark, write $X^{\prime }=DX$, where $D=\mathrm{diag}(10/3,1,1,1)$. The sample covariance matrix transforms asThe physics variance is multiplied by $100/9$, becoming approximately $10044.44$. Each covariance between physics and another subject is multiplied by $10/3$, and all other entries stay unchanged. Principal component analysis on this new sample covariance matrix maximizes variance in the new numerical units, so its eigenvectors and explained variance of a principal component generally change. Physics receives disproportionately greater influence simply because its unit has changed.

One solution is to convert the physics marks back to the original scale before doing principal component analysis. Alternatively, use principal component analysis on a correlation matrix: standardize every subject byA positive change of units multiplies both a centered variable and its standard deviation by the same factor. Thus it leaves $Z_j$ and the sample correlation matrix $R$ unchanged.

Correlation-based principal component analysis is invariant under positive changes of units; covariance-based principal component analysis is not. Standardizing gives each subject unit variance, which is a different weighting choice and can emphasize relatively low-variance subjects. If the revised assessment changes the scores themselves rather than just their units, its sample correlation matrix may also change, and mere standardization cannot recover the original analysis.

Write $x_r,x_t$ for two rows of the data matrix and $\delta =x_r-x_t$. Three possible dissimilarities for continuous variables are the following.

The Euclidean distance isIt is simple and has the usual geometric interpretation when coordinates have comparable units. Its disadvantage is sensitivity to the numerical scales: a change of units or a high-variance variable can dominate it. Strongly correlated coordinates can also give repeated weight to essentially the same information. Squared Euclidean distance is useful in some methods, but is not itself a metric because it need not satisfy the triangle inequality.

The standardized Euclidean distance iswhere $s_j^2$ is the sample variance of coordinate $j$ across individuals. It measures differences in standard-deviation units and is invariant under positive coordinate rescalings when the scales are estimated consistently. It is useful for differently measured variables, but requires nonzero sample variances, ignores correlations between coordinates and can give excessive influence to a low-variance noisy variable. With fixed positive scales it is a Euclidean distance after a diagonal change of coordinates, so it is a metric.

The Mahalanobis distance iswhere $S$ is the pooled sample covariance matrix of all individuals. If $S=C{C}^T$ is its Cholesky decomposition, then $d_M=\Vert C^{-1}\delta {\Vert }_2$: it is Euclidean distance in whitened coordinates. It adjusts for both scales and correlations, avoiding duplicate weight for highly correlated variables, and is invariant under nonsingular affine transformations when the sample covariance matrix is transformed accordingly. No multivariate normal distribution assumption is required merely to define the distance. However, $S$ must be a positive-definite matrix; when there are too few observations or dependent coordinates, its inverse is unavailable. Estimation of $S^{-1}$ can also be unstable and sensitive to outliers. These trade-offs determine whether covariance adjustment is preferable to the simpler distances.

A metric dissimilarity $d$ must satisfy nonnegativity, separation, symmetry and the triangle inequality:The underlying objects must be specified: here they are binary measurement profiles, rather than labels attached to individuals.

Let $n_{11},n_{00},n_{10},n_{01}$ count coordinate pairs of the indicated types. The simple matching coefficient is $S=(n_{11}+n_{00})/4$, so its complementary dissimilarity is $(n_{10}+n_{01})/4$. Counting mismatches in every pair gives the dissimilarity matrixFor the Jaccard coefficient, joint absences are excluded: $J=n_{11}/(n_{11}+n_{10}+n_{01})$. Thus the Jaccard distance is $1-J$, givingOnly the pair of individuals two and four has a joint absence. Their profiles have one shared presence, two mismatched presences and one joint absence, giving dissimilarities $2/4$ and $2/3$, respectively. All other pairs have a presence in at least one profile at every coordinate, so the two denominators coincide.

For the requested metric property of simple matching dissimilarity, write a general pair of $p$-coordinate binary profiles as $x,y$. Thenwhich is the normalized Hamming distance. It is nonnegative and symmetric, and equals zero exactly when all coordinates match. For each coordinate, a mismatch between $x_j$ and $z_j$ forces a mismatch between $x_j,y_j$ or between $y_j,z_j$. ThereforeSumming and dividing by $p$ proves the triangle inequality, establishing all four metric properties. Simple matching dissimilarity is a metric on binary profiles. If two separately labelled individuals have identical profiles, it gives zero distance between them, so on those labels it is only a pseudometric. The four profiles here are distinct, so this distinction does not prevent $D_S$ from defining a metric on the given sample.

There is a defect in the stated geometric interpretation of the data. The supplied numbers cannot be Euclidean distances, becauseThey violate the triangle inequality; independently, $d(C,D)=10.3>5.9+1.2$. Nevertheless, both clustering algorithms are defined for this symmetric dissimilarity matrix. The calculations below retain the supplied numbers and use agglomerative clustering of nonmetric dissimilarities, rather than silently inventing Euclidean replacement data.

In single-linkage clustering, the intercluster dissimilarity isThe closest pair is $A,D$, merging at $1.2$. Among the remaining clusters the next closest pair is $B,E$, merging at $2.6$. The three intercluster dissimilarities are nowConsequently $AD$ merges with $BE$ at $4.2$, using the cross-pair $A,B$. The remaining point $C$ joins at $\min (5.9,7.6,10.3,5.4)=5.4$, using $C,E$. The dendrogram thus hasThe close pairs $AD$ and $BE$ are the clearest early groupings. A cut between $2.6$ and $4.2$ produces these two pairs and the singleton $C$; a cut between $4.2$ and $5.4$ produces $ADBE$ and $C$. Single-linkage clustering builds chains through close cross-pairs, so the merge at $4.2$ does not mean every pair in $ADBE$ is that close: $d(D,E)=7.8$. The data alone do not prescribe a unique number of clusters.

For complete-linkage clustering, replace the minimum by the maximum:The first two merges remain $AD$ at $1.2$ and $BE$ at $2.6$. SubsequentlyThus $C$ joins $BE$ at $7.6$. The final maximum between $AD$ and $BCE$ is $d(D,C)=10.3$, givingBoth the topology and the later heights of the dendrogram change: complete-linkage clustering attaches $C$ to $BE$ before joining that group to $AD$, rather than merging the two pairs first. Its control of the largest within-cluster separation makes it less susceptible to the chaining seen with single-linkage clustering.