Past exam of the mathematics course of the University of Cambridge / 2005 / iii / Paper 46 (Applied Multivariate Analysis)

Write the partitioned mean as $({\mu }_1^T,{\mu }_2^T{)}^T$ and assume $V_{22}$ is nonsingular, as required by the displayed inverse. For the conditional multivariate normal distribution, form the residualA linear transformation of a multivariate normal distribution is again normal. Direct calculation givesThe jointly normal blocks $U$ and $X_2$ are therefore independent: their joint characteristic function has no cross term in its quadratic exponent and factors into the two marginal characteristic functions. Conditioning leaves the distribution of $U$ unchanged. Reconstructing $X_1$ proves the full conditional law, including the requested Schur complement covariance:This describes a regular conditional law; it does not require the equality event to have positive probability.

Put $D_1=X_1-Y_1$, $D_2=X_2-Y_2$ andThe difference vector is $D=BZ$. Its mean is zero because the two sons have the same mean vector. Its covariance matrix isFor example, $\mathrm{Cov}(D_1,D_2)=b-c-c+b=2(b-c)$. ThusThe positive definiteness assumption supplies $a-b>0$ and $a+b-2c>0$: these are eigenvalues of the original matrix on within-son contrast and between-son difference directions. In particular the resulting difference covariance is positive definite.

Taking the first marginal of the normal vector in part (i) givesEquivalently its variance is $a+a-2c$; the covariance between the two sons must be retained when taking their difference.

Applying the conditional multivariate normal distribution to the two differences gives, more generally,At $d=0$ the result isBoth this law and the marginal law have mean zero, but the conditional variance is smaller. Specifically, with $\rho =(b-c)/(a-c)$, the conditional variance is the marginal variance times $1-{\rho }^2$. The assumptions imply $0<\rho <1$, so the decrease is strict. Matching the breadth differences to zero gives information about the correlated length differences; it does not force the length differences themselves to be zero. The residual uncertainty remains positive. The original PDF has the between-son differences used here; the self-subtractions in the converted TeX are transcription errors.

Let ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$ be the eigenvalues of the symmetric covariance matrix $V$, and choose an orthonormal basis of eigenvectors $v_1,\dots ,v_p$. If $\ell ={\sum }_j{a}_j{v}_j$ has unit length, then ${\sum }_j{a}_j^2=1$ andEquality is attained by $v_1$, and precisely by unit vectors in the top eigenspace. Hence choose ${\ell }_1$ to be any unit eigenvector for the largest eigenvalue; the maximum variance is ${\lambda }_1$. This proves the maximizing property of the Rayleigh quotient, rather than merely specifying a stationary point.

Extend the selected ${\ell }_1$ to an orthonormal basis of eigenvectors, writing it as $v_1$. For a unit vector orthogonal to it, $a_1=0$, and thereforeA unit eigenvector $v_2$ attains equality. Thus ${\ell }_2$ is a unit eigenvector for the largest eigenvalue on ${\ell }_1^{\perp }$, and the maximum is ${\lambda }_2$. If the largest eigenvalue is repeated, a second orthogonal vector in that eigenspace is allowed and ${\lambda }_2={\lambda }_1$; there is no uniquely determined first pair of directions.

For the sample, put $\bar{x}=n^{-1}{\sum }_i{x}_i$ and use the sample covariance matrixOrder its eigenvalues ${\hat{\lambda }}_j$ decreasingly and choose unit orthogonal eigenvectors ${\hat{\ell }}_j$. The sample principal components are the score columnsTheir sample means are zero and their sample covariance is diagonal, since ${\hat{\ell }}_j^{T}S{\hat{\ell }}_k={\hat{\lambda }}_j{1}_{\{j=k\}}$. The $j$th sample variance is ${\hat{\lambda }}_j$, and its explained variance of a principal component is ${\hat{\lambda }}_j/\mathrm{tr}(S)$. Using divisor $n$ instead changes the numerical variances by a common factor but leaves the component directions unchanged.

Standardizing means replacing a variable with positive sample variance by $(x_{ij}-{\bar{x}}_j)/s_j$. This gives principal component analysis on a correlation matrix, using $C=D^{-1/2}S{D}^{-1/2}$ with $D=\mathrm{diag}(S)$. It removes arbitrary differences of measurement units and prevents a large-variance coordinate from dominating solely through its scale. It also changes the question being optimized: small-variance and possibly noisy coordinates receive equal marginal weight. Principal component analysis based on the covariance matrix can be more appropriate when the variables have common meaningful units and absolute variability matters. Standardization is therefore a substantive choice, not an automatic improvement. A zero-variance variable must first be removed. In the word-rating example the variables share the same numerical range, but their across-word dispersions can differ; standardization gives each attribute equal initial variance.

For the standardized eight-variable analysis, total variance is $\mathrm{tr}(C)=8$. The first three fractions of explained variance of a principal component are approximately 59.6%, 19.1% and 10.1%, respectively. The first two account for about 78.8%, and the first three for 88.9%. The first direction is dominant but one direction alone loses substantial variation. Retaining two principal components is a plausible descriptive summary: their eigenvalues exceed $1$, whereas the third is below $1$. That cutoff is a heuristic, and the third principal component may still have interpretable content.

The reported coefficient rows are not unit-length vectors. The printed rounded rows have squared lengths $4.7514$, $1.5222$ and $0.8063$, consistent with their eigenvalues to the precision of the coefficient table. They are consistent with the scaled principal component loading convention $r_j=\sqrt{{\lambda }_j}{\ell }_j$. Such rows remain eigenvectors, but the score direction with unit length is ${\ell }_j=r_j/\sqrt{{\lambda }_j}$, or more accurately the row divided by its actual Euclidean norm when using rounded entries. Under this convention $r_{ij}$ is the correlation of standardized attribute $i$ with the variance-one score for component $j$:This explains both their sizes and why they must not be treated as the unit vectors from parts (i) and (ii).

The first principal component has positive coefficients on every attribute, strongest on friendliness, goodness, niceness, bravery and strength. With the favorable endpoints coded in the positive direction, it describes a broad favorable or impressive evaluation, with a potency contribution. The second contrasts movement and speed, with some size and strength, against the first three evaluative attributes; it is chiefly an activity dimension. The third is dominated by size, with smaller positive strength and negative movement/speed coefficients, distinguishing size or potency from activity. These are interpretations of directions of variation, not evidence of causal latent traits. Reversing a principal component's overall sign changes neither its meaning as an axis nor its explained variance; endpoint coding fixes only how its positive side is described.

The formula makes species the categorical response and the dot denotes all four other columns as candidate predictors. The classification tree is grown by recursive binary partitioning, not by fitting a linear regression to numerical species codes. At a node $t$, count observations of each class, $n_{tk}$, and set ${\hat{p}}_{tk}=n_{tk}/n_t$. These are the multinomial maximum-likelihood probabilities. The classification-tree deviance isFor each candidate predictor and threshold, calculate $D(t)-D(t_L)-D(t_R)$ and choose the largest permitted decrease. Continue recursively until a node is pure or the stopping controls prevent another split, for example because it has too few observations or insufficient deviance to warrant further growth. Only petal length, petal width and sepal length appear in the selected splits; sepal width was available but did not win a split. This greedy construction need not find a globally optimal tree. The default deviance criterion and recursive partitioning are documented in the tree package manual.

Each node number encodes its position: node $j$ has left and right children $2j$ and $2j+1$. The reported sample size is the number reaching the node; its deviance is calculated from that node's class counts. The fitted class is the class with largest fitted probability, with a convention resolving ties. The root's three counts are equal, so its displayed class is only a tie choice. An asterisk marks a terminal node, where prediction stops.

The resulting rule is displayed below. A left edge satisfies the stated strict inequality and a right edge its complement. None of the recorded training values lies exactly at these midpoint thresholds; the diagram adopts the right branch at equality for prediction.

Its six terminal regions have the following derived class counts, in the order setosa, versicolor, virginica:The root split identifies all 50 setosa observations. Among the remaining specimens, the petal-width split is the principal separation of versicolor from virginica; further petal-length and sepal-length splits refine it. The leaf errors add to $4/150\simeq 2.67\%$. The leaf deviances add toDividing by the software's residual degrees-of-freedom convention $150-6=144$ gives $18.0489/144\simeq 0.1253$, as reported. The root deviance is $-2(150)\log (1/3)=300\log 3\simeq 329.6$. The deviance is a likelihood measure, whereas the error rate counts wrong majority-class predictions; these are different fit summaries.

Both quantities are in-sample, so the low error does not establish prediction accuracy on new specimens. Cost-complexity tree pruning, with tree size chosen by cross-validation, can remove weak splits and reduce overfitting. An independent test set is needed for a final estimate of the selected tree's prediction error.

Multivariate analysis of variance extends analysis of variance to vector responses. For example, a treatment may affect several correlated physiological measurements jointly. Separate univariate tests do not use the correlation geometry and require adjustment if they are interpreted as one multiple-testing procedure. A joint test asks whether group mean vectors differ in any direction.

In a one-way model, take independent $Y_{ri}\sim N_p({\mu }_r,\Sigma )$, $r=1,\dots ,g$, $i=1,\dots ,n_r$, with a common positive-definite covariance. Put $N={\sum }_r{n}_r$, and let ${\bar{Y}}_r$ and $\bar{Y}$ be group and overall sample means. The within-group and between-group scatter matrices areExpanding around each group mean makes the cross terms vanish, giving total scatter $E+H$. To test $H_0:{\mu }_1=\cdots ={\mu }_g$, use the Wilks lambda statisticwhere ${\theta }_j$ solve the generalized eigenvalue problem $Hv=\theta Ev$ and $E$ is nonsingular. Under the alternative, maximize the normal likelihood at the group means and covariance $E/N$; under the null, use the overall mean and covariance $(E+H)/N$. Their likelihood ratio is ${\Lambda }^{N/2}$, so small $\Lambda$ rejects equal means. Under the null, orthogonal projections of the Gaussian data give independent $E\sim W_p(\Sigma ,N-g)$ and $H\sim W_p(\Sigma ,g-1)$, supplying its null calibration. Here $E$ is nonsingular almost surely when $N-g\ge p$. Merely having more total observations than responses is not enough if group fitting consumes too many statistical degrees of freedom.

The test is invariant under a nonsingular linear change of response coordinates: both determinants acquire the same factor $|A{|}^2$. The directions associated with large ${\theta }_j$ identify mean separation relative to residual variability. With one response it becomes the ordinary ANOVA F-test. With two groups, writing $S_p=E/(N-2)$ and $d={\bar{Y}}_1-{\bar{Y}}_2$, the version using Hotelling's T-squared statistic iswith $N>p+1$. This formula demonstrates explicitly how the covariance weights a group difference.

The sketch shows hypothetical groups with strongly positively correlated responses and mean separation along their low-variance contrast. Marginal comparisons can look weak even when the joint comparison is informative. The ellipses represent individual-response covariance, not confidence regions for the means. Independence of observations, approximate multivariate normality and common covariance are substantive assumptions; outliers and unequal covariance can invalidate the usual calibration. Follow-up contrasts can identify the affected responses or groups, with appropriate multiplicity control. MANOVA uses shared covariance to test a multivariate mean difference, rather than testing unrelated coordinates in isolation.

Factor analysis models a large set of correlated measurements using a smaller collection of shared latent factors and variable-specific variation. A classical orthogonal factor model isTake $q<p$, centered errors, and positive specific variances for the inverse-covariance and likelihood formulas below. The factor loadings form the $p\times q$ matrix $L$, and the model covariance isThus for $i\ne k$, $\mathrm{Cov}(X_i,X_k)={\sum }_j{L}_{ij}{L}_{kj}$; the common latent factors explain off-diagonal correlation. The communality of variable $i$ is $h_i^2={\sum }_j{L}_{ij}^2$, and $\mathrm{Var}(X_i)=h_i^2+{\psi }_i$. For standardized variables this is $1=h_i^2+{\psi }_i$. The specific variance represents measurement noise or genuinely unshared variation; it is not another common latent factor.

This differs from principal component analysis. A principal component maximizes projected total variance and is a determined linear score after choosing its direction; a factor model attempts to explain shared covariance while reserving diagonal specific variance. Low-dimensional PCA truncation can help initialize a factor fit, but it does not justify setting all specific variances to zero. Latent factors are unobserved, not directly observed scores. When the latent factors and the errors are jointly Gaussian, their conditional mean iswhich supplies one factor-score estimate, with remaining uncertainty. This follows from the block conditional normal formula in Question 1; it does not make the latent factor uniquely recoverable from one observation.

For the Gaussian likelihood, use the empirical covariance $S_0=n^{-1}{\sum }_i(x_i-\bar{x})(x_i-\bar{x}{)}^T$. Fit the factor model by minimizing $\log |L{L}^T+\Psi |+\mathrm{tr}(S_0(L{L}^T+\Psi {)}^{-1})$ subject to admissible specific variances. Other approaches estimate communalities and extract latent factors from the reduced correlation matrix. Choose $q$ using substantive plausibility, a scree plot, fitted residual correlations and model-fit assessment. Under local identifiability, a likelihood goodness-of-fit count for the covariance model iswhere the subtracted rotational dimension must be removed from the loading parameter count. This count alone does not establish identifiability, and boundary fits with zero specific variance need special care.

Factor rotation addresses interpretability and nonuniqueness. For every orthogonal matrix $Q$, replace $(L,F)$ by $(LQ,Q^{T}F)$; then $(LQ)(LQ{)}^T=L{L}^T$, so the fit is unchanged. The absolute orientation of unrestrained latent factors is therefore not identified by the covariance. Varimax rotation chooses orthogonal axes favoring a simple pattern of large and small squared loadings. Oblique rotation can permit correlated latent factors, provided their covariance is transformed consistently. Sign changes and permutations are further harmless relabelings; an interpretation must state its chosen convention.

The sketch uses four standardized hypothetical measurements sharing two orthogonal latent factors. Rotating the factor coordinates changes the loading descriptions while preserving every implied covariance and communality; the scree plot belongs to that same constructed covariance matrix. Factor analysis separates shared variation from specific variance, and its latent factors require an identifying or interpretive orientation.

Cluster analysis groups observations without given class labels. It starts by deciding what similarity means, usually through a dissimilarity matrix. Euclidean distance is natural for comparable continuous coordinates; a Mahalanobis distance can account for differing scales and correlations when a meaningful nonsingular covariance estimate is available. Scaling and variable selection can change the groups. Binary, ordinal or categorical data may need a different dissimilarity, rather than arbitrary numerical codes and Euclidean distance.

Agglomerative hierarchical clustering begins with singleton groups, repeatedly merges the pair with smallest intergroup dissimilarity, and records the merges in a dendrogram. The height represents the algorithm's merge criterion. Cutting the dendrogram gives partitions at different resolutions, and merges cannot subsequently be reversed. In single-linkage clustering, the criterion is the nearest cross-pair distance; this detects chains and nonconvex groups, but a thin bridge can connect otherwise separated clouds. Complete-linkage clustering uses the farthest cross-pair distance and tends to produce compact groups, at the cost of sensitivity to extreme points. Average-linkage clustering uses the mean cross-pair distance, lying between these two criteria for a given pair of clusters.

For squared Euclidean geometry, Ward clustering merges the pair that least increases the within-group sum of squares. Expanding deviations about the merged mean givesThis favors low within-group dispersion, but interpreting it as a sum-of-squares increase requires Euclidean data. Different software may plot a rescaled height, so the vertical units of a dendrogram from Ward clustering must be stated.

A nonhierarchical alternative is k-means clustering. For a prescribed $K$, minimizeAt fixed assignments, expanding squared distances or differentiating shows that the minimizing $m_k$ is the group mean. At fixed centroids, each point is assigned to a nearest centroid. Alternating these operations does not increase $W_K$; with a consistent tie rule and nonempty groups it reaches a locally stable partition. Initialization matters, so use multiple starts, and give an explicit rule for empty groups. It favors roughly spherical clusters and is sensitive to outliers and scaling. It cannot be expected to reproduce a nonconvex single-linkage partition.

The sketch displays a constructed point configuration and the complete-linkage merges computed from its actual Euclidean distances. A cut between the within-pair and between-pair merge heights yields three groups. Such visual separation is useful evidence, not proof of a uniquely correct number of populations. Select a resolution using subject knowledge, stability under resampling, separation diagnostics or the change of $W_K$ with $K$. Clustering always gives a partition even for unstructured data, and different distance or linkage choices can yield different sensible summaries. Clustering is an exploratory construction whose conclusions depend on the geometry and the chosen resolution.