Past exam of the mathematics course of the University of Cambridge / 2007 / iii / Paper 47 (Applied Multivariate Analysis)

Use the unbiased sample covariance matrixand assume $\Sigma$ is positive-definite and $n>p$, so $S$ is invertible almost surely. For a nonzero fixed direction $a$, the observations $Y_j=a^T{X}_j$ have mean $a^T\mu$ and variance $a^T\Sigma a$. Their squared Student t-test statistic for the projected null mean isPut $b=\bar{X}-{\mu }_0$. The Cauchy-Schwarz inequality giveswith equality when $a$ is proportional to $S^{-1}b$, if $b\ne 0$. For $b=0$ every projected statistic is zero. Maximizing the squared discrepancy over directions therefore gives Hotelling's T-squared statistic:This uses the squared generalized Rayleigh quotient. The preliminary formula on the printed cover has an unsquared numerator and is not valid as written: setting $a=t{C}^{-1}b$ makes that ratio $1/t$ for $b\ne 0$, so it is unbounded. The displayed Cauchy-Schwarz argument supplies the needed corrected identity independently.

Under the null, let $Z=\sqrt{n}(\bar{X}-{\mu }_0)$ and $C=(n-1)S$. Then $Z\sim N_p(0,\Sigma )$, $C\sim W_p(n-1,\Sigma )$, and $Z,C$ are independent. To see these sample properties, apply an orthogonal transformation to the $n$ observation indices with first row $n^{-1/2}(1,\dots ,1)$. Its first centered row is $Z$; its other $n-1$ rows are independent $N_p(0,\Sigma )$ residual contrasts, independent of the first. Their outer products sum to $C$, establishing the Wishart distribution and independence.

Since $T^2=(n-1)Z^T{C}^{-1}Z$, use the supplied Gaussian-Wishart quadratic-form law with $k=n-1$ to obtainThus a level-$\eta$ test rejects when this scaled statistic exceeds the $(1-\eta )$ quantile of the F-distribution. The Gaussian-Wishart law requires independence; it holds here because of the orthogonal sample decomposition. Its printed use of $X$ instead of the introduced $Z$ is a notational slip. A fixed-direction Student law alone would not calibrate the maximized statistic, because its maximizing direction depends on the sample.

Transform each observation to $Y_j=A^T{X}_j$. If $A^T$ has full row rank $m$, the transformed sample is multivariate normal with mean $A^T\mu$ and positive-definite covariance matrix $A^T\Sigma A$. Its sample mean is $A^T\bar{X}$ and its unbiased sample covariance matrix is $A^{T}SA$. The Hotelling test of linear hypotheses therefore usesHere $n>m$ is required. Reject for a large scaled statistic using the relevant upper F-distribution quantile. If the rows of $A^T$ are dependent, choose a basis for their row space, use that basis as the contrast matrix, and replace $m$ by its rank $r$. The null hypothesis is unchanged, but redundant contrasts must not be treated as independent dimensions. Rank zero gives no restrictions and hence no nontrivial test.

At equal spacing $h$, the adjacent increments of an affine mean profile are all $\beta h$. Their differences are therefore zero. Define the $(p-2)\times p$ matrix $B$ whose row associated with the interior time $i$ has coefficients $1,-2,1$ in columns $i-1,i,i+1$. ThenConversely these equations make all adjacent first differences equal, so ${\mu }_i$ is affine in $i$, equivalently affine in the equally spaced times. Thus the restriction is exactly $B\mu =0$, not just a necessary condition. Its null space has basis $1$ and $t=(t_1,\dots ,t_p{)}^T$, so $B$ has rank $r=p-2$.

For $p\ge 3$ and $n>r$, the second-difference test of a linear mean profile is the Hotelling test of linear hypotheses withReject the affine-profile hypothesis when the scaled statistic exceeds its level-$\eta$ upper F-distribution critical value. This removes the nuisance intercept and slope without estimating them separately.

For unequal ordered distinct times, put $h_i=t_{i+1}-t_i>0$ and replace each second-difference row by the adjacent-slope contrastIts three coefficients are $1/h_{i-1}$, $-(1/h_{i-1}+1/h_i)$ and $1/h_i$. Vanishing means all adjacent slopes agree, again giving null space $\mathrm{span}(1,t)$ and rank $p-2$. Use this new $B$ in the same statistic and null distribution. For $p\le 2$, every mean profile at distinct times is affine, so there is no lack-of-linearity restriction to test.

For an integrable real function $g$, let $R_{-}=\{x:g(x)<0\}$. For every measurable region $R$,The first term is nonnegative because $g\ge 0$ there, and the integral subtracted in the second term is nonpositive. Hence the negative set minimizes the integral. Including or excluding any points where $g=0$ has no effect; changes on null sets do not matter either. This is the minimum-integral decision region principle. Integrability ensures the subtraction is defined, and it holds automatically for the weighted differences of probability density functions used below.

There are two disjoint ways to make an error: an individual from class one falls outside $R_1$, or an individual from class two falls inside $R_1$. The law of total probability givesSince $f_1$ integrates to one, this also equalsThis form isolates the only term changed by choosing the decision region.

Apply the minimum-integral decision region principle to $g(x)={\pi }_2{f}_2(x)-{\pi }_1{f}_1(x)$. The optimal Bayes classifier chooses class one when ${\pi }_1{f}_1(x)>{\pi }_2{f}_2(x)$, and class two for the reverse inequality. Ties can be assigned either way. An everywhere valid discriminant score isWhere both weighted densities are positive one may equivalently use $\log ({\pi }_1{f}_1(x)/({\pi }_2{f}_2(x)))$ and threshold zero. This chooses the largest posterior class probability, because both posterior probabilities have the same positive mixture-density denominator.

With zero cost for correct assignments, the total expected misclassification cost isThe part depending on $R_1$ is the integral of $c(1\mid 2){\pi }_2{f}_2-c(2\mid 1){\pi }_1{f}_1$. Therefore the cost-sensitive Bayes classifier isFor positive costs and weighted densities, the log posterior-odds score from part (ii) now has threshold $\log (c(1\mid 2)/c(2\mid 1))$. Larger cost of sending class one to class two expands the region assigned to class one. The weighted-density comparison also handles zero costs without taking an undefined logarithm.

For common positive-definite covariance, subtracting the two Gaussian log densities cancels the quadratic term. Equal priors leave the linear discriminant analysis scoreHereThusAssign class two if the expression is negative; a tie has zero probability under either continuous Gaussian law.

To compute the error rather than just specify the boundary, put $D^2={\Delta }^T{\Sigma }^{-1}\Delta =27/2$. Within class one the score has mean $D^2/2=27/4$, and within class two mean $-27/4$. In both classes its variance is ${\Delta }^T{\Sigma }^{-1}\Sigma {\Sigma }^{-1}\Delta =D^2$. Hence both conditional error probabilities are $\Phi (-D/2)$, where $\Phi$ is the standard normal distribution function. Equal priors give the equal-covariance Gaussian classification errorFor the specified observation, the score is $\delta (2,1)=27/4>0$, so assign it to class one.

Principal component analysis replaces correlated measurements by uncorrelated linear compounds ordered by decreasing variance. Keeping the leading compounds gives a low-dimensional summary that preserves as much variance as possible under orthonormal projection. It is an unsupervised description of variation, rather than a guarantee that the high-variance directions are best for classification or prediction.

Write $X_c=X-\mu$. For a unit loading vector $q$, the variance of $q^T{X}_c$ is $q^T\Sigma q$. The spectral theorem gives an orthonormal eigenbasis $q_1,\dots ,q_p$ with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. If $q={\sum }_j{b}_j{q}_j$ and ${\sum }_j{b}_j^2=1$, thenEquality is attained at $q_1$. Requiring the next loading to be orthogonal to the first excludes its eigendirection, so the same argument gives $q_2$ and variance ${\lambda }_2$, and continues inductively. Repeated eigenvalues permit any orthonormal basis of the tied eigenspace. These are the population principal components.

With $Q=(q_1,\dots ,q_p)$, set $Y=Q^T{X}_c$. ThenIn particular the components are uncorrelated. They need not be independent unless, for example, $X$ is multivariate normal. FinallyThe trace equality follows from $Q^{T}Q=I$ and cyclicity of trace, so the rotation preserves total variance. The explained variance of a principal component is its eigenvalue divided by this total.

The given equicorrelation covariance matrix can be writtenOn $1=(1,1,1,1{)}^T$, the rank-one matrix $11^T$ acts as multiplication by four. On the three-dimensional orthogonal complement $\{v:{\sum }_j{v}_j=0\}$ it acts as zero. Hence the eigenvalues are $1+3r$ and $1-r$ with multiplicity three.

For $0<r<1$, the largest eigenvalue is $1+3r$ and its normalized eigenvector is $1/2$. Therefore the first population principal component isOmitting the centering merely changes the component by a constant and has no effect on its variance. Since $\mathrm{tr}\Sigma =4$, its explained fraction isIt rises from one quarter towards one as the positive common correlation increases. Outside the specified range, a valid covariance requires $-1/3\le r\le 1$; for negative $r$ the sum direction is no longer the leading one.

Covariance-based PCA uses centered protein measurements in their original gram units. Its total variance is $218.4$. Correlation-based PCA first divides each centered food measurement by its sample standard deviation, giving nine unit-variance variables and total variance nine. Thus it is principal component analysis on a correlation matrix, not an equivalent rotation of the original unscaled measurements.

The variances of cereals and milk together account for $170.9/218.4\approx 78.3\%$ of the original total. Consequently those variables have considerable influence on covariance-based PCA. Its first component is dominated by a cereal coefficient $-0.86$ opposed to milk $0.42$, broadly a cereal-versus-milk contrast. Its second is dominated by milk $0.83$, with cereals $0.40$ and fish $-0.29$. Its third is chiefly white meat $0.80$ versus fish $-0.52$, with a smaller negative milk term. Component signs may all be reversed without changing the analysis; the important features are relative signs and magnitudes.

After standardization, eggs and starch can contribute as strongly as large-variance foods. The first correlation-based component gives positive weight to meat, eggs, milk and starch, opposed principally to cereals and pulses/nuts; eggs now has coefficient $0.43$ despite its small raw variance. The second emphasizes fish and fruit/vegetables negatively, opposed especially to starch and white meat. The third contrasts white meat and fruit/vegetables with milk, fish and red meat. Thus standardization changes both the emphasis and the directions, rather than just the displayed units of the same components.

The explained variance of a principal component must use the appropriate total:Raw and standardized percentages describe different variance objectives and must not be compared as if one method were always superior.

Standardize when relative patterns across variables, rather than their absolute gram-scale variation, should have equal marginal weight. Correlation-based PCA is also insensitive to changing a variable's measurement unit, which covariance-based PCA is not. Here every food group already has the same units, so differing units do not force standardization. If large fluctuations in actual protein amounts are scientifically important, covariance-based PCA is appropriate. If dietary composition across all food groups is the focus, correlation-based PCA can be more informative. Standardization can nevertheless amplify noise in nearly constant variables; the decision should follow the scientific purpose and measurement reliability.

The three retained covariance components explainleaving only $7.7\%$ of the raw total variance. Three components are a reasonable covariance-based descriptive compression, if that amount of discarded variation is acceptable for the intended use. Even two explain about $85.1\%$, so three is a useful choice rather than a uniquely forced one.

For the correlation analysis,so three retain $75.3\%$ and discard $24.7\%$ of standardized variation. Three can provide an interpretable summary, but the supplied information does not justify treating them as sufficient for every purpose. Tasks requiring high standardized reconstruction accuracy may need more.

Each of the three displayed correlation eigenvalues exceeds the average eigenvalue one, but that alone does not show all remaining eigenvalues are below one. Their sum is $9-6.78=2.22$, and a fourth eigenvalue just above one is compatible with both that sum and the ordering. A complete eigenvalue sequence, a plot of successive eigenvalues, and the stability and interpretation of the directions would help decide the cutoff. With only 25 countries, estimated directions also have sampling uncertainty. The retained covariance and correlation dimensions should therefore be judged against their own variance criterion and substantive aim, not chosen solely because three columns were printed.

For rows $x_i,x_j$, three useful choices are Euclidean distance, standardized Euclidean distance, and Mahalanobis distance:The Euclidean choice is simple, rotation invariant and appropriate when the original coordinate scales carry comparable meaning. It is sensitive to scale changes, may be dominated by a high-variance variable and gives correlated or duplicate variables repeated influence. Squared Euclidean dissimilarity is sometimes used algorithmically, but removing the square root generally loses the triangle inequality.

The standardized Euclidean choice removes marginal units using fixed positive sample scales $s_r$. It gives equal marginal weight to variables, but ignores correlations and can magnify noise when an estimated variance is small. Constant variables require omission or an explicitly chosen positive external scale.

The Mahalanobis choice incorporates the full positive-definite sample covariance $S$, so it is Euclidean distance after whitening. It accounts for both scale and redundancy and is invariant under an invertible change of coordinates when $S$ is transformed consistently. However estimating $S$ requires enough observations and can be sensitive to extreme values; inversion is unstable near singularity and unavailable when $S$ is singular. With fixed positive scales or positive-definite $S$, all three square-root expressions are genuine metrics on the measurement space. Their different geometries encode different notions of which individuals should be considered similar.

A metric dissimilarity must satisfy, for all $x,y,z$, the four propertiesThey are nonnegativity, identity of indiscernibles, symmetry and the triangle inequality. The following two transformations of the simple matching coefficient behave differently at the identity.

For fixed $p\ge 1$, the complement of the simple matching coefficient is the fraction of coordinates that disagree:This is the normalized Hamming distance. It is nonnegative, symmetric, and is zero exactly when every coordinate agrees. At each coordinate,Summing and dividing by $p$ proves the triangle inequality. Therefore $d_1$ is a metric on the binary vectors.

For every binary vector $x$, $S_1(x,x)=1$, and henceThus $d_2$ is not a metric, because it fails identity of indiscernibles. It is nonnegative and symmetric. In fact it also satisfies the triangle inequality trivially: every value lies in $[1/2,1]$, so $d_2(x,z)\le 1\le d_2(x,y)+d_2(y,z)$. That does not repair its nonzero self-distance.

In agglomerative hierarchical clustering, start with eight singleton groups. Single-linkage clustering uses the smallest distance between cross-pairs; complete-linkage clustering uses the largest. Updating these cluster distances after each merge, one valid order of the tied merges givesThe merges at height $0.6$ may be reordered; they give the same requested three-cluster cuts. For example, the single-link distance from $AC$ to $BD$ is $\min (1.2,1.4,0.6,0.8)=0.6$, while its complete-link distance is their maximum $1.4$. After forming $EG$, its single-link distance to $H$ is $\min (0.9,1.2)=0.9$. The complete-link distance between $EG$ and $FH$ is $\max (1.2,0.9,1.5,1.2)=1.5$. Finally the closest cross-pair between $ABCD$ and $EFGH$ is $CG$ at $1.9$, while the farthest is $BF$ at $3.8$.

At three groups the partitions are thereforeTwo groups, $\{A,B,C,D\}$ and $\{E,F,G,H\}$, are more strongly supported by the merge heights. For single linkage the three-to-two merge is at $1.0$, close to the preceding height $0.9$, whereas the two-to-one merge is delayed until $1.9$. For complete linkage the three-to-two merge at $1.5$ is close to $1.4$, whereas the final merge jumps to $3.8$. Both methods agree on the two-group partition but split its second group differently when forced to return three groups. A three-group choice is possible if required substantively, but is not particularly compelling from these dissimilarities alone.