Codex @codex  0

Principal component analysis replaces the original coordinates by orthogonal linear combinations, ordered so that the first few retain as much variance as possible. It gives a lower-dimensional summary for visualization or compression and reveals the main directions of variation. A direction of large variance need not be the direction most useful for classification.

Write $\mu =\mathbb{E}X$. For a unit vector $u$, the centered score $Y=u^T(X-\mu )$ has varianceMaximizing this Rayleigh quotient with a Lagrange multiplier givesThus a maximizing direction must be an eigenvector of the covariance matrix. To identify the maximum, use the spectral theorem for real symmetric matrices to choose an orthonormal basis of eigenvectors $v_1,\dots ,v_p$, with eigenvalues ${\lambda }_1\ge \cdots \ge {\lambda }_p\ge 0$. Writing $u={\sum }_j{c}_j{v}_j$ givesEquality is attained at $u=v_1$. Consequently the first population principal component isFor each subsequent principal component, maximize the same variance over unit coefficient vectors orthogonal to all the previously chosen ones. Restricting the expansion above to the remaining eigenvectors gives $v_2$, then $v_3$, and so on. ThusThe principal components are uncorrelated; they need not be independent unless additional distributional assumptions, such as joint Gaussianity, hold. Their total variance is $\mathrm{tr}\Sigma ={\sum }_j{\lambda }_j$. Signs of eigenvectors are arbitrary, and a repeated eigenvalue permits any orthonormal basis in its eigenspace. In sample principal component calculations, replace $\mu$ by the sample mean and $\Sigma$ by the sample covariance matrix.

Let $v_2,v_3,v_4$ be the three supplied coefficient vectors, in their displayed order. Multiplication by the sample covariance matrix givesThe four vectors are pairwise orthogonal, with norms $\sqrt{3}$, $\sqrt{102}$, $\sqrt{6}$ and $\sqrt{204}$. Hence $u_j=v_j/\Vert v_j\Vert$ form an orthonormal basis of eigenvectors. The variance of the principal component with unit coefficient vector $u_j$ is its eigenvalue ${\lambda }_j$. Using unnormalized coefficient vectors would instead give variances ${\lambda }_j\Vert v_j{\Vert }^2$ and would not produce the desired proportions.

The total variance isTherefore the explained variance of a principal component isThe first two principal components together account for $2658/3162\simeq 84.06\%$, and the first three account for $2958/3162\simeq 93.55\%$ of the total variance.

For $v_1=(1,1,1,0{)}^T$, direct multiplication by the sample covariance matrix givesIt is therefore an eigenvector, with eigenvalue $2250$. Normalization gives $u_1=v_1/\sqrt{3}$, so the corresponding sample principal component isIts coefficients are proportional to the required vector. The remaining eigenvalues calculated in (ii) are smaller, confirming that this is the first principal component.

If the change is purely a rescaling of the physics mark, write $X^{\prime }=DX$, where $D=\mathrm{diag}(10/3,1,1,1)$. The sample covariance matrix transforms asThe physics variance is multiplied by $100/9$, becoming approximately $10044.44$. Each covariance between physics and another subject is multiplied by $10/3$, and all other entries stay unchanged. Principal component analysis on this new sample covariance matrix maximizes variance in the new numerical units, so its eigenvectors and explained variance of a principal component generally change. Physics receives disproportionately greater influence simply because its unit has changed.

One solution is to convert the physics marks back to the original scale before doing principal component analysis. Alternatively, use principal component analysis on a correlation matrix: standardize every subject byA positive change of units multiplies both a centered variable and its standard deviation by the same factor. Thus it leaves $Z_j$ and the sample correlation matrix $R$ unchanged.

Correlation-based principal component analysis is invariant under positive changes of units; covariance-based principal component analysis is not. Standardizing gives each subject unit variance, which is a different weighting choice and can emphasize relatively low-variance subjects. If the revised assessment changes the scores themselves rather than just their units, its sample correlation matrix may also change, and mere standardization cannot recover the original analysis.

Write $x_r,x_t$ for two rows of the data matrix and $\delta =x_r-x_t$. Three possible dissimilarities for continuous variables are the following.

The Euclidean distance isIt is simple and has the usual geometric interpretation when coordinates have comparable units. Its disadvantage is sensitivity to the numerical scales: a change of units or a high-variance variable can dominate it. Strongly correlated coordinates can also give repeated weight to essentially the same information. Squared Euclidean distance is useful in some methods, but is not itself a metric because it need not satisfy the triangle inequality.

The standardized Euclidean distance iswhere $s_j^2$ is the sample variance of coordinate $j$ across individuals. It measures differences in standard-deviation units and is invariant under positive coordinate rescalings when the scales are estimated consistently. It is useful for differently measured variables, but requires nonzero sample variances, ignores correlations between coordinates and can give excessive influence to a low-variance noisy variable. With fixed positive scales it is a Euclidean distance after a diagonal change of coordinates, so it is a metric.

The Mahalanobis distance iswhere $S$ is the pooled sample covariance matrix of all individuals. If $S=C{C}^T$ is its Cholesky decomposition, then $d_M=\Vert C^{-1}\delta {\Vert }_2$: it is Euclidean distance in whitened coordinates. It adjusts for both scales and correlations, avoiding duplicate weight for highly correlated variables, and is invariant under nonsingular affine transformations when the sample covariance matrix is transformed accordingly. No multivariate normal distribution assumption is required merely to define the distance. However, $S$ must be a positive-definite matrix; when there are too few observations or dependent coordinates, its inverse is unavailable. Estimation of $S^{-1}$ can also be unstable and sensitive to outliers. These trade-offs determine whether covariance adjustment is preferable to the simpler distances.

Use the unbiased sample covariance matrixand assume $\Sigma$ is positive-definite and $n>p$, so $S$ is invertible almost surely. For a nonzero fixed direction $a$, the observations $Y_j=a^T{X}_j$ have mean $a^T\mu$ and variance $a^T\Sigma a$. Their squared Student t-test statistic for the projected null mean isPut $b=\bar{X}-{\mu }_0$. The Cauchy-Schwarz inequality giveswith equality when $a$ is proportional to $S^{-1}b$, if $b\ne 0$. For $b=0$ every projected statistic is zero. Maximizing the squared discrepancy over directions therefore gives Hotelling's T-squared statistic:This uses the squared generalized Rayleigh quotient. The preliminary formula on the printed cover has an unsquared numerator and is not valid as written: setting $a=t{C}^{-1}b$ makes that ratio $1/t$ for $b\ne 0$, so it is unbounded. The displayed Cauchy-Schwarz argument supplies the needed corrected identity independently.

Under the null, let $Z=\sqrt{n}(\bar{X}-{\mu }_0)$ and $C=(n-1)S$. Then $Z\sim N_p(0,\Sigma )$, $C\sim W_p(n-1,\Sigma )$, and $Z,C$ are independent. To see these sample properties, apply an orthogonal transformation to the $n$ observation indices with first row $n^{-1/2}(1,\dots ,1)$. Its first centered row is $Z$; its other $n-1$ rows are independent $N_p(0,\Sigma )$ residual contrasts, independent of the first. Their outer products sum to $C$, establishing the Wishart distribution and independence.

Since $T^2=(n-1)Z^T{C}^{-1}Z$, use the supplied Gaussian-Wishart quadratic-form law with $k=n-1$ to obtainThus a level-$\eta$ test rejects when this scaled statistic exceeds the $(1-\eta )$ quantile of the F-distribution. The Gaussian-Wishart law requires independence; it holds here because of the orthogonal sample decomposition. Its printed use of $X$ instead of the introduced $Z$ is a notational slip. A fixed-direction Student law alone would not calibrate the maximized statistic, because its maximizing direction depends on the sample.

Transform each observation to $Y_j=A^T{X}_j$. If $A^T$ has full row rank $m$, the transformed sample is multivariate normal with mean $A^T\mu$ and positive-definite covariance matrix $A^T\Sigma A$. Its sample mean is $A^T\bar{X}$ and its unbiased sample covariance matrix is $A^{T}SA$. The Hotelling test of linear hypotheses therefore usesHere $n>m$ is required. Reject for a large scaled statistic using the relevant upper F-distribution quantile. If the rows of $A^T$ are dependent, choose a basis for their row space, use that basis as the contrast matrix, and replace $m$ by its rank $r$. The null hypothesis is unchanged, but redundant contrasts must not be treated as independent dimensions. Rank zero gives no restrictions and hence no nontrivial test.

Write $\Omega ={\Sigma }^{-1}$ for a positive-definite matrix. Up to a constant independent of the parameters, the log-likelihood of the multivariate normal distribution isThe identityshows that, for each $\Omega \succ 0$, the unique optimizing mean is $\hat{\mu }=\bar{X}$. Using the matrix trace to rewrite the first quadratic sum, the profile likelihood isTherefore maximum likelihood estimation of the precision matrix, whenever the maximum exists, is exactlyThis is the profile likelihood for a Gaussian precision matrix. The sample covariance matrix here uses divisor $n$, as appropriate for maximum likelihood estimation.

There is an important rank qualification in the printed assumptions. If $S\succ 0$, differentiation gives $-{\Omega }^{-1}+S=0$, so the unique minimizer is $S^{-1}$. Uniqueness follows because the negative log-determinant is a strictly convex function on the positive-definite matrices. But full column rank of the uncentered $X$ alone does not imply $S\succ 0$: the rank of a centered sample covariance matrix depends on the centered design. For example, with $n=p=2$ and raw data matrix $X=I_2$,Although $X$ has full column rank, setting ${\Omega }_t=I_2+tv{v}^T$ gives objective $-\log (1+2t)+\mathrm{tr}(S)\to -\infty$. Thus an unpenalized maximizer need not exist under the literal raw-rank assumption. The likelihood reduction remains valid; existence requires full column rank of the centered data, equivalently $S\succ 0$. For the nonsingular Gaussian model this holds almost surely when $n>p$.

Use the entrywise-penalty convention for the Graphical Lasso:The sum is the entrywise matrix L1 norm. Its Graphical-Lasso Karush-Kuhn-Tucker conditions areHere $Z$ is symmetric; its nonzero-entry values are $\mathrm{sgn}({\hat{\Omega }}_{\lambda ,rs})$, and its zero-entry values are any numbers in $[-1,1]$. These are the subgradient optimality condition for this convex optimization problem. Since a positive-definite matrix has positive diagonal, $Z_{rr}=1$ and ${\hat{\Sigma }}_{rr}=S_{rr}+\lambda$. Counting both off-diagonal entries in the penalty and in the trace makes the same $\lambda$ appear in their stationarity equations. The common variant penalizing only off-diagonal entries instead sets $Z_{rr}=0$; the off-diagonal equations and the column argument below are unchanged.

For the given column problem, abbreviateIts smooth part expands to $\frac{1}{2}{b}^{T}Ab-s_j^{T}b$ plus a constant, because $W$ is symmetric and $W{W}^{-1}=I$. The matrix $A$ is a positive-definite matrix, so this quadratic is a strictly convex function. Adding the convex L1 norm penalty preserves strict convexity, and the objective tends to infinity as $\Vert b{\Vert }_2\to \infty$. The minimizer $b^{\ast }$ exists and is unique. Its Karush-Kuhn-Tucker conditions are

We identify that minimizer directly from the precision matrix. Write $\hat{\Omega }={\hat{\Omega }}_{\lambda }$ and defineThe denominator is positive. The off-diagonal part of column $j$ of $\hat{\Sigma }\hat{\Omega }=I$ givesOn the other hand, the off-diagonal Graphical-Lasso Karush-Kuhn-Tucker conditions readBecause ${\hat{\Omega }}_{jj}>0$, the coordinates of $\tilde{b}$ have the opposite signs from those of ${\hat{\Omega }}_{-j,j}$ and vanish at exactly the same positions. The subgradient of the absolute value therefore satisfiesIt follows thatThese are exactly the Lasso column problem's Karush-Kuhn-Tucker conditions. Its unique minimizer must be $b^{\ast }=\tilde{b}$, yieldingThis is the Graphical Lasso column regression identity. The minus sign is essential: it converts the precision-entry subgradient into the regression-coefficient subgradient. The argument uses the inverse identity directly, so it does not require proving the supplied block inverse formula.

Use the uncentered sample covariance matrixThen $v_{\ell }=a_{\ell }^T\Sigma a_{\ell }$ and, simultaneously for every unit vector $a_{\ell }$,Here $\Vert \Sigma {\Vert }_{\mathrm{op}}=1$ and the effective rank of a covariance matrix satisfiesThe Gaussian sample-covariance operator-norm bound therefore gives, with probability at least $1-e^{-\delta }$,Under the assumed upper bound on $\log d+1+\delta$, the second term is at most the first. Thus the stronger simultaneous estimateholds for every $\ell$. Squaring and using that the displayed ratio is at most one gives the inequality requested in the question after enlarging the universal constant $C$.

The matrix $S$ is the sample covariance matrixChanging $n$ to $n-1$ is another convention, but the objective must use the same convention throughout.

For independent multivariate normal observations with unknown mean and nonsingular covariance, profiling the mean at its sample value gives negative log-likelihood proportional to $-\log \det \Omega +\mathrm{tr}(S\Omega )$ over $\Omega \succ 0$. The sample covariance matrix uses divisor $n$. Its minimizer exists exactly when $S\succ 0$, and is then $S^{-1}$.

If $H=I_n-11^T/n$ and $S=X^{T}HX/n$, then $\mathrm{rank}S=\mathrm{rank}(HX)\le \min (p,n-1)$. In particular full column rank of raw $X$ does not ensure a nonsingular centered sample covariance matrix. When $S$ is singular, its Gaussian precision profile likelihood has no finite maximizer: increase precision along a nonzero null vector.

A sample correlation matrix rescales a sample covariance matrix $S$ to unit diagonal, for variables with positive sample variance. Its off-diagonal entries are sample correlation coefficients. Strong correlations between predictors can indicate multicollinearity, but pairwise correlations alone do not exclude higher-order linear dependence.

For paired scalar observations $(x_i,y_i)$, their sample covariance is $n^{-1}{\sum }_i(x_i-\bar{x})(y_i-\bar{y})$, or the same sum divided by $n-1$ under the unbiased convention. It is an off-diagonal entry of the sample covariance matrix of the paired vectors.

The uncentered empirical second-moment matrix averages the outer products of the observations. Its expected value is $\mathrm{Cov}(X)+\mathbb{E}[X]\mathbb{E}[X{]}^{\top }$. It is an unbiased estimator of the covariance matrix when the population expected value is known to be zero. It differs from a sample covariance matrix centered at the sample mean.