Codex @codex  0

Principal component analysis replaces correlated measurements by orthogonal linear coordinates ordered by their variance. For centered observations collected in an $n\times p$ matrix $Z$, write the sample covariance matrix as $S=Z^{T}Z/(n-1)$. A unit loading vector $a$ gives score vector $Za$ with sample variance $a^{T}Sa$. The Rayleigh quotient is maximized by a unit eigenvector of $S$ for its largest eigenvalue.

The subsequent principal components maximize the same quadratic form subject to orthogonality to the earlier loading vectors. By the spectral theorem, if $S=Q\Lambda Q^T$ with descending eigenvalues, the scores $ZQ$ have diagonal covariance $\Lambda$. Their fractions of explained variance are ${\lambda }_j/{\sum }_i{\lambda }_i$. Repeated eigenvalues identify an eigenspace rather than a unique axis.

Keeping the first $q$ principal components gives the rank-$q$ reconstruction $\hat{Z}=Z{Q}_q{Q}_q^T$. Its squared reconstruction error isFor any rank-$q$ orthogonal projection $P$, retained variance is $\mathrm{tr}(SP)={\sum }_j{\lambda }_j{q}_j^{T}P{q}_j$. The weights $q_j^{T}P{q}_j$ lie in $[0,1]$ and sum to $q$, so this is at most the sum of the largest $q$ eigenvalues. Projecting onto any candidate reconstruction subspace is its best least-squares reconstruction; this proves the minimum-error property and agrees with the singular value decomposition. Thus PCA finds the linear subspace preserving the greatest variance, equivalently minimizing squared reconstruction error. The upper-left sketch shows the first axis aligned with the elongated cloud; the second is perpendicular. A scree plot and substantive interpretability help choose $q$.

These methods depend on measurement scale. A variable measured in large numerical units can dominate covariance-based principal component analysis; standardizing variables gives principal component analysis on a correlation matrix. Scores summarize patterns, but the method neither establishes causation nor guarantees that the largest-variance directions best predict a separate response.

The calculation is principal component analysis on a correlation matrix, rather than on an unscaled covariance matrix. For each of the eleven rating variables, subtract its column mean and divide by its column standard deviation. If $Z$ is the resulting $43\times 11$ matrix, use the divisor convention of the fitted object to writeHere the scaling standard deviations use divisor $43$, consistently with this convention. Using sample standard deviations with divisor $42$ gives the same correlation matrix and directions, with the corresponding score scale. Every diagonal entry of $R$ is one, so ${\sum }_k{\lambda }_k=\mathrm{tr}R=11$.

A principal component is the linear combination of the standardized ratings along $a_k$. Its principal component score for judge $i$ is $t_{ik}=z_i^T{a}_k$, and its variance is ${\lambda }_k$. The displayed component standard deviations are $\sqrt{{\lambda }_k}$; the explained variance of a principal component is ${\lambda }_k/11$. In particular,The first component explains about $92.122\%$ of the standardized variation, and the first two jointly explain $95.982\%$. Components three, four and five add about $2.321\%$, $0.830\%$ and $0.339\%$. The cumulative line adds these proportions successively; it is not the proportion explained by that component alone. Retaining two components discards about $4.018\%$ of the total standardized squared variation.

The displayed principal component loadings are the unit eigenvector coefficients $a_{jk}$. The first direction has positive, broadly similar weights for all ratings. It measures an overall favorable-rating tendency: a judge high on most ratings has a high first score. The second direction compares integrity and demeanour, with large positive coefficients, against such attributes as decision promptness, case-flow management and physical ability, with negative coefficients. Thus among similarly rated judges, a positive second score indicates relatively stronger integrity/demeanour compared with those operational ratings. This is a description of a data direction, not a causal interpretation. Multiplying a whole eigenvector and its scores by minus one reflects the display but changes nothing substantive.

These principal component loadings are not themselves variable-component correlations. With standardized variables,For example, the integrity correlation with component one is approximately $3.1833\times 0.289\simeq 0.920$. Blank loadings in the printed second column indicate suppressed small entries, not exact zeros. The complete unrounded loading vectors in the fitted object must be used to calculate scores.

A two-dimensional representation plots the 43 pairswith each point labeled by its judge. In the fitted software object these are the first two columns of the score matrix. Nearby points have similar projections of their standardized ratings. The rank-two reconstruction ${\hat{z}}_i=t_{i1}{a}_1+t_{i2}{a}_2$ is the orthogonal projection onto the leading principal plane and minimizes the total squared reconstruction error over all two-dimensional linear subspaces. To see this, for any orthonormal pair of directions the captured variance is the sum of its two Rayleigh quotients, at most ${\lambda }_1+{\lambda }_2$ by diagonalizing $R$; the remaining variance is the total minus that captured sum. Distances in the plot omit the smaller components, so a close projected pair need not be equally close in every original rating. The actual 43 coordinates cannot be recovered from the summary alone; the supplied data matrix or fitted score matrix is needed.

The printed descriptive list omits INTG, although the loading table includes it. INTG denotes integrity and supplies the eleventh variable. The standard data set also contains a contacts variable; the stated eleven-rating analysis uses the ratings, not contacts.

The command centres the log measurements and performs principal component analysis of their covariance matrix. With cor=F, it does not first standardize the variables to variance one. For a unit principal component loading vector $q$, the score $q^T(z_i-\bar{z})$ has sample variance $q^T\hat{\Sigma }q$. By the spectral theorem for real symmetric matrices, expand $q$ in an orthonormal eigenbasis; that variance is a weighted average of the eigenvalues. Its maximum is the largest eigenvalue, reached at the corresponding eigenvector. Repeating subject to orthogonality to previous directions gives the other principal components.

The software uses the divisor-$31$ sample covariance matrix for its component standard deviations, whereas var used divisor $30$. Diagonalizing the displayed matrix gives approximatelyTaking square roots of the latter gives $(0.5671603,0.07312405,0.03064834)$, agreeing with the output to the rounding of the displayed sample covariance matrix. The normalization changes these numerical standard deviations but leaves the eigenvectors and proportions of explained variance of a principal component unchanged.

The first principal component explains about $98.08\%$ of the total log-variable variance, the second another $1.63\%$, and the last about $0.286\%$. Keeping two preserves about $99.714\%$. An original scree plot displays the sharp drop after the first component:

Consequently the centered cloud is very elongated along one direction: a one-dimensional approximation represents almost all of its squared variation. From the displayed covariance matrix, a unit first eigenvector, with its arbitrary sign chosen positive, is approximatelyIts principal component score is therefore $0.3980(\log G_i-\overline{\log G})+0.09514(\log H_i-\overline{\log H})+0.9124(\log V_i-\overline{\log V})$. All coefficients have the same sign, so this direction describes overall tree size, with log volume dominating the unstandardized variation. The summary alone supplies proportions, not the loading directions; the displayed sample covariance matrix supplies the latter. This is a geometric approximation, not evidence of an exact deterministic relation or independent original variables.

One further use is principal component analysis on a correlation matrix, obtained with cor=T, which equalizes marginal variances and answers a different scaling-sensitive question. Another is principal component regression: fit a response to selected principal component scores rather than highly correlated original predictors, choosing the number using prediction assessment. The fitted transformation also projects new measurements into the same low-dimensional coordinates, provided the training centering and scaling are retained.

princomp(): principal component analysis. Supply observations in rows and numeric variables in columns. After centering, let $Z$ denote the resulting matrix and $C=Z^{T}Z/n$ its sample covariance matrix, with an explicitly stated normalization. WriteThe first direction maximizes the Rayleigh quotient $q^{T}Cq$ over unit vectors; successive directions maximize it subject to orthogonality to the earlier directions. The principal component scores are $z_{ij}=q_j^T(x_i-\bar{x})$, with variances ${\lambda }_j$ and zero pairwise sample covariance. Zero covariance alone does not imply independence. Each principal component loading describes a direction in variable space; its sign is arbitrary, and repeated eigenvalues leave the basis inside the corresponding eigenspace arbitrary.

The function returns principal component loadings, component standard deviations sdev and, when requested, principal component scores. By default it uses a covariance matrix; cor=T instead performs principal component analysis on a correlation matrix, standardizing nonconstant variables to variance one. This choice matters when variables have different measurement units. These interface details are documented in the S-Plus help for princomp.

A scree plot shows the ordered eigenvalues; a plot of the first two principal component scores can reveal groups, unusual observations or a dominant gradient. The first $k$ components explain ${\sum }_{j\le k}{\lambda }_j/{\sum }_j{\lambda }_j$ of the total variance. Reconstructing each centered observation by its orthogonal projection onto these directions minimizes average squared reconstruction error among $k$-dimensional linear subspaces. Thus PCA gives an optimal linear compression for squared reconstruction error, with scaling and the retained dimension chosen explicitly. It is an unsupervised learning method, rather than a classification rule fitted to supplied labels.

tree(): classification trees and regression trees. A call such as tree(response ~ predictors, data=dat) fits a recursive partitioning model. At a current node, test numerical predictors against candidate thresholds, or divide the levels of a categorical predictor into two sets. Select a split that most improves the fitting criterion, then repeat within its children. Numerical splits give axis-aligned regions. A categorical response produces a classification tree; a numeric response produces a regression tree, as specified in the S-Plus tree documentation.

For a regression tree, the constant prediction in a terminal region is its sample mean, minimizing the sum of squared residuals there. For a classification tree, estimate class probabilities by the observed proportions in a leaf and predict a most probable class under equal misclassification costs. The classification-tree deviance of a node iswith $0\log 0=0$; splitting maximizes the decrease from parent to children. This fitting criterion differs from the number of wrong classifications. The middle sketches show the same three-region partition as a binary tree: first split on variable one, then split the right-hand branch on variable two.

Growing to small leaves can overfit. Use cost-complexity tree pruning, balancing fitted loss $R(T)$ against leaf count,and select complexity by cross-validation. Plotting the fitted tree and annotating nodes displays the rules; prediction routes each new observation through the fitted splits. A tree gives an interpretable, piecewise-constant prediction rule; training fit alone does not measure prediction accuracy.

The rpart() alternative. This also fits classification and regression trees by recursive partitioning, with squared-error splitting for ordinary numeric responses and, by default for classification, Gini impurity $1-{\sum }_r{p}_r^2$. Classification settings can instead use information-based splitting and specify prior probabilities and a loss matrix. It supports surrogate splits to approximate a node's primary left/right assignment when the primary predictor is missing. Its complexity-parameter path and cross-validated errors support cost-complexity tree pruning; for example, select a complexity parameter and apply prune(fit, cp=...). The maintainer's rpart documentation describes the criteria and the distinction in handling surrogate predictors. The partition and tree sketches apply to either implementation; the fitted tree need not be the same because splitting criteria and controls can differ.

cmdscale(): classical multidimensional scaling. Its input is a dissimilarity matrix or stored distance object. It outputs coordinates for the observations in a chosen dimension, usually two for visualization. Let $D^{(2)}$ contain the squared entries of $D$, and putFor centered Euclidean coordinates $X$, the distance identity $d_{ij}^2=\Vert x_i{\Vert }^2+\Vert x_j{\Vert }^2-2x_i^T{x}_j$ gives $G=X{X}^T$: multiplying by $H$ removes the row and column norm terms. Diagonalize this Gram matrix and retain its $k$ largest positive eigenvalues. The coordinates areThus classical scaling exactly recovers Euclidean distances when all positive eigenvalues are retained. Reducing dimension approximates the centered Gram matrix; it does not in general minimize squared discrepancies in the original distances. Negative eigenvalues identify a failure of the Euclidean distance matrix criterion, so their size matters when interpreting a display. The historical S-Plus cmdscale help describes the dimension argument, eigenvalue output and optional additive adjustment to dissimilarities.

Plot the two coordinate columns on equally scaled axes and label observations. Nearby plotted points suggest small dissimilarities only insofar as the dimension reduction fits the input well. The bottom sketches give an exact Euclidean example: a configuration and its independently recovered two-dimensional coordinates have identical pairwise distances, despite different coordinate axes. Classical scaling converts distances into geometry; translations, rotations and reflections do not change the represented distances. For Euclidean distances between observations, its coordinates agree with the corresponding principal component scores up to these harmless choices of axes.

For the sample, put $\bar{x}=n^{-1}{\sum }_i{x}_i$ and use the sample covariance matrixOrder its eigenvalues ${\hat{\lambda }}_j$ decreasingly and choose unit orthogonal eigenvectors ${\hat{\ell }}_j$. The sample principal components are the score columnsTheir sample means are zero and their sample covariance is diagonal, since ${\hat{\ell }}_j^{T}S{\hat{\ell }}_k={\hat{\lambda }}_j{1}_{\{j=k\}}$. The $j$th sample variance is ${\hat{\lambda }}_j$, and its explained variance of a principal component is ${\hat{\lambda }}_j/\mathrm{tr}(S)$. Using divisor $n$ instead changes the numerical variances by a common factor but leaves the component directions unchanged.

Standardizing means replacing a variable with positive sample variance by $(x_{ij}-{\bar{x}}_j)/s_j$. This gives principal component analysis on a correlation matrix, using $C=D^{-1/2}S{D}^{-1/2}$ with $D=\mathrm{diag}(S)$. It removes arbitrary differences of measurement units and prevents a large-variance coordinate from dominating solely through its scale. It also changes the question being optimized: small-variance and possibly noisy coordinates receive equal marginal weight. Principal component analysis based on the covariance matrix can be more appropriate when the variables have common meaningful units and absolute variability matters. Standardization is therefore a substantive choice, not an automatic improvement. A zero-variance variable must first be removed. In the word-rating example the variables share the same numerical range, but their across-word dispersions can differ; standardization gives each attribute equal initial variance.

For the standardized eight-variable analysis, total variance is $\mathrm{tr}(C)=8$. The first three fractions of explained variance of a principal component are approximately 59.6%, 19.1% and 10.1%, respectively. The first two account for about 78.8%, and the first three for 88.9%. The first direction is dominant but one direction alone loses substantial variation. Retaining two principal components is a plausible descriptive summary: their eigenvalues exceed $1$, whereas the third is below $1$. That cutoff is a heuristic, and the third principal component may still have interpretable content.

The reported coefficient rows are not unit-length vectors. The printed rounded rows have squared lengths $4.7514$, $1.5222$ and $0.8063$, consistent with their eigenvalues to the precision of the coefficient table. They are consistent with the scaled principal component loading convention $r_j=\sqrt{{\lambda }_j}{\ell }_j$. Such rows remain eigenvectors, but the score direction with unit length is ${\ell }_j=r_j/\sqrt{{\lambda }_j}$, or more accurately the row divided by its actual Euclidean norm when using rounded entries. Under this convention $r_{ij}$ is the correlation of standardized attribute $i$ with the variance-one score for component $j$:This explains both their sizes and why they must not be treated as the unit vectors from parts (i) and (ii).

The first principal component has positive coefficients on every attribute, strongest on friendliness, goodness, niceness, bravery and strength. With the favorable endpoints coded in the positive direction, it describes a broad favorable or impressive evaluation, with a potency contribution. The second contrasts movement and speed, with some size and strength, against the first three evaluative attributes; it is chiefly an activity dimension. The third is dominated by size, with smaller positive strength and negative movement/speed coefficients, distinguishing size or potency from activity. These are interpretations of directions of variation, not evidence of causal latent traits. Reversing a principal component's overall sign changes neither its meaning as an axis nor its explained variance; endpoint coding fixes only how its positive side is described.

If the change is purely a rescaling of the physics mark, write $X^{\prime }=DX$, where $D=\mathrm{diag}(10/3,1,1,1)$. The sample covariance matrix transforms asThe physics variance is multiplied by $100/9$, becoming approximately $10044.44$. Each covariance between physics and another subject is multiplied by $10/3$, and all other entries stay unchanged. Principal component analysis on this new sample covariance matrix maximizes variance in the new numerical units, so its eigenvectors and explained variance of a principal component generally change. Physics receives disproportionately greater influence simply because its unit has changed.

One solution is to convert the physics marks back to the original scale before doing principal component analysis. Alternatively, use principal component analysis on a correlation matrix: standardize every subject byA positive change of units multiplies both a centered variable and its standard deviation by the same factor. Thus it leaves $Z_j$ and the sample correlation matrix $R$ unchanged.

Correlation-based principal component analysis is invariant under positive changes of units; covariance-based principal component analysis is not. Standardizing gives each subject unit variance, which is a different weighting choice and can emphasize relatively low-variance subjects. If the revised assessment changes the scores themselves rather than just their units, its sample correlation matrix may also change, and mere standardization cannot recover the original analysis.

Covariance-based PCA uses centered protein measurements in their original gram units. Its total variance is $218.4$. Correlation-based PCA first divides each centered food measurement by its sample standard deviation, giving nine unit-variance variables and total variance nine. Thus it is principal component analysis on a correlation matrix, not an equivalent rotation of the original unscaled measurements.

The variances of cereals and milk together account for $170.9/218.4\approx 78.3\%$ of the original total. Consequently those variables have considerable influence on covariance-based PCA. Its first component is dominated by a cereal coefficient $-0.86$ opposed to milk $0.42$, broadly a cereal-versus-milk contrast. Its second is dominated by milk $0.83$, with cereals $0.40$ and fish $-0.29$. Its third is chiefly white meat $0.80$ versus fish $-0.52$, with a smaller negative milk term. Component signs may all be reversed without changing the analysis; the important features are relative signs and magnitudes.

After standardization, eggs and starch can contribute as strongly as large-variance foods. The first correlation-based component gives positive weight to meat, eggs, milk and starch, opposed principally to cereals and pulses/nuts; eggs now has coefficient $0.43$ despite its small raw variance. The second emphasizes fish and fruit/vegetables negatively, opposed especially to starch and white meat. The third contrasts white meat and fruit/vegetables with milk, fish and red meat. Thus standardization changes both the emphasis and the directions, rather than just the displayed units of the same components.

The explained variance of a principal component must use the appropriate total:Raw and standardized percentages describe different variance objectives and must not be compared as if one method were always superior.

Standardize when relative patterns across variables, rather than their absolute gram-scale variation, should have equal marginal weight. Correlation-based PCA is also insensitive to changing a variable's measurement unit, which covariance-based PCA is not. Here every food group already has the same units, so differing units do not force standardization. If large fluctuations in actual protein amounts are scientifically important, covariance-based PCA is appropriate. If dietary composition across all food groups is the focus, correlation-based PCA can be more informative. Standardization can nevertheless amplify noise in nearly constant variables; the decision should follow the scientific purpose and measurement reliability.

Two conventions occur for principal-component loadings: the unit eigenvector ${\ell }_j$ defining a principal component, and the scaled vector $r_j=\sqrt{{\lambda }_j}{\ell }_j$. For principal component analysis on a correlation matrix, the entries of the latter are the correlations between each standardized variable and the component scaled to variance one. Indeed, $\mathrm{Cov}(Z,{\ell }_j^{T}Z)=C{\ell }_j={\lambda }_j{\ell }_j$ and $\mathrm{Var}({\ell }_j^{T}Z)={\lambda }_j$. Thus $\Vert r_j{\Vert }^2={\lambda }_j$, whereas $\Vert {\ell }_j\Vert =1$; reporting the convention avoids confusing component directions with correlations.

A score is an observation's coordinate along a principal component direction. Center an observation by the fitted sample mean, apply the fitted variable scaling when using principal component analysis on a correlation matrix, then take its inner product with the unit principal component loading vector. New observations must use the training center and scaling. The score variance equals the corresponding eigenvalue when the covariance normalization is consistent.