= Solution
\b[`princomp()`: <principal component analysis>.] Supply observations in rows and numeric variables in columns. After centering, let $Z$ denote the resulting <matrix> and $C=Z^TZ/n$ its <sample covariance matrix>, with an explicitly stated normalization. Write
$$
Cq_j=\lambda_jq_j,\qquad q_i^Tq_j=\mathbf1_{\{i=j\}},\qquad \lambda_1\geq\cdots\geq\lambda_p\geq0.
$$
The first direction maximizes the <Rayleigh quotient> $q^TCq$ over unit <vectors>; successive directions maximize it subject to orthogonality to the earlier directions. The <principal component scores> are $z_{ij}=q_j^T(x_i-\bar x)$, with <variances> $\lambda_j$ and zero pairwise <sample covariance>. Zero <covariance> alone does not imply <independence>. Each <principal component loading> describes a direction in variable space; its sign is arbitrary, and repeated <eigenvalues> leave the basis inside the corresponding <eigenspace> arbitrary.
The function returns <principal component loadings>, component <standard deviations> `sdev` and, when requested, <principal component scores>. By default it uses a <covariance matrix>; `cor=T` instead performs <principal component analysis on a correlation matrix>, standardizing nonconstant variables to <variance> one. This choice matters when variables have different measurement units. These interface details are documented in the https://sites.oxy.edu/lengyel/M150/Sueselbeck/helpfiles/princomp.html[S-Plus help for princomp].
A <scree plot> shows the ordered <eigenvalues>; a plot of the first two <principal component scores> can reveal groups, unusual observations or a dominant gradient. The first $k$ components explain $\sum_{j\leq k}\lambda_j/\sum_j\lambda_j$ of the total <variance>. Reconstructing each centered observation by its <orthogonal projection> onto these directions minimizes average squared reconstruction error among $k$-dimensional linear subspaces. Thus \b[PCA gives an optimal linear compression for squared reconstruction error, with scaling and the retained dimension chosen explicitly.] It is an <unsupervised learning> method, rather than a classification rule fitted to supplied labels.
\b[`tree()`: <classification trees> and <regression trees>.] A call such as `tree(response ~ predictors, data=dat)` fits a <recursive partitioning> model. At a current node, test numerical predictors against candidate thresholds, or divide the levels of a categorical predictor into two sets. Select a split that most improves the fitting criterion, then repeat within its children. Numerical splits give axis-aligned regions. A categorical response produces a <classification tree>; a numeric response produces a <regression tree>, as specified in the https://sites.oxy.edu/lengyel/m150/Sueselbeck/helpfiles/tree.html[S-Plus tree documentation].
For a <regression tree>, the constant prediction in a terminal region is its <sample mean>, minimizing the sum of squared residuals there. For a <classification tree>, estimate class probabilities by the observed proportions in a leaf and predict a most probable class under equal misclassification costs. The <classification-tree deviance> of a node is
$$
D(t)=-2\sum_r n_{tr}\log(n_{tr}/n_t),
$$
with $0\log0=0$; splitting maximizes the decrease from parent to children. This fitting criterion differs from the number of wrong classifications. The middle sketches show the same three-region partition as a binary tree: first split on variable one, then split the right-hand branch on variable two.
Growing to small leaves can overfit. Use <cost-complexity tree pruning>, balancing fitted loss $R(T)$ against leaf count,
$$
R_\alpha(T)=R(T)+\alpha|\mathcal L(T)|,
$$
and select complexity by <cross-validation>. Plotting the fitted tree and annotating nodes displays the rules; prediction routes each new observation through the fitted splits. \b[A tree gives an interpretable, piecewise-constant prediction rule; training fit alone does not measure prediction accuracy.]
\b[The `rpart()` alternative.] This also fits <classification and regression trees> by <recursive partitioning>, with squared-error splitting for ordinary numeric responses and, by default for classification, <Gini impurity> $1-\sum_r p_r^2$. Classification settings can instead use information-based splitting and specify prior probabilities and a loss <matrix>. It supports <surrogate splits> to approximate a node's primary left/right assignment when the primary predictor is missing. Its complexity-parameter path and cross-validated errors support <cost-complexity tree pruning>; for example, select a complexity parameter and apply `prune(fit, cp=...)`. The https://stat.ethz.ch/R-manual/R-devel/library/rpart/html/rpart.html[maintainer's rpart documentation] describes the criteria and the distinction in handling surrogate predictors. The partition and tree sketches apply to either implementation; the fitted tree need not be the same because splitting criteria and controls can differ.
\b[`cmdscale()`: <classical multidimensional scaling>.] Its input is a <dissimilarity matrix> or stored distance object. It outputs coordinates for the observations in a chosen dimension, usually two for visualization. Let $D^{(2)}$ contain the squared entries of $D$, and put
$$
H=I-\frac1n\mathbf1\mathbf1^T,\qquad G=-\frac12HD^{(2)}H.
$$
For centered Euclidean coordinates $X$, the distance identity $d_{ij}^2=\|x_i\|^2+\|x_j\|^2-2x_i^Tx_j$ gives $G=XX^T$: multiplying by $H$ removes the row and column norm terms. Diagonalize this <Gram matrix> and retain its $k$ largest positive <eigenvalues>. The coordinates are
$$
\boxed{X_k=U_k\operatorname{diag}(\sqrt{\lambda_1},\ldots,\sqrt{\lambda_k}).}
$$
Thus <classical scaling> exactly recovers Euclidean distances when all positive <eigenvalues> are retained. Reducing dimension approximates the centered <Gram matrix>; it does not in general minimize squared discrepancies in the original distances. Negative <eigenvalues> identify a failure of the <Euclidean distance matrix criterion>, so their size matters when interpreting a display. The historical https://sites.oxy.edu/lengyel/m150/Sueselbeck/helpfiles/cmdscale.html[S-Plus cmdscale help] describes the dimension argument, eigenvalue output and optional additive adjustment to dissimilarities.
Plot the two coordinate columns on equally scaled axes and label observations. Nearby plotted points suggest small dissimilarities only insofar as the dimension reduction fits the input well. The bottom sketches give an exact Euclidean example: a configuration and its independently recovered two-dimensional coordinates have identical pairwise distances, despite different coordinate axes. \b[Classical scaling converts distances into geometry; translations, rotations and reflections do not change the represented distances.] For Euclidean distances between observations, its coordinates agree with the corresponding <principal component scores> up to these harmless choices of axes.
\Image[/past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2004/iii/paper-43-methods.png]
{title=Illustrative PCA directions and scree plot, equivalent recursive partition and tree, and exact classical-scaling recovery of Euclidean distances}
Back to article page
