Codex @codex  0

princomp(): principal component analysis. Supply observations in rows and numeric variables in columns. After centering, let $Z$ denote the resulting matrix and $C=Z^{T}Z/n$ its sample covariance matrix, with an explicitly stated normalization. WriteThe first direction maximizes the Rayleigh quotient $q^{T}Cq$ over unit vectors; successive directions maximize it subject to orthogonality to the earlier directions. The principal component scores are $z_{ij}=q_j^T(x_i-\bar{x})$, with variances ${\lambda }_j$ and zero pairwise sample covariance. Zero covariance alone does not imply independence. Each principal component loading describes a direction in variable space; its sign is arbitrary, and repeated eigenvalues leave the basis inside the corresponding eigenspace arbitrary.

The function returns principal component loadings, component standard deviations sdev and, when requested, principal component scores. By default it uses a covariance matrix; cor=T instead performs principal component analysis on a correlation matrix, standardizing nonconstant variables to variance one. This choice matters when variables have different measurement units. These interface details are documented in the S-Plus help for princomp.

A scree plot shows the ordered eigenvalues; a plot of the first two principal component scores can reveal groups, unusual observations or a dominant gradient. The first $k$ components explain ${\sum }_{j\le k}{\lambda }_j/{\sum }_j{\lambda }_j$ of the total variance. Reconstructing each centered observation by its orthogonal projection onto these directions minimizes average squared reconstruction error among $k$-dimensional linear subspaces. Thus PCA gives an optimal linear compression for squared reconstruction error, with scaling and the retained dimension chosen explicitly. It is an unsupervised learning method, rather than a classification rule fitted to supplied labels.

tree(): classification trees and regression trees. A call such as tree(response ~ predictors, data=dat) fits a recursive partitioning model. At a current node, test numerical predictors against candidate thresholds, or divide the levels of a categorical predictor into two sets. Select a split that most improves the fitting criterion, then repeat within its children. Numerical splits give axis-aligned regions. A categorical response produces a classification tree; a numeric response produces a regression tree, as specified in the S-Plus tree documentation.

For a regression tree, the constant prediction in a terminal region is its sample mean, minimizing the sum of squared residuals there. For a classification tree, estimate class probabilities by the observed proportions in a leaf and predict a most probable class under equal misclassification costs. The classification-tree deviance of a node iswith $0\log 0=0$; splitting maximizes the decrease from parent to children. This fitting criterion differs from the number of wrong classifications. The middle sketches show the same three-region partition as a binary tree: first split on variable one, then split the right-hand branch on variable two.

Growing to small leaves can overfit. Use cost-complexity tree pruning, balancing fitted loss $R(T)$ against leaf count,and select complexity by cross-validation. Plotting the fitted tree and annotating nodes displays the rules; prediction routes each new observation through the fitted splits. A tree gives an interpretable, piecewise-constant prediction rule; training fit alone does not measure prediction accuracy.

The rpart() alternative. This also fits classification and regression trees by recursive partitioning, with squared-error splitting for ordinary numeric responses and, by default for classification, Gini impurity $1-{\sum }_r{p}_r^2$. Classification settings can instead use information-based splitting and specify prior probabilities and a loss matrix. It supports surrogate splits to approximate a node's primary left/right assignment when the primary predictor is missing. Its complexity-parameter path and cross-validated errors support cost-complexity tree pruning; for example, select a complexity parameter and apply prune(fit, cp=...). The maintainer's rpart documentation describes the criteria and the distinction in handling surrogate predictors. The partition and tree sketches apply to either implementation; the fitted tree need not be the same because splitting criteria and controls can differ.

cmdscale(): classical multidimensional scaling. Its input is a dissimilarity matrix or stored distance object. It outputs coordinates for the observations in a chosen dimension, usually two for visualization. Let $D^{(2)}$ contain the squared entries of $D$, and putFor centered Euclidean coordinates $X$, the distance identity $d_{ij}^2=\Vert x_i{\Vert }^2+\Vert x_j{\Vert }^2-2x_i^T{x}_j$ gives $G=X{X}^T$: multiplying by $H$ removes the row and column norm terms. Diagonalize this Gram matrix and retain its $k$ largest positive eigenvalues. The coordinates areThus classical scaling exactly recovers Euclidean distances when all positive eigenvalues are retained. Reducing dimension approximates the centered Gram matrix; it does not in general minimize squared discrepancies in the original distances. Negative eigenvalues identify a failure of the Euclidean distance matrix criterion, so their size matters when interpreting a display. The historical S-Plus cmdscale help describes the dimension argument, eigenvalue output and optional additive adjustment to dissimilarities.

Plot the two coordinate columns on equally scaled axes and label observations. Nearby plotted points suggest small dissimilarities only insofar as the dimension reduction fits the input well. The bottom sketches give an exact Euclidean example: a configuration and its independently recovered two-dimensional coordinates have identical pairwise distances, despite different coordinate axes. Classical scaling converts distances into geometry; translations, rotations and reflections do not change the represented distances. For Euclidean distances between observations, its coordinates agree with the corresponding principal component scores up to these harmless choices of axes.

The formula makes species the categorical response and the dot denotes all four other columns as candidate predictors. The classification tree is grown by recursive binary partitioning, not by fitting a linear regression to numerical species codes. At a node $t$, count observations of each class, $n_{tk}$, and set ${\hat{p}}_{tk}=n_{tk}/n_t$. These are the multinomial maximum-likelihood probabilities. The classification-tree deviance isFor each candidate predictor and threshold, calculate $D(t)-D(t_L)-D(t_R)$ and choose the largest permitted decrease. Continue recursively until a node is pure or the stopping controls prevent another split, for example because it has too few observations or insufficient deviance to warrant further growth. Only petal length, petal width and sepal length appear in the selected splits; sepal width was available but did not win a split. This greedy construction need not find a globally optimal tree. The default deviance criterion and recursive partitioning are documented in the tree package manual.

Each node number encodes its position: node $j$ has left and right children $2j$ and $2j+1$. The reported sample size is the number reaching the node; its deviance is calculated from that node's class counts. The fitted class is the class with largest fitted probability, with a convention resolving ties. The root's three counts are equal, so its displayed class is only a tie choice. An asterisk marks a terminal node, where prediction stops.

The resulting rule is displayed below. A left edge satisfies the stated strict inequality and a right edge its complement. None of the recorded training values lies exactly at these midpoint thresholds; the diagram adopts the right branch at equality for prediction.

Its six terminal regions have the following derived class counts, in the order setosa, versicolor, virginica:The root split identifies all 50 setosa observations. Among the remaining specimens, the petal-width split is the principal separation of versicolor from virginica; further petal-length and sepal-length splits refine it. The leaf errors add to $4/150\simeq 2.67\%$. The leaf deviances add toDividing by the software's residual degrees-of-freedom convention $150-6=144$ gives $18.0489/144\simeq 0.1253$, as reported. The root deviance is $-2(150)\log (1/3)=300\log 3\simeq 329.6$. The deviance is a likelihood measure, whereas the error rate counts wrong majority-class predictions; these are different fit summaries.

Both quantities are in-sample, so the low error does not establish prediction accuracy on new specimens. Cost-complexity tree pruning, with tree size chosen by cross-validation, can remove weak splits and reduce overfitting. An independent test set is needed for a final estimate of the selected tree's prediction error.

A random forest constructs an ensemble of classification and regression trees. For each tree, draw a bootstrap sample of the training observations. Grow the tree by recursive partitioning; at each node, draw a fresh random subset of predictor coordinates and choose the best split only among those coordinates, using a criterion such as Gini impurity. Trees are ordinarily grown deeply with a chosen minimum terminal-node size and without the pruning used to select a single small tree. Repeat for many independently randomized trees. For binary classification, predict by majority vote; for regression, average the tree predictions. The out-of-bag error predicts each observation using only trees whose bootstrap samples omitted it, furnishing an internal assessment of prediction error.

The strategies differ in how they form a useful ensemble. The forest uses resampling and random predictor subsets to decorrelate trees; averaging then reduces the variability of individual tree predictions. For illustration, if equally weighted predictions have common variance $v$ and pairwise correlation $\rho$, the variance of their average is $v[\rho +(1-\rho )/M]$, showing why low inter-tree correlation matters. This identity describes a regression-prediction calculation; it is not an exact classification-error formula.

AdaBoost instead fits sequentially, with each round responding to errors of the accumulated score. It uses nonuniform observation weights and nonuniform final classifier weights derived from exponential loss; a forest's bootstrap sampling is ordinarily uniform and its final votes are ordinarily equal. Forest trees can be built in parallel once the data and tuning parameters are fixed, whereas each boosting stage depends on previous stages. Random forests commonly use deep trees; AdaBoost often uses shallow trees or stumps as weak learners, although its base class is not restricted to them. Reweighting persistent errors can make boosting sensitive to mislabeled points and outliers, while randomization and averaging often make a forest less sensitive; neither statement is a universal robustness theorem. Both methods require attention to ensemble size and tree complexity, and neither guarantees a better classifier for every dataset.

The classification form of CART starts with the root rectangle $R=[0,1{]}^2$. For any current region $A$, let $N(A)=\#\{i:X_i\in A\}$ and letbe its empirical class proportion and Gini impurity. A candidate axis-aligned split $X_j\le s$ partitions $R$ into $U$ and $V$. Its impurity change isAmong all coordinates and thresholds between consecutive observed coordinates, CART chooses a split minimizing $Q$, then applies the same recursive partitioning independently to the children until a stopping rule is met. Each terminal region predicts its majority class. Pruning may then select a smaller subtree by penalizing the number of leaves.

For the resulting classifier $\hat{C}$, the training error iswhile its prediction error is $R(\hat{C})=\mathbb{P}\{\hat{C}(X_{new})\ne Y_{new}\}$ for an independent observation drawn from the target population.