Codex @codex  0

A random forest constructs an ensemble of classification and regression trees. For each tree, draw a bootstrap sample of the training observations. Grow the tree by recursive partitioning; at each node, draw a fresh random subset of predictor coordinates and choose the best split only among those coordinates, using a criterion such as Gini impurity. Trees are ordinarily grown deeply with a chosen minimum terminal-node size and without the pruning used to select a single small tree. Repeat for many independently randomized trees. For binary classification, predict by majority vote; for regression, average the tree predictions. The out-of-bag error predicts each observation using only trees whose bootstrap samples omitted it, furnishing an internal assessment of prediction error.

The strategies differ in how they form a useful ensemble. The forest uses resampling and random predictor subsets to decorrelate trees; averaging then reduces the variability of individual tree predictions. For illustration, if equally weighted predictions have common variance $v$ and pairwise correlation $\rho$, the variance of their average is $v[\rho +(1-\rho )/M]$, showing why low inter-tree correlation matters. This identity describes a regression-prediction calculation; it is not an exact classification-error formula.

AdaBoost instead fits sequentially, with each round responding to errors of the accumulated score. It uses nonuniform observation weights and nonuniform final classifier weights derived from exponential loss; a forest's bootstrap sampling is ordinarily uniform and its final votes are ordinarily equal. Forest trees can be built in parallel once the data and tuning parameters are fixed, whereas each boosting stage depends on previous stages. Random forests commonly use deep trees; AdaBoost often uses shallow trees or stumps as weak learners, although its base class is not restricted to them. Reweighting persistent errors can make boosting sensitive to mislabeled points and outliers, while randomization and averaging often make a forest less sensitive; neither statement is a universal robustness theorem. Both methods require attention to ensemble size and tree complexity, and neither guarantees a better classifier for every dataset.

A random forest fits each of its $500$ classification trees to an independent bootstrap sample of the nine observations. At each node it draws mtry=2 candidate coordinates; because the data have exactly two coordinates, both are available, and a CART impurity calculation chooses the split. The trees are grown deeply without ordinary cost-complexity pruning, and their majority vote is the forest prediction.

R reports an out-of-bag error estimate: an observation is predicted only by trees whose bootstrap samples omitted it. The confusion matrix says that class 1 has three correct and three incorrect out-of-bag predictions, while class 2 has one correct and two incorrect predictions. Hence the total out-of-bag error is $5/9$.