= Solution
A <random forest> constructs an ensemble of <classification and regression trees>. For each tree, draw a <bootstrap> sample of the training observations. Grow the tree by <recursive partitioning>; at each node, draw a fresh random subset of predictor coordinates and choose the best split only among those coordinates, using a criterion such as <Gini impurity>. Trees are ordinarily grown deeply with a chosen minimum terminal-node size and without the pruning used to select a single small tree. Repeat for many independently randomized trees. For binary classification, predict by <majority vote>; for regression, average the tree predictions. The <out-of-bag error> predicts each observation using only trees whose bootstrap samples omitted it, furnishing an internal assessment of prediction error.
The strategies differ in how they form a useful ensemble. The forest uses resampling and random predictor subsets to decorrelate trees; averaging then reduces the variability of individual tree predictions. For illustration, if equally weighted predictions have common variance $v$ and pairwise correlation $\rho$, the variance of their average is $v[\rho+(1-\rho)/M]$, showing why low inter-tree correlation matters. This identity describes a regression-prediction calculation; it is not an exact classification-error formula.
<AdaBoost> instead fits sequentially, with each round responding to errors of the accumulated score. It uses nonuniform observation weights and nonuniform final classifier weights derived from exponential loss; a forest's bootstrap sampling is ordinarily uniform and its final votes are ordinarily equal. Forest trees can be built in parallel once the data and tuning parameters are fixed, whereas each boosting stage depends on previous stages. Random forests commonly use deep trees; AdaBoost often uses shallow trees or stumps as <weak learners>, although its base class is not restricted to them. Reweighting persistent errors can make boosting sensitive to mislabeled points and outliers, while randomization and averaging often make a forest less sensitive; neither statement is a universal robustness theorem. Both methods require attention to ensemble size and tree complexity, and neither guarantees a better classifier for every dataset.
\b[Random forests diversify and average randomized trees; AdaBoost sequentially fits errors and combines classifiers with loss-derived weights.]
Back to article page
