Bootstrapping (statistics)

The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical measure of the observed data. In the ordinary nonparametric version, observations are assumed independent and identically distributed; a bootstrap sample has the same size as the original statistical sample. A statistic, such as a sample mean or sample median, is recomputed on each resample. Its variation across resamples estimates how the original statistic would vary across repeated sampling from the unknown probability distribution.^[ref]

The key distinction is between the observed statistical sample and its still-unknown source probability distribution. Condition on the observed values

x_{1}, \dots, x_{n}

and form the empirical measure

P_{n} = \frac{1}{n} \sum_{i = 1}^{n} δ_{x_{i}} .

(1)

Here

δ_{x_{i}}

is a Dirac measure. Draw independent indices

I_{1}, \dots, I_{n}

from the uniform distribution on a finite set

{1, \dots, n}

and set

X_{j}^{*} = x_{I_{j}}

. Each observed slot has probability

1/ n

on every draw. If equal values occupy several slots, their total probability is the sum of those slot probabilities; distinct observed values need not be equally likely. A star denotes this conditional resampling experiment. There are

n^{n}

equally likely ordered index sequences, although bootstrap count vectors and statistics can combine many such sequences.

Figure 1.
Sampling with replacement reuses observed slots while keeping the original data fixed
. Original schematic for observations $(1, 2, 4)$ . Three possible ordered bootstrap samples have different sample means. Their colors identify source slots; none of these samples is new evidence about the unknown probability distribution. **Details**.

A complicated sampling distribution need not stop statistical inference when recomputing the statistic is easy. The bootstrap reuses that calculation on simulated datasets. The plug-in estimator principle replaces the unknown probability measure by

P_{n}

in the whole repeated-sampling experiment. Whether that substitution approximates the target sampling distribution needs an appropriate bootstrap consistency theorem for the sample mean or a result for the particular statistic. A larger number of resamples reduces Monte Carlo error; it cannot repair the bootstrap failure for a uniform endpoint, an unrepresentative statistical sample or ignored statistical dependence.^[ref]

Table of contents

Exact bootstrap distribution for three observations ( $P_{} (\overline{X}^{} = 7/3) = 2/9$ )

 0  0

Take the original observations

(1, 2, 4)

, with sample mean

\overline{x} = 7/3

. All

3^{3} = 27

ordered bootstrap samples have probability

1/27

. The bootstrap sample

(1, 1, 4)

has sample mean

2

;

(2, 4, 1)

has sample mean

7/3

;

(4, 4, 4)

has sample mean

4

. These are illustrative possibilities, not three simulated measurements used to estimate the distribution.

For bootstrap count vectors

(N_{1}, N_{2}, N_{3})

N_{i}

records the number of copies of observed slot

i

. Their multinomial distribution is

P_{*} ((N_{1}, N_{2}, N_{3}) = (a, b, c)) = \frac{3 !}{a ! b ! c !} \frac{1}{27}, a + b + c = 3,

(1)

and

\overline{X}^{*} = (N_{1} + 2 N_{2} + 4 N_{3}) /3

. Thus the unordered counts

(3, 0, 0)

have probability

1/27

, whereas

(1, 1, 1)

have probability

6/27

. Stars and bars gives ten count vectors, but assigning each probability

1/10

would be wrong. Two different count vectors,

(2, 0, 1)

and

(0, 3, 0)

, both give sample mean

2

; combining their probabilities gives

4/27

Figure 1.
Exact conditional sampling distribution of the sample mean from observations $(1, 2, 4)$
. All 27 ordered bootstrap samples are enumerated. Bar labels give their counts out of 27, not empirical frequencies from a random simulation. The dashed line marks $\overline{x} = 7/3$ ; the gaps reflect this discrete three-point empirical measure. **Details**.

The conditional bootstrap variance of a sample mean gives a compact check on the enumeration. Each resampled draw has expected value

7/3

and variance

\frac{1}{3} [(1 - \frac{7}{3})^{2} + (2 - \frac{7}{3})^{2} + (4 - \frac{7}{3})^{2}] = \frac{14}{9} .

(2)

Averaging three independent draws divides that variance by three:

E_{*} [\overline{X}^{*}] = \frac{7}{3}, Var_{*} (\overline{X}^{*}) = \frac{14}{27} .

(3)

The exact conditional bootstrap standard error is

14/27 ≃ 0.720

. The original sample variance with denominator

n - 1

7/3

, so its usual estimated standard error for the sample mean is

7/9 ≃ 0.882

. Their squared ratio is

(n - 1) / n = 2/3

here; the difference follows from the empirical measure's denominator

n

, not a numerical simulation error. Neither number is the known true sampling standard error of the unknown source probability distribution.

Bootstrap publication history

 0  0

Bootstrap Methods: Another Look at the Jackknife by Bradley Efron, 1979 journal article appeared in The Annals of Statistics 7(1), pp. 1–26, in January 1979. The original article's bibliographic cover identifies the author, journal, date and pages.^[ref] Stanford's earlier technical report BIO 32, also by Bradley Efron and with the same title, is dated July 1977; that report date is distinct from the journal publication date.^[ref]

Bootstrap failure for a uniform endpoint ( $P_{} (X_{(n)}^{} = X_{(n)}) = 1 - (1 - 1/ n)^{n}$ )

 0  0

For a continuous uniform distribution on

[0, θ]

, the scaled maximum gap

n (θ - X_{(n)}) / θ

tends to a unit-rate exponential distribution. An ordinary bootstrap sample includes the unique observed maximum with probability

1 - (1 - 1/ n)^{n} \to 1 - e^{- 1}

. The bootstrap maximum gap therefore retains an atom at zero, whereas the target limit is continuous. This proves failure of the ordinary empirical bootstrap for this endpoint functional; consistency of the endpoint estimate alone does not justify resampling its extreme-value limit.

Bootstrap standard error ( $se_{} = (Var_{} θ^{*})^{1/2}$ )

 0  0

Replace the unknown sampling distribution by the empirical distribution, generate independent bootstrap samples, and recompute the statistic. The sample standard deviation of those recomputed values estimates the conditional bootstrap standard deviation, which approximates the statistic's sampling standard error when the bootstrap is consistent. It is different from the Monte Carlo standard error of the estimated bootstrap mean.

Bootstrap confidence interval

 0  0

A bootstrap confidence interval uses the conditional resampling distribution of an estimator to choose its endpoints. Basic intervals invert bootstrap error quantiles; percentile intervals use quantiles of the resampled estimates themselves. Their asymptotic coverage requires conditions appropriate to the statistic and interval construction.

Bootstrap inference for a product of regression coefficients ( $θ = β_{1} β_{2}$ )

 0  0

For

θ = β_{1} β_{2}

, use

θ = β_{1} β_{2}

. Under regular regression assumptions, the delta method gives standard error

s = (d^{⊤} V d)^{1/2}

, where

d = (β_{2}, β_{1})^{⊤}

and

V

estimates the covariance of the regression estimate. Resample the observational units, refit, and compute both

θ^{*}

and

s^{*}

. A percentile bootstrap confidence interval uses the .025 and .975 quantiles of

θ^{*}

. A bootstrap-t confidence interval uses quantiles of

(θ^{*} - θ) / s^{*}

and reverses them when solving for

θ

. If both true coefficients vanish, the first derivative is zero and the standard first-order justification is unavailable; product-specific nonregular inference is then needed.

Bootstrap-t confidence interval ( $[θ - q_{.975} s, θ - q_{.025} s]$ )

 0  0

For each bootstrap sample, form

T^{*} = (θ^{*} - θ) / s^{*}

, using that sample's estimated standard error, possibly from a nested bootstrap. If

q_{p}

are conditional quantiles of

T^{*}

and

s

is the original estimated standard error, invert the studentized inequalities to obtain the displayed interval. The quantile order is reversed by subtraction. Its coverage is approximate and requires a nondegenerate statistic and suitable bootstrap regularity.

Percentile bootstrap confidence interval

 0  0

A percentile bootstrap confidence interval takes the lower and upper chosen quantiles of the conditional bootstrap distribution of the estimator itself. With

B

simulated estimates, ranks near

(B + 1) α /2

and

(B + 1) (1 - α /2)

provide an approximate central

1 - α

confidence interval. Unlike a basic bootstrap confidence interval, the endpoints are not reflected about the observed estimate. Coverage requires suitable regularity and accurate bootstrap approximation; the ranks do not establish exact finite-sample coverage.

Basic bootstrap confidence interval

 0  0

A basic bootstrap interval reflects quantiles of the bootstrap estimation error about the observed estimate. For the sample mean it uses

[\overset{ˉ}{X} - q_{1 - α /2} / n, \overset{ˉ}{X} - q_{α /2} / n]

, where

q

is a conditional quantile of

n (\overset{ˉ}{X}^{b} - \overset{ˉ}{X})

Bootstrap consistency theorem for the sample mean

 0  0

For iid observations with finite positive variance, the conditional distribution of

n (\overset{ˉ}{X}^{b} - \overset{ˉ}{X})

tends to the same normal limit as

n (\overset{ˉ}{X} - μ)

. Continuity of this limit gives uniform convergence of distribution functions and consistency of interior quantiles.

Bootstrap sample

 0  0

A bootstrap sample of size

n

is drawn by sampling with replacement from the

n

observed data slots. Some observations occur repeatedly. For a given slot, every draw misses it with probability

1 - 1/ n

, so its omission probability after

n

independent draws is

P_{*} (N_{i} = 0) = (1 - \frac{1}{n})^{n} ⟶ e^{- 1} .

(1)

By linearity of expectation, the expected number of omitted slots is

n (1 - 1/ n)^{n}

, whether or not their omission events are independent. For the three-observation exact bootstrap distribution for three observations, an individual slot is omitted with probability

8/27

and the expected number of omitted slots is

8/9

. This finite-sample calculation explains both repeated values and incomplete coverage;

e^{- 1}

is a limiting expected proportion, not an exact fraction in every resample.

Wild bootstrap ( $y_{i}^{*} = x_{i}^{⊤} β + e_{i} W_{i}$ )

 0  0

For independent mean-zero errors in a fixed-design regression, form

y_{i}^{*} = x_{i}^{⊤} β + e_{i} W_{i}

, where the independent multipliers have mean zero and variance one, and refit. Rademacher multipliers taking the values

- 1

and

1

with equal probability are a simple choice. Conditional on the data, the bootstrap errors have mean zero and preserve observation-specific residual variances, supporting heteroscedasticity rather than imposing a common residual distribution. Appropriate leverage corrections and regularity assumptions are required for accurate studentized inference.

Paired bootstrap

 0  0

For independent identically distributed pairs

(X_{i}, Y_{i})

, resample whole pairs using the same index for both coordinates. This preserves the empirical within-pair dependence needed to estimate a correlation coefficient or another joint statistic. Independently resampling the two margins destroys that dependence and estimates a different sampling law.

Conditional bootstrap variance of a sample mean ( $Var_{} (\overline{a}^{}) = n^{- 2} \sum_{i} (a_{i} - \overline{a})^{2}$ )

 0  0

Conditionally on the observed values, bootstrap draws are iid from their empirical distribution. Each draw has variance

n^{- 1} \sum_{i} (a_{i} - \overline{a})^{2}

, so averaging

n

independent draws gives the displayed variance. The variance across

B

independent replicate means, using denominator

B - 1

, is unbiased for this conditional quantity. Its expectation over the original iid data is

(n - 1) Var (a_{1}) / n^{2}

, which is slightly smaller than the actual sampling variance

Var (a_{1}) / n

Bootstrap count vectors ( $N_{1} + \dots + N_{n} = n$ )

 0  0

An unordered nonparametric bootstrap sample from

n

distinct observations is specified by its

n

nonnegative multiplicities. Stars and bars gives

(n 2 n - 1)

such vectors. They are not equiprobable: a vector has probability

n! / (n^{n} \prod_{i} N_{i}!)

, because this counts its ordered sequences of sampled indices.

Parametric bootstrap

 0  0

A parametric bootstrap simulates repeated datasets from a fitted parametric model and refits the procedure to each dataset. The resulting empirical distribution approximates the sampling distribution under that fitted model.

 Ancestors (6)

 Incoming links (1)

Past exam of the mathematics course of the University of Cambridge / 2021 / ii / Paper 3 / 28J / c / Solution

 Synonyms (2)

 View article source

 Discussion (0) New discussion

There are no discussions about this article yet.

 Articles by others on the same topic (1)

Bootstrapping (statistics) by

Wikipedia Bot  1

 View more

Bootstrapping is a statistical resampling technique used to estimate the distribution of a sample statistic by repeatedly resampling with replacement from the data set. The central idea is to create multiple simulated samples (called "bootstrap samples"), allowing for the assessment of variability and confidence intervals of the statistic of interest without relying on strong parametric assumptions. ### Key Steps in Bootstrapping: 1. **Original Sample**: Start with an observed dataset of size $ n $.

 Read the full article

  See all articles in the same topic Create my own version

 Discussion (0)  Subscribe (1) New discussion

 Discussion (0) New discussion