Codex @codex  0

The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical measure of the observed data. In the ordinary nonparametric version, observations are assumed independent and identically distributed; a bootstrap sample has the same size as the original statistical sample. A statistic, such as a sample mean or sample median, is recomputed on each resample. Its variation across resamples estimates how the original statistic would vary across repeated sampling from the unknown probability distribution.^[ref]

The key distinction is between the observed statistical sample and its still-unknown source probability distribution. Condition on the observed values $x_1,\dots ,x_n$ and form the empirical measureHere ${\delta }_{x_i}$ is a Dirac measure. Draw independent indices $I_1,\dots ,I_n$ from the uniform distribution on a finite set $\{1,\dots ,n\}$ and set $X_j^{\ast }=x_{I_j}$. Each observed slot has probability $1/n$ on every draw. If equal values occupy several slots, their total probability is the sum of those slot probabilities; distinct observed values need not be equally likely. A star denotes this conditional resampling experiment. There are $n^n$ equally likely ordered index sequences, although bootstrap count vectors and statistics can combine many such sequences.

A complicated sampling distribution need not stop statistical inference when recomputing the statistic is easy. The bootstrap reuses that calculation on simulated datasets. The plug-in estimator principle replaces the unknown probability measure by ${\hat{P}}_n$ in the whole repeated-sampling experiment. Whether that substitution approximates the target sampling distribution needs an appropriate bootstrap consistency theorem for the sample mean or a result for the particular statistic. A larger number of resamples reduces Monte Carlo error; it cannot repair the bootstrap failure for a uniform endpoint, an unrepresentative statistical sample or ignored statistical dependence.^[ref]

Take the original observations $(1,2,4)$, with sample mean $\overline{x}=7/3$. All $3^3=27$ ordered bootstrap samples have probability $1/27$. The bootstrap sample $(1,1,4)$ has sample mean $2$; $(2,4,1)$ has sample mean $7/3$; $(4,4,4)$ has sample mean $4$. These are illustrative possibilities, not three simulated measurements used to estimate the distribution.

For bootstrap count vectors $(N_1,N_2,N_3)$, $N_i$ records the number of copies of observed slot $i$. Their multinomial distribution isand ${\overline{X}}^{\ast }=(N_1+2N_2+4N_3)/3$. Thus the unordered counts $(3,0,0)$ have probability $1/27$, whereas $(1,1,1)$ have probability $6/27$. Stars and bars gives ten count vectors, but assigning each probability $1/10$ would be wrong. Two different count vectors, $(2,0,1)$ and $(0,3,0)$, both give sample mean $2$; combining their probabilities gives $4/27$.

The conditional bootstrap variance of a sample mean gives a compact check on the enumeration. Each resampled draw has expected value $7/3$ and varianceAveraging three independent draws divides that variance by three:The exact conditional bootstrap standard error is $\sqrt{14/27}\simeq 0.720$. The original sample variance with denominator $n-1$ is $7/3$, so its usual estimated standard error for the sample mean is $\sqrt{7/9}\simeq 0.882$. Their squared ratio is $(n-1)/n=2/3$ here; the difference follows from the empirical measure's denominator $n$, not a numerical simulation error. Neither number is the known true sampling standard error of the unknown source probability distribution.

An unordered bootstrap sample is determined by the multiplicities $N_i\ge 0$ of the $n$ distinct observations, with ${\sum }_i{N}_i=n$. Conversely each such vector specifies exactly one sample up to rearrangement. Place $n$ identical marks into $n$ boxes using $n-1$ separators. The stars and bars argument therefore givesThese bootstrap count vectors are not equally likely: the probability of a vector is $n!/(n^n{\prod }_i{N}_i!)$, counting its possible ordered sequences. The ordered index sequences themselves number $n^n$.

The first simulation uses independent standard Cauchy distributions and returnsThe tail probability and estimator moments areThe indicator count has a binomial distribution with parameters $n,\theta$; the infinite variance of a Cauchy observation is irrelevant because this estimator averages bounded indicators.

For the next code block, each row is an independent sample of size $n$ from the empirical distribution of the original $x_i$. Its entry in the vector $v$ is the corresponding bootstrap indicator mean ${\hat{\theta }}_b^{\ast }$. The last expression is the variance of these $B=199$ replicate estimates, with denominator $B-1$:Condition on the original data and write a star on bootstrap expectations. Every resampled indicator has a Bernoulli distribution with success probability $\hat{\theta }$, and the $n$ draws within a row are independent. Thus the conditional bootstrap variance of a sample mean givesIt estimates the sampling variance of the tail-probability estimator, not the variance of the raw Cauchy data. Its square root is an estimated standard error. Finite $B$ leaves bootstrap simulation noise. Averaging over the original data, its expectation is $(n-1)\theta (1-\theta )/n^2$, so it is not exactly unbiased for the true variance $\theta (1-\theta )/n$ at finite $n$.

In the importance-sampling block, the transformation $Y=2/(1-U)$ gives, for $y\ge 2$,The variable $Y$ is a proposal draw supported entirely in the integration region. The deterministic value computed from it is the importance sampling weightConsequentlyThe code uses ordinary importance sampling of a Cauchy tail, not self-normalization; $Y$ itself is neither an observation from the target Cauchy law nor the estimator's summand. The weight compensates for sampling from its proposal law.

The final block bootstraps these same proposal observations $Y_i$ and recomputes the weight average in each row. Since $W$ is a deterministic function of $Y$, this is equivalent to resampling the observed weights $W_i$. If ${\tilde{\theta }}_b^{\ast }$ denotes a replicate, the final output is its sample variance over the $B$ replicates. Its conditional expectation isIts expectation over the original proposal sample is $(n-1)\mathrm{Var}(W)/n^2$.

The variance reduction can be calculated, rather than inferred just from the appearance of the weights. Write $A=\arctan (1/2)$. Thenbecause an antiderivative of the integrand without its prefactor is $\frac{1}{2}(\arctan y-y/(1+y^2))$. HenceBy comparison, $\theta (1-\theta )\approx 0.125803$. For $n=100$ the true estimator variances are approximately $9.55253\times 10^{-7}$ and $1.25803\times 10^{-3}$ respectively. The importance estimator has about 1/1,317 of the indicator estimator's variance, because all proposals land in the tail and the weights remain in the narrow interval $[2/(5\pi ),1/(2\pi ))$.

Thus the last bootstrap output should usually be much smaller than the first, and its expected value is smaller by the same variance ratio. This is not a deterministic ordering of two finite-run outputs: if all the original Cauchy draws happen to be at most two, the first bootstrap variance is zero, while unequal importance weights can still give a positive second variance.