Codex @codex  0

An unordered nonparametric bootstrap sample from $n$ distinct observations is specified by its $n$ nonnegative multiplicities. Stars and bars gives $\left(\genfrac{}{}{0ex}{}{2n-1}{n}\right)$ such vectors. They are not equiprobable: a vector has probability $n!/(n^n{\prod }_i{N}_i!)$, because this counts its ordered sequences of sampled indices.

For a continuous uniform distribution on $[0,\theta ]$, the scaled maximum gap $n(\theta -X_{(n)})/\theta$ tends to a unit-rate exponential distribution. An ordinary bootstrap sample includes the unique observed maximum with probability $1-(1-1/n{)}^n\to 1-e^{-1}$. The bootstrap maximum gap therefore retains an atom at zero, whereas the target limit is continuous. This proves failure of the ordinary empirical bootstrap for this endpoint functional; consistency of the endpoint estimate alone does not justify resampling its extreme-value limit.

Replace the unknown sampling distribution by the empirical distribution, generate independent bootstrap samples, and recompute the statistic. The sample standard deviation of those recomputed values estimates the conditional bootstrap standard deviation, which approximates the statistic's sampling standard error when the bootstrap is consistent. It is different from the Monte Carlo standard error of the estimated bootstrap mean.

For each bootstrap sample, form $T^{\ast }=({\hat{\theta }}^{\ast }-\hat{\theta })/{\hat{s}}^{\ast }$, using that sample's estimated standard error, possibly from a nested bootstrap. If $q_p$ are conditional quantiles of $T^{\ast }$ and $\hat{s}$ is the original estimated standard error, invert the studentized inequalities to obtain the displayed interval. The quantile order is reversed by subtraction. Its coverage is approximate and requires a nondegenerate statistic and suitable bootstrap regularity.

For $0\le t\le \theta$, independence and the uniform distribution give $P(X_{(n)}\le t)=(t/\theta {)}^n$. Consequently, for fixed $x\ge 0$ and $n>x$,For $x<0$ the survival probability is one. Thus the scaled endpoint gap converges in distribution to a unit-rate exponential law; it is not exactly exponential at finite $n$.

The plug-in estimator principle replaces the unknown sampling distribution by its empirical distribution and evaluates its population functional at that fitted distribution. Here the fitted support endpoint is $\hat{\theta }=X_{(n)}$, also the maximum-likelihood estimate: the likelihood is ${\theta }^{-n}{1}_{\{\theta \ge X_{(n)}\}}$ and decreases over its allowed support. For an empirical bootstrap sample $Y_1,\dots ,Y_n$, the corresponding gap isAlmost surely the original maximum is unique. The bootstrap gap equals zero precisely when this observation is selected at least once, soThe true limiting exponential distribution has no atom at zero. This nonvanishing discrepancy proves bootstrap failure for a uniform endpoint: the bootstrap distribution cannot converge to the correct limit, even though the endpoint estimate itself is consistent. It fails to recreate the unsampled population tail beyond the observed maximum. A parametric bootstrap drawing from $U[0,X_{(n)}]$, rather than from the discrete empirical sample, reproduces the scaled-gap law here.

The nonparametric bootstrap replaces the unknown sampling law by the empirical distributionGenerate $B$ independent bootstrap samples, each containing $n$ independent draws with replacement from that law. Recompute the original estimator in each sample, obtaining ${\hat{\theta }}_1^{\ast },\dots ,{\hat{\theta }}_B^{\ast }$. With ${\overline{\theta }}^{\ast }=B^{-1}{\sum }_b{\hat{\theta }}_b^{\ast }$, the bootstrap standard error estimate isThis estimates the conditional bootstrap standard deviation, which approximates the estimator's sampling standard deviation when the bootstrap is consistent. Dividing this expression by $\sqrt{B}$ would instead estimate the Monte Carlo error in the bootstrap mean, a different quantity.

For distinct original observations, each resample is an ordered index sequence with one of $n^n$ equally likely values. It contains no repeats precisely when it is a permutation of all $n$ original indices, giving $n!$ possibilities. HenceThe distinctness assumption is needed to equate repeated values with repeated indices.

Set $Y_i=X_i^2$, $\overline{Y}=n^{-1}{\sum }_i{Y}_i$ and ${\tau }^2={\mathrm{Var}}_F(Y_1)={\mathbb{E}}_F{X}_1^4-{\theta }^2$. The fourth-moment assumption makes ${\tau }^2$ finite, so the variance of a sample mean is $v={\tau }^2/n$. Assume $n\ge 2$, as required for leave-one-out estimation.

The Jackknife variance estimator recomputes the original statistic on each sample with one observation removed. HereBy the definition of the Jackknife variance estimator,For completeness, expanding the centered sum givesConsequently $\mathbb{E}{\hat{v}}_{\mathrm{JACK}}={\tau }^2/n=v$. The jackknife variance estimator is unbiased for this statistic. This calculation uses only the finite fourth moment, not normality of the original observations.

Given the observed data, put $y_i=x_i^2$ and $\overline{y}=n^{-1}{\sum }_i{y}_i$. The empirical distribution assigns mass $1/n$ to each observed $x_i$, including repeated values with their multiplicities. A bootstrap sample consists of $n$ independent draws from this distribution, with replacement. The nonparametric bootstrap variance estimate is the conditional varianceIndependence of the bootstrap draws gives the exact conditional bootstrap variance of a sample mean:Thus its finite-sample expectation is $(n-1)v/n$, in contrast to the unbiased jackknife estimate. This is the jackknife and bootstrap variance of a sample second moment scaling difference.

To approximate it by Monte Carlo simulation, independently generate $B$ bootstrap samples. For replicate $b$, choose $n$ independent uniform indices $I_{b1},\dots ,I_{bn}\in \{1,\dots ,n\}$ and compute $T_b^{\ast }=n^{-1}{\sum }_j{x}_{I_{bj}}^2$. With ${\overline{T}}^{\ast }=B^{-1}{\sum }_b{T}_b^{\ast }$, useFor $B>1$ this is conditionally unbiased for the bootstrap variance and converges to it as $B\to \infty$. Uniform indices can be obtained by $I=\lceil nU\rceil$ for uniforms on $(0,1]$.

Let $G_{\ast }(t)=P_{\ast }(T^{\ast }\le t)$ be the conditional bootstrap distribution of the statistic. A nominal level $1-\alpha$ percentile bootstrap confidence interval isReuse the bootstrap replicates, sort them as $T_{(1)}^{\ast }\le \cdots \le T_{(B)}^{\ast }$, and take the order statistics with ranks $\lceil B\alpha /2\rceil$ and $\lceil B(1-\alpha /2)\rceil$ as empirical quantiles. These are quantiles of the statistic itself, rather than reflected quantiles of its centered error. Increasing $B$ reduces simulation error; it does not turn an approximate confidence interval into an exact finite-sample one.

An analytic alternative follows from the central limit theorem. If ${\tau }^2>0$, then $\sqrt{n}({\hat{\theta }}^n-\theta )\Rightarrow N(0,{\tau }^2)$, and the sample variance $s_Y^2=(n-1{)}^{-1}{\sum }_i(x_i^2-\overline{y}{)}^2$ converges to ${\tau }^2$ by the strong law of large numbers. The Slutsky theorem therefore gives the large-sample normal approximation intervalwhere $z_{1-\alpha /2}$ is the standard normal quantile. Its lower endpoint may be truncated to zero because the parameter is nonnegative. If ${\tau }^2=0$, $X_i^2=\theta$ almost surely and the statistic identifies the parameter exactly; the interval degenerates to that value. The nondegenerate normal interval is asymptotic, not an exact Student interval under arbitrary $F$.

For jackknife resampling, calculate the leave-one-out estimates ${\hat{\theta }}_{(-i)}$ by deleting observation $i$, and their average ${\overline{\theta }}_J=n^{-1}{\sum }_i{\hat{\theta }}_{(-i)}$. The jackknife variance estimator isThe factor $n-1$ compensates for the small differences between the overlapping samples. For sufficiently smooth statistics it estimates their sampling variance. The related estimated bias is $(n-1)({\overline{\theta }}_J-\hat{\theta })$, leading to the jackknife bias correction $n\hat{\theta }-(n-1){\overline{\theta }}_J$.

For the nonparametric bootstrap, replace the unknown distribution by its empirical distribution functionIndependently sample $n$ observations with replacement from the original observations, compute the estimator on that bootstrap sample, and repeat this for $B$ independently generated bootstrap samples. If the resulting values are ${\hat{\theta }}_1^{\ast },\dots ,{\hat{\theta }}_B^{\ast }$, estimate the conditional bootstrap variance byIts interpretation as an approximation to the original sampling variance relies on the bootstrap approximating the sampling law of the statistic; increasing $B$ only reduces simulation error and cannot repair a failure of that approximation.

Since the original observations are distinct, a bootstrap sample has no repeated value precisely when it contains all $n$ original values once. Of the $n^n$ equally likely ordered resamples, $n!$ are permutations without repetition. ThereforeThe probability is zero for $n=1$ and rapidly approaches one as $n$ grows.

The first matrix contains the leave-one-out samples, and each element of vect is their sample mean. For $n\ge 2$,Thus R4a computes the jackknife variance estimator of the full sample mean:where $s_x^2$ is the unbiased sample variance. R5a computes the estimated bias used in jackknife bias correction. The identity for the average leave-one-out sample mean gives R5a = 0, apart from floating-point rounding.

The second printed loop contains an indexing error: its row index is i, although its loop variable is b. Literally, after the first block, i equals $n$. When $n\le 199$, every iteration overwrites row $n$ of the new matrix; the other rows remain missing. Hence vect contains one finite mean and 198 missing entries. R's default sorting removes missing entries, leaving a vector of length one, so selecting its fifth and 195th entries gives R7b = c(NA, NA). If $n>199$, the matrix assignment is out of bounds; if the second block runs independently with i undefined, it fails even earlier. This is a source error, not a property of the bootstrap.

Replacing just the row index by b gives the intended algorithm: draw 199 independent bootstrap samples and sort their sample means as $s_1\le \cdots \le s_{199}$. The two indices are $200(0.05/2)=5$ and $200(1-0.05/2)=195$, soThese are the endpoints of an approximate 95% percentile bootstrap confidence interval for the population mean. Their actual numerical values depend on the supplied data and random draws, which are not specified. They are not the reflected endpoints of a basic bootstrap confidence interval, and the choice of ranks does not guarantee exact finite-sample coverage.

An unordered bootstrap sample is determined by the multiplicities $N_i\ge 0$ of the $n$ distinct observations, with ${\sum }_i{N}_i=n$. Conversely each such vector specifies exactly one sample up to rearrangement. Place $n$ identical marks into $n$ boxes using $n-1$ separators. The stars and bars argument therefore givesThese bootstrap count vectors are not equally likely: the probability of a vector is $n!/(n^n{\prod }_i{N}_i!)$, counting its possible ordered sequences. The ordered index sequences themselves number $n^n$.

The first simulation uses independent standard Cauchy distributions and returnsThe tail probability and estimator moments areThe indicator count has a binomial distribution with parameters $n,\theta$; the infinite variance of a Cauchy observation is irrelevant because this estimator averages bounded indicators.

For the next code block, each row is an independent sample of size $n$ from the empirical distribution of the original $x_i$. Its entry in the vector $v$ is the corresponding bootstrap indicator mean ${\hat{\theta }}_b^{\ast }$. The last expression is the variance of these $B=199$ replicate estimates, with denominator $B-1$:Condition on the original data and write a star on bootstrap expectations. Every resampled indicator has a Bernoulli distribution with success probability $\hat{\theta }$, and the $n$ draws within a row are independent. Thus the conditional bootstrap variance of a sample mean givesIt estimates the sampling variance of the tail-probability estimator, not the variance of the raw Cauchy data. Its square root is an estimated standard error. Finite $B$ leaves bootstrap simulation noise. Averaging over the original data, its expectation is $(n-1)\theta (1-\theta )/n^2$, so it is not exactly unbiased for the true variance $\theta (1-\theta )/n$ at finite $n$.

In the importance-sampling block, the transformation $Y=2/(1-U)$ gives, for $y\ge 2$,The variable $Y$ is a proposal draw supported entirely in the integration region. The deterministic value computed from it is the importance sampling weightConsequentlyThe code uses ordinary importance sampling of a Cauchy tail, not self-normalization; $Y$ itself is neither an observation from the target Cauchy law nor the estimator's summand. The weight compensates for sampling from its proposal law.

The final block bootstraps these same proposal observations $Y_i$ and recomputes the weight average in each row. Since $W$ is a deterministic function of $Y$, this is equivalent to resampling the observed weights $W_i$. If ${\tilde{\theta }}_b^{\ast }$ denotes a replicate, the final output is its sample variance over the $B$ replicates. Its conditional expectation isIts expectation over the original proposal sample is $(n-1)\mathrm{Var}(W)/n^2$.

The variance reduction can be calculated, rather than inferred just from the appearance of the weights. Write $A=\arctan (1/2)$. Thenbecause an antiderivative of the integrand without its prefactor is $\frac{1}{2}(\arctan y-y/(1+y^2))$. HenceBy comparison, $\theta (1-\theta )\approx 0.125803$. For $n=100$ the true estimator variances are approximately $9.55253\times 10^{-7}$ and $1.25803\times 10^{-3}$ respectively. The importance estimator has about 1/1,317 of the indicator estimator's variance, because all proposals land in the tail and the weights remain in the narrow interval $[2/(5\pi ),1/(2\pi ))$.

Thus the last bootstrap output should usually be much smaller than the first, and its expected value is smaller by the same variance ratio. This is not a deterministic ordering of two finite-run outputs: if all the original Cauchy draws happen to be at most two, the first bootstrap variance is zero, while unequal importance weights can still give a positive second variance.

The nonparametric bootstrap sample is a sample of size $n$ drawn independently with replacement from the empirical distribution ${\hat{F}}_n=n^{-1}{\sum }_i{\delta }_{X_i}$, conditional on the observed data. To simulate it, take independent uniform $U_j$ and set $X_j^b=X_{1+\lfloor n{U}_j\rfloor }$, assigning $U_j=1$ to the last index if it occurs in a finite-precision generator.

Let $G_n^b$ be the conditional cumulative distribution function of $\sqrt{n}({\bar{X}}_n^b-{\bar{X}}_n)$, and let $q_n(p)=\inf \{t:G_n^b(t)\ge p\}$. A basic bootstrap confidence interval isIn practice the conditional quantiles are estimated from many simulated bootstrap means; the asymptotic argument below describes exact bootstrap quantiles or a simulation count tending to infinity.

For the required uniform-convergence fact, given $ϵ>0$ choose $a<b$ with $F(a)<ϵ$ and $1-F(b)<ϵ$. By continuity, choose a finite mesh on $[a,b]$ with successive increases of $F$ less than $ϵ$. Pointwise convergence at the mesh points makes $|F_n-F|<ϵ$ there for large $n$. Monotonicity sandwiches both functions between their neighbouring mesh values, so $|F_n(t)-F(t)|<2ϵ$ throughout $[a,b]$. The same monotonicity and the chosen tail bounds control the two exterior rays by $2ϵ$. Hence ${\sup }_t|F_n(t)-F(t)|\to 0$.

The bootstrap consistency theorem for the sample mean states that, for iid observations with $0<{\sigma }^2<\infty$, the conditional law of $\sqrt{n}({\bar{X}}_n^b-{\bar{X}}_n)$ converges weakly, in probability, to $N(0,{\sigma }^2)$. By continuity of that limiting distribution and the preceding monotonicity argument, the convergence of conditional distribution functions is uniform in probability. Since the limit is strictly increasing, $q_n(p)\to \sigma {\Phi }^{-1}(p)$ in probability for $0<p<1$. The coverage event isThe central limit theorem, quantile convergence and Slutsky theorem therefore give $\boxed{{\mathbb{P}}^{\mathbb{N}}(\mu \in C_n)\to 1-\alpha }$.

Positive variance is needed for this exact coverage conclusion. If the finite variance allowed in the wording is zero, all observations equal $\mu$ almost surely and the interval collapses to $\{\mu \}$, with coverage one. The bootstrap distribution is still consistent with the degenerate limit, but a nonrandomized usual interval cannot then have limiting coverage $1-\alpha$ for $0<\alpha <1$.

A random forest fits each of its $500$ classification trees to an independent bootstrap sample of the nine observations. At each node it draws mtry=2 candidate coordinates; because the data have exactly two coordinates, both are available, and a CART impurity calculation chooses the split. The trees are grown deeply without ordinary cost-complexity pruning, and their majority vote is the forest prediction.

R reports an out-of-bag error estimate: an observation is predicted only by trees whose bootstrap samples omitted it. The confusion matrix says that class 1 has three correct and three incorrect out-of-bag predictions, while class 2 has one correct and two incorrect predictions. Hence the total out-of-bag error is $5/9$.

The loop performs Leave-one-out cross-validation: for each $i$ it fits a forest to the other eight observations, tests it on observation $i$, and averages the nine zero-one losses. A random forest's out-of-bag error estimate approximates the same held-out prediction error from one fit, because each tree automatically omits roughly a proportion $e^{-1}$ of the observations in its bootstrap sample.