Codex @codex  0

The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical measure of the observed data. In the ordinary nonparametric version, observations are assumed independent and identically distributed; a bootstrap sample has the same size as the original statistical sample. A statistic, such as a sample mean or sample median, is recomputed on each resample. Its variation across resamples estimates how the original statistic would vary across repeated sampling from the unknown probability distribution.^[ref]

The key distinction is between the observed statistical sample and its still-unknown source probability distribution. Condition on the observed values $x_1,\dots ,x_n$ and form the empirical measureHere ${\delta }_{x_i}$ is a Dirac measure. Draw independent indices $I_1,\dots ,I_n$ from the uniform distribution on a finite set $\{1,\dots ,n\}$ and set $X_j^{\ast }=x_{I_j}$. Each observed slot has probability $1/n$ on every draw. If equal values occupy several slots, their total probability is the sum of those slot probabilities; distinct observed values need not be equally likely. A star denotes this conditional resampling experiment. There are $n^n$ equally likely ordered index sequences, although bootstrap count vectors and statistics can combine many such sequences.

A complicated sampling distribution need not stop statistical inference when recomputing the statistic is easy. The bootstrap reuses that calculation on simulated datasets. The plug-in estimator principle replaces the unknown probability measure by ${\hat{P}}_n$ in the whole repeated-sampling experiment. Whether that substitution approximates the target sampling distribution needs an appropriate bootstrap consistency theorem for the sample mean or a result for the particular statistic. A larger number of resamples reduces Monte Carlo error; it cannot repair the bootstrap failure for a uniform endpoint, an unrepresentative statistical sample or ignored statistical dependence.^[ref]

The coordinatewise median of vectors applies the univariate sample median separately to each coordinate. It is easy to compute but depends on the chosen coordinate axes.

Here $d_H$ counts the observations replaced. This curve measures the largest change after at most $m$ arbitrary replacements. Under the first-unbounded convention, the replacement breakdown point is $n^{-1}\min \{m:b_n(m;y)=\infty \}$. The largest fraction still guaranteed bounded is one grid step smaller; those conventions must not be confused. A sample mean breaks after one replacement, while a scalar sample median with a fixed ordinary tie convention needs about half the sample replaced.

Uniform convergence in probability of empirical criteria on a compact set proves consistency for minimizers in that set; it does not confine a minimizer over a larger domain. A bounded identifiable regression function can approach its true value again at infinity. For median regression, $a(\theta )={\theta }^2/(1+{\theta }^4)$ has unique zero at zero, but a small positive fitted sample median $c$ has a second solution ${\theta }^2=(1+\sqrt{1-4c^2})/(2c)$. This solution diverges as $c\downarrow 0$, although uniform convergence on any fixed compact interval holds. Constraining the estimator, or proving suitable global separation and localization, repairs the argument.

The Huber location estimator solves ${\sum }_i{\psi }_k(x_i-\theta )=0$, where ${\psi }_k$ clips its argument to $[-k,k]$.

More generally, fix a known scale $s>0$ and a dimensionless threshold $k>0$ before fitting. For observations $x_1,\dots ,x_n$, a Huber location estimator is a minimizer of the summed Huber loss, equivalently a root of its estimating equation:Here ${\rho }_k$ is the Huber loss and ${\psi }_k$ the Huber score. The physical clipping distance is $ks$, in the observations' units; the initial formula uses $s=1$. A large observation keeps contributing a bounded signed score rather than being discarded. The original clipped location criterion appears in Peter J. Huber's 1964 paper.^[ref]

For the original data $(0,1,2,3,4,40)$, take $s=1$ and $k=2$. At $\theta =2.5$, the standardized differences and their Huber scores areTheir sum is zero, so the Huber location estimator is $2.5$. Four scores remain in the central linear region, making the root unique. By comparison, the sample mean is $50/6\simeq 8.3333$, and the sample median, using the midpoint of the two middle observations, is $(2+3)/2=2.5$. Agreement with the sample median is a feature of these data, rather than an identity between the two estimators.

The geometry explains both existence and possible nonuniqueness. The Huber loss is a convex function, so $Q_{k,s}$ is a convex function of $\theta$ and its derivative is $-s^{-1}{\sum }_i{\psi }_k((x_i-\theta )/s)$. For $n\ge 1$, $Q_{k,s}(\theta )$ tends to infinity as $|\theta |$ grows, hence attains a minimum; its zero derivative characterizes every minimizer. Between clipping transitions, the summed score has slope $-m/s$, where $m$ is the number of observations strictly within $ks$ of $\theta$. A root with $m>0$ is unique. If $m=0$, a flat interval is possible: for data $(0,10)$ with $s=1,k=2$, every $\theta \in [2,8]$ is a minimizer. A complete single-valued estimator then needs a selection rule, such as the interval's midpoint.

A bounded Huber score limits one observation's contribution to the estimating equation. The influence function of a Huber location estimator additionally divides by the population's central mass, so the size of its bound depends on the reference probability distribution. Clipping does not by itself make the bias of an estimator zero or prove a universal breakdown point; the separate local influence versus replacement breakdown argument specifies what fixed scale permits. Estimating $s$ from the same data changes the procedure and requires separate analysis.

Hold $(0,1,2,3,4)$ fixed and move only the sixth observation $M$. For $M\ge 5$, the sample mean is $(10+M)/6$, the sample median is $2.5$, and the Huber location estimator with fixed $s=1,k=2$ stays at $2.5$. Once $M-2.5\ge 2$, its Huber score is already $2$; moving it farther changes the Huber loss but leaves the root unchanged.

For a closer view of the transition, the exact Huber location estimator over $0\le M\le 40$ isTo verify the first branch, when $0\le M\le 2$ the score of $4$ is saturated and the other five scores sum with it to $8+M-5\theta$. When $2\le M\le 4.5$, the score of $0$ is saturated instead, giving the same expression. At $M=4.5$, the moving observation saturates too, and the equation becomes $10-4\theta =0$. The piecewise formula is continuous at the transition.

The rising sample mean comes from an unbounded linear score, the flat distant-outlier response from a bounded Huber score. This is a finite-data response curve with a specified scale and threshold, rather than a population influence function or a general claim about arbitrary numbers of outliers.

Suppose $X_1,\dots ,X_n$, $n\ge 1$, are independent and identically distributed with Laplace distribution $\mathrm{Laplace}(\mu ,b)$, where $b>0$. For realized values $x_1,\dots ,x_n$, the likelihood function and log-likelihood function areFor fixed $b$, maximum likelihood estimation of the location therefore minimizes the sum of absolute losses. This connects the Laplace distribution to robust statistics: each observation contributes a cost growing linearly with its distance, whereas squared-error loss grows quadratically.

Why does a sample median minimize $S$? Between successive order statistics, moving $a$ a little to the right increases each distance to a value on the left and decreases each distance to a value on the right. Hence the slope is the number on the left minus the number on the right. It changes from negative to positive at the middle. More precisely, if $x_{(1)}\le \cdots \le x_{(n)}$, all minimizing locations areFor an even sample size, the objective is flat throughout the interval between the two central order statistics. The midpoint convention for the sample median selects one solution; it does not prove uniqueness. ^[ref] This is the finite-sample counterpart of median minimizes expected absolute loss.

If $S(\hat{\mu })>0$, maximizing over $b>0$ givesThe derivative is positive below this value and negative above it. Moreover the maximized likelihood function is proportional to $S(\mu {)}^{-n}$ wherever $S(\mu )>0$, so the same minimizing locations give the joint maximum-likelihood estimates. ^[ref] The scale estimate divides by $n$; this formula is not an assertion that it is an unbiased estimator.

The boundary case matters: if all observations equal $c$, choose $\mu =c$. Then $S(c)=0$ and $L(c,b)=(2b{)}^{-n}\to \infty$ as $b\downarrow 0$. There is no finite joint maximum-likelihood estimate in the parameter space $b>0$. In particular, this always occurs for $n=1$. Writing $\hat{b}=0$ does not produce an admissible Laplace distribution.

Choose the five numbers $1,2,2,5,10$ as an original illustration. Their sample median is 2 and their sample mean is 4. The sum of absolute losses at the sample median isIf these numbers were modeled as an independent and identically distributed Laplace distribution sample, the joint maximum-likelihood estimates would therefore be $\hat{\mu }=2$ and $\hat{b}=12/5=2.4$. Using the sample mean instead gives $S(4)=14$, so it does not maximize this likelihood function.

Now replace only 10 by 100. The sample median stays at 2, while the sample mean becomes $(1+2+2+5+100)/5=22$. The minimized sum of absolute losses rises to $S(2)=102$, and the fitted scale rises to $\hat{b}=102/5=20.4$. Stability of the fitted location therefore does not mean stability of every fitted parameter.

The left panel subtracts each objective's minimum so the two absolute loss curves can be compared despite the different totals. The right panel tracks the sample mean and sample median as the largest chosen value changes. This is an outlier sensitivity illustration, not actual observations or a check that a Laplace distribution fits any population. In applications, regression residuals and the data-generating assumptions need separate examination; a stable sample median alone does not validate a statistical model.

For linear regression with fixed predictor vectors $X_i$, assume conditionally independent responses with a common scale $b>0$:Each conditional median and mean is $X_i^T\beta$. The log-likelihood function is $-n\log (2b)-S(\beta )/b$, so maximum likelihood estimation minimizes the sum of absolute regression residuals:This is least absolute deviations. With only an intercept, it reduces to the Laplace location and scale maximum likelihood problem, so a sample median is a fitted intercept. Provided the minimum $S(\hat{\beta })>0$, the scale estimate is $\hat{b}=S(\hat{\beta })/n$; the conditional standard deviation is $\sqrt{2}b$. If the scale is denoted $\sigma$ instead of $b$, the conditional standard deviation is $\sqrt{2}\sigma$.

Minimizing a sum of absolute losses does not automatically give a unique coefficient vector. Even an intercept-only fit to an even sample size can have an interval of solutions, and redundant predictor columns can introduce further nonuniqueness. If a fit makes every regression residual zero, the joint likelihood function is unbounded as $b\downarrow 0$, so there is no finite joint maximum-likelihood estimate with positive scale.

The linear growth of absolute loss limits sensitivity to an extreme response at fixed predictors compared with squared-error loss. It does not protect against every outlier: an extreme predictor can have large regression leverage. The numerical example above changes a response in an intercept-only problem, and makes no general claim of immunity to such predictor changes.

The influence function describes an infinitesimal change at a specified reference probability distribution. The replacement breakdown point asks how many observations in a particular finite sample can be replaced before an estimator becomes unbounded. The finite-sample maximum bias records the intervening changes. These are different questions: a bounded influence function at one baseline is not a general bound over all epsilon-contamination neighborhoods, nor a proof of a positive replacement breakdown point.^[ref]

The mean illustrates agreement between local and global fragility: its gross-error sensitivity is infinite and one replacement can send the sample mean arbitrarily far. The median has finite gross-error sensitivity under the stated positive-density assumptions, while its sample median withstands fewer than about half the observations being replaced under ordinary tie conventions. Its population contamination bias above still diverges as $\epsilon \uparrow 1/2$ when the atom remains beyond the shifted median. Here $x$ must also move outward; taking that limit with a fixed finite atom would invalidate the displayed quantile formula.

For scalar location with a fixed positive scale, the Huber score has a direct global argument: if strictly more than half of the sample stays fixed, then beyond every retained observation by more than $ks$, their saturated scores outweigh all possible replaced scores. Every root therefore stays in a bounded interval. This is a separate finite-replacement argument; estimating scale changes its premises. A smooth bounded influence function is valuable local protection, but global robustness also depends on the full estimating rule, scale choice, and root selection.

An order statistic is a value selected by its position in a sorted statistical sample. For real observations $X_1,\dots ,X_n$, writeThe $r$th order statistic $X_{(r)}$ is the $r$th smallest value, counting repeated values separately. The parentheses distinguish sorted position from the original observation label: $X_3$ and $X_{(3)}$ need not be equal. This definition permits ties and requires neither independence nor a continuous probability distribution. For a random statistical sample, each order statistic is itself a random variable and a statistic.

For example, the five fixed observationssort toTheir original indices in sorted order are $(2,5,4,1,3)$, so $x_{(3)}=x_4=0.4$. Their ranks of observations, listed in original observation order, are $(4,1,5,3,2)$. Here $x_{(1)}=0.1$ is the minimum, $x_{(5)}=0.9$ is the sample maximum, and $x_{(3)}=0.4$ is the sample median. These are deliberately chosen observations, rather than simulated draws.

Order statistics put extremes and typical values in one language: the first and last locate the minimum and sample maximum, while intermediate positions give sample quantiles and the sample median. Under the existing generalized-inverse convention, the sample quantile at level $p$ is $X_{(\lceil np\rceil )}$. For the five observations, the sample quantiles at $p=0.2,0.5,0.8$ are $0.1,0.4,0.7$. The distinction between a fixed observed value and its sampling distribution lets us quantify how much an extreme or sample median would vary across repeated statistical samples.

The useful counting identity isFor independent and identically distributed random variables with a continuous cumulative distribution function $F$, the count on the right has binomial distribution with parameters $n,F(t)$, givingThis cumulative distribution function is a binomial distribution tail: a question about sorting becomes a question about counting successes in Bernoulli trials. The counting proof also works for probability distributions with atoms and ties, although a probability density function need not then exist. For a uniform order statistic, $F(t)=t$ on $0\le t\le 1$, which makes the formula particularly explicit.