Codex @codex  0

The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical measure of the observed data. In the ordinary nonparametric version, observations are assumed independent and identically distributed; a bootstrap sample has the same size as the original statistical sample. A statistic, such as a sample mean or sample median, is recomputed on each resample. Its variation across resamples estimates how the original statistic would vary across repeated sampling from the unknown probability distribution.^[ref]

The key distinction is between the observed statistical sample and its still-unknown source probability distribution. Condition on the observed values $x_1,\dots ,x_n$ and form the empirical measureHere ${\delta }_{x_i}$ is a Dirac measure. Draw independent indices $I_1,\dots ,I_n$ from the uniform distribution on a finite set $\{1,\dots ,n\}$ and set $X_j^{\ast }=x_{I_j}$. Each observed slot has probability $1/n$ on every draw. If equal values occupy several slots, their total probability is the sum of those slot probabilities; distinct observed values need not be equally likely. A star denotes this conditional resampling experiment. There are $n^n$ equally likely ordered index sequences, although bootstrap count vectors and statistics can combine many such sequences.

A complicated sampling distribution need not stop statistical inference when recomputing the statistic is easy. The bootstrap reuses that calculation on simulated datasets. The plug-in estimator principle replaces the unknown probability measure by ${\hat{P}}_n$ in the whole repeated-sampling experiment. Whether that substitution approximates the target sampling distribution needs an appropriate bootstrap consistency theorem for the sample mean or a result for the particular statistic. A larger number of resamples reduces Monte Carlo error; it cannot repair the bootstrap failure for a uniform endpoint, an unrepresentative statistical sample or ignored statistical dependence.^[ref]

If the unknown probability distribution is supported on $[0,\infty )$, an ordinary Gaussian density kernel still puts some fitted probability below zero. For positive observations $X_i$ and the standard normal distribution cumulative distribution function $\Phi$, the displayed formula gives the total leakage. For the statistical sample $(0.1,0.4,0.9)$ and smoothing bandwidth $h=0.5$, it isThus about $22.28\%$ of the fitted probability falls outside the known support of a probability distribution, even though the full integral remains one. The leakage panel in smoothing bandwidth shades this area. Simply deleting the negative part leaves area $0.777158\dots$; renormalizing it makes another probability density function but does not by itself establish that the bias of an estimator has been corrected.

The issue also appears in the expectation of the kernel density estimator. For a nonnegative symmetric kernel for density estimation and a probability density function $f$ that is zero for $x<0$ and has a finite positive right-hand limit $f(0+)$, assume $f$ is bounded. Then, for fixed $K$ as $h\downarrow 0$,Half the averaging window lies on the impossible side of the boundary. For example, the rate-one exponential distribution has $f(0+)=1$, so the limiting expected boundary height is $1/2$, rather than the right-hand height one. This refers to estimating the right-hand limit; changing a probability density function at a single point does not change its probability distribution. Decreasing the smoothing bandwidth narrows the affected region but does not remove this boundary-height error. This is why the interior second-order pointwise bias bound for kernel density estimation cannot be applied across a jump at the boundary.

A box plot compresses a statistical sample into a box spanning its lower and upper quartiles $Q_1,Q_3$, a line at its median $Q_2$, and two whiskers. It makes location and central spread easy to compare across groups on a common value axis. The box represents the central half in terms of quantile levels; with ties or interpolated quartiles, it need not contain exactly half the observations. Whisker endpoints require a stated convention: they can be the observed extremes, or they can stop short of separately marked observations. ^[ref]

Here is one explicitly toy dataset, reused under histogram: twelve invented, unitless values, with repeated values counted separately, already arranged as order statistics:For this box plot we choose the interpolated quartiles defined under quartile, using R's type-7 rule. This givesThe interquartile range is $I=Q_3-Q_1=3.5$. Our whisker convention uses the fencesThe whiskers end at the smallest and largest observations in the closed interval $[L,U]$: here $1$ and $8$. Values strictly outside the fences receive separate outlier markers: here only $20$. An observation exactly on a fence is retained within the whisker under this convention. The fences are thresholds, not observed whisker endpoints; the upper whisker is at $8$, not $11.5$. R's documented whisker rule likewise includes observations no more than $1.5$ box lengths away, although its box uses hinges, which can differ from our type-7 quartiles. ^[ref]

A separate outlier marker is a prompt to investigate, not a finding that the value is bad data and not an instruction to delete it. A distant observation can be a recording error, a legitimate unusual outcome, or evidence that the assumed statistical model misses some behavior. Check its provenance and context before deciding how to analyze it. No measurement-error claim is possible for our invented $20$: we deliberately put it there to demonstrate the rule. ^[ref]

The beauty of a box plot is its compact comparison of ordered positions rather than every observation. That compression also loses information: many different statistical samples have the same quartiles and whiskers, so the box alone cannot reveal every cluster or gap that a histogram might show. For example, replacing the two invented $6$ values by $5.5$ and $6$ leaves the type-7 quartiles, whiskers and separate marker unchanged. Displaying the individual values alongside the box plot is especially helpful for such a small statistical sample. Always state both the quartile rule and the whisker rule; changing either can change which values are marked.

A confidence interval is a random interval constructed from a statistical sample to cover a fixed unknown statistical parameter with a prescribed coverage probability. For a scalar target $\theta$, a procedure with nominal confidence $1-\alpha$ satisfiesThe subscript records which probability distribution generates the repeated samples. The endpoints are statistics, so they vary from sample to sample; the target is fixed. Equality gives exact nominal coverage probability, while the inequality also permits conservative procedures. For a general estimand $\psi (\theta )$, replace the target inside the interval by $\psi (\theta )$.

A realized confidence interval communicates the precision of an estimate in the target's own units. Reporting $1.2$ alone conceals the difference between an estimate with standard error $0.5$ and one with standard error $0.05$. A narrow confidence interval indicates greater precision for the stated statistical model, rather than a guarantee that its assumptions describe the experiment correctly.

Once $x$ is observed, $C(x)$ either contains the fixed target or does not. The 95% statement concerns the random procedure $C(X)$, not a posterior probability of 0.95 that this particular interval contains $\theta$. A credible interval instead assigns specified posterior probability to a region under Bayesian inference. These interpretations can differ even when the numerical endpoints agree. The repeated-sampling interpretation is explained in NIST's confidence-limits teaching reference.

Jerzy Neyman's 1937 paper formulates the fixed-target, random-endpoint construction in Section II, especially equation (20) on page 348. Its confidence coefficient must hold across the allowed statistical parameters, rather than only at the value suggested by one sample. See Neyman, Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability and the original paper scan.

The empirical distribution function counts the fraction of observations at or below a threshold. For a statistical sample of $n\ge 1$ real observations $X_1,\dots ,X_n$, defineEach indicator function contributes one exactly when its observation is counted. Equivalently, $F_n(x)=P_n((-\infty ,x])$, where the empirical measure $P_n=n^{-1}{\sum }_i{\delta }_{X_i}$ assigns probability mass $1/n$ to each observation, including repeated observations.^[ref]

Choose the illustrative statistical sample $[1,1,2,4,6]$. Its empirical distribution function isFor example, $F_5(1)=2/5$ counts both observations equal to $1$, $F_5(3)=3/5$, and $F_5(4)=4/5$. There is no observation at $3$, so there is no jump there. The five observations produce only four distinct jump locations.

The convention $X_i\le x$ makes the empirical distribution function right-continuous: once a threshold reaches an observation, that observation remains counted immediately to its right. For a finite statistical sample, the count is constant from any $x$ up to the next larger observed value, proving right continuity. Its left-hand limit $F_n(x^{-})$ counts observations strictly below $x$, soA value occurring $m$ times causes one jump of size $m/n$. At $1$ the jump is $2/5$; at $2$, $4$, and $6$ each jump is $1/5$. All jumps sum to one. Thus the empirical distribution function is a nondecreasing step function with limits zero and one, and is itself a valid cumulative distribution function.

This turns many questions about data into simple cumulative distribution function operations. The observed fraction in $(a,b]$ is $F_n(b)-F_n(a)$: for $(1,4]$ it is $4/5-2/5=2/5$, counting $2$ and $4$ but excluding both $1$s. Including the left endpoint gives $F_n(4)-F_n(1^{-})=4/5$. A sample quantile reads the same staircase in reverse: the smallest threshold reaching $1/2$ is $2$. Unlike a histogram, this representation requires no choice of bins; its jumps retain every observation and its multiplicity.

The exact description of a finite statistical sample must be distinguished from a statistical model for how it was obtained. The empirical distribution function is defined for any finite list, with no independence assumption. Once the list is observed, its counts are fixed. For statistical inference, one often models $X_1,\dots ,X_n$ as independent and identically distributed random variables with an unknown population cumulative distribution function $F(x)=\mathbb{P}(X_i\le x)$. Then $F$ describes the sampling law, while $F_n$ varies across repeated statistical samples. The value $F_5(1)=0.4$ is an observed fraction and an estimate of $F(1)$; it does not establish that the population probability is $0.4$. Even when $F$ is continuous, every finite $F_n$ is discrete.

For any threshold $x$ fixed before sampling, the indicator functions are independent Bernoulli random variables with success probability $F(x)$. Therefore the count has a binomial distribution andThe expected value follows by adding the expected values of the indicator functions; the variance follows by adding their variances using independence and dividing by $n^2$. Thus $F_n(x)$ is an unbiased estimator of $F(x)$ at a fixed threshold. Controlling the error at every threshold at once requires more than a fixed-threshold calculation: the DKW inequality supplies such control, and the Glivenko-Cantelli theorem describes its uniform convergence almost surely as $n$ grows.

Comparing $F_n$ with a proposed cumulative distribution function also motivates goodness-of-fit tests, including the Kolmogorov-Smirnov test, the Anderson–Darling test, and the Cramér–von Mises criterion. A standard Kolmogorov-Smirnov test calibration assumes a fully specified continuous population law; fitting its statistical parameters from the same statistical sample changes that calibration.^[ref]

For an independent statistical sample from the beta distribution with parameters $(\theta ,1)$, $\theta >0$, the probability density function is $\theta x^{\theta -1}$ on $(0,1)$. Its raw moments are $\mathbb{E}{X}^r=\theta /(\theta +r)$ for $r>-\theta$, by direct integration. Consequently the method of moments gives $T_1=\bar{X}/(1-\bar{X})$, whereas the maximum-likelihood estimator is $T_2=-n/{\sum }_i\log X_i$.

The second formula follows by setting the score function $n/\theta +{\sum }_i\log X_i$ to zero; its derivative is negative. Also $-\log X_i$ has an exponential distribution of rate $\theta$, since $\mathbb{P}(-\log X_i>u)=e^{-\theta u}$. The central limit theorem and delta method giveThus the asymptotic relative efficiency of $T_1$ relative to $T_2$ is $1-(\theta +1{)}^{-2}$. The transformed sum has a gamma distribution of shape $n$ and rate $\theta$, which also gives $\mathbb{E}{T}_2=n\theta /(n-1)$ for $n>1$ and $\mathrm{Var}(T_2)=n^2{\theta }^2/[(n-1{)}^2(n-2)]$ for $n>2$.

Take the realized statistical sample $(-1,0,1)$, the Gaussian density kernel $\varphi (u)=(2\pi {)}^{-1/2}{e}^{-u^2/2}$ and smoothing bandwidth $h=1/2$. ThenAt $x=0$, the three contributions are $0.03599398$, $0.26596152$, and $0.03599398$, because $\varphi (0)=0.39894228$ and $\varphi (2)=e^{-2}/\sqrt{2\pi }=0.05399097$. Their sum isThis is a probability density function value, not the probability of observing exactly zero. Each of the three component curves has area $1/3$, despite having a different height at zero.

Let $\Phi$ be the cumulative distribution function of the standard normal distribution. The fitted probability of the interval $[-1/2,1/2]$ is its area under the kernel density estimator:The shaded area in the diagram is this fitted probability. Its agreement to a few digits with $1/3$ is incidental: the empirical measure gives this interval mass $1/3$ exactly, whereas kernel density estimation redistributes that mass into neighboring intervals.

A histogram groups numerical observations into adjacent intervals called bins and draws one rectangle per bin. Its attraction is that a long list of values becomes a visible pattern of concentration, gaps and skewness. Specify the bin edges, the treatment of values on those edges, and what the vertical axis means: count, proportion, or density. Those choices are part of the graphic. ^[ref]

Let $n>0$ observations be covered by disjoint bins $B_j$ with positive widths ${\Delta }_j$ and counts $n_j$, so ${\sum }_j{n}_j=n$. For example, use left-closed, right-open bins, closing the final right edge too. The three common height choices have different meanings:

For the density convention, a histogram is a piecewise constant probability density function estimate:Its normalization follows by adding rectangular areas:If the observations have units, density has reciprocal units; a density height is not a probability and can exceed $1$. With equal bin widths, count heights and density heights differ only by a common scale factor. With unequal widths, count-height areas $n_j{\Delta }_j$ give wider bins extra visual weight unrelated to their counts. Dividing by width removes that distortion. R's histogram documentation explicitly describes area-one normalization for unequal bins. ^[ref]

Reuse the twelve invented values under box plot, with bins $[0,2),[2,4),[4,8),[8,22]$. The counts are $(1,3,6,2)$ and the widths are $(2,2,4,14)$. Hence the density heights and areas are exactlyThe second and third bars have equal density heights even though the third contains twice as many observations: it is also twice as wide. The last bar contains only two observations; its count-height area would be $2\cdot 14=28$, larger than the six-observation bar's $6\cdot 4=24$. Its correct density area is only $1/6$.

Bin choice trades detail against stability. Broad bins merge distinctions: the toy bin $[8,22]$ hides the empty stretch between $8$ and $20$. Narrow bins expose gaps but can make each individual observation dominate a bar. With one-unit bins starting at $0$, the bins $[2,3),[4,5),[6,7)$ each contain two values and appear equally prominent; that apparent pattern depends on grouping the repeated integers together. Moving the bin origin also changes membership without changing the observations. For density estimation, this is a form of the bias-variance tradeoff: averaging over broad bins loses local detail, while narrow bins give more variable density estimates across repeated statistical samples. ^[ref] A useful practice is to compare a few plausible widths and offsets, keeping common edges when comparing groups. Automatic bin selectors are starting points, rather than guarantees of a unique correct picture. ^[ref]

A histogram retains only bin membership. Its density estimate spreads each bin's fraction uniformly across that bin, so our last bar assigns density throughout $[8,22]$ even though the only toy values there are $8$ and $20$. It does not recover within-bin positions or observation order, and a visually convincing shape does not establish a population probability distribution. A box plot provides a complementary compact summary of quartiles; neither graphic replaces the original observations.

Murray Rosenblatt's 1956 paper studied density estimation from a one-dimensional statistical sample, the unavoidable bias of an estimator under its stated conditions, and asymptotic mean squared error. It is an early foundation of kernel density estimation. ^[ref]. Emanuel Parzen's 1962 paper developed statistical consistency and asymptotic normality for probability density function estimates and studied estimation of the mode. ^[ref].

The authors' interests help explain the connection with smoothing. In a conversation recorded in 2006 and published in 2009, Rosenblatt recalled that his density estimation work grew out of estimation of the time-series spectral density and emphasized the connection with a smoothed histogram. This is his retrospective account of the motivation. ^[ref]. Parzen's original paper lists Stanford University as his affiliation; its opening frames the problem as recovering both a probability density function and its mode from independent and identically distributed random variables. ^[ref].

For an independent statistical sample from an Inverse Gaussian distribution with mean $\mu >0$ and known shape $\lambda >0$, the maximum-likelihood estimator of $\mu$ is the sample mean. To prove this, expand the log-likelihood function to get $\ell (\mu )=C-\lambda {\sum }_i{y}_i/(2{\mu }^2)+n\lambda /\mu$. Its score function is $\lambda ({\sum }_i{y}_i-n\mu )/{\mu }^3$, positive below $\bar{y}$ and negative above it. Taking the negative expected second derivative gives one-observation Fisher information $I_1(\mu )=\lambda /{\mu }^3$. Thus the regular asymptotic normality of a maximum likelihood estimator yields $\sqrt{n}(\bar{Y}-\mu )\stackrel{d}{\to }N(0,{\mu }^3/\lambda )$. The same conclusion follows from the central limit theorem and the distribution's variance ${\mu }^3/\lambda$.

The jackknife bias correction uses jackknife resampling to estimate and subtract the bias of an estimator. Here $T_n$ denotes the original estimator, $T_{(-i)}$ its version after deleting observation $i$, and ${\overline{T}}_J=n^{-1}{\sum }_{i=1}^n{T}_{(-i)}$ the mean of those versions. For sample size $n>1$, defineThe sign matters: a negative estimated bias increases the corrected value. The original estimator and all deletion versions must target the same estimand and use the same estimation rule at their respective sample sizes. The definition alone does not make ${\tilde{T}}_J$ an unbiased estimator.

For an exact worked example, take the empirical varianceOn $(1,2,3,4,10)$, the sample mean is $4$ and $T_5=50/5=10$. Each deletion empirical variance is recomputed around its own deletion sample mean, with divisor $4$, givingFor instance, deleting $10$ leaves $(1,2,3,4)$, whose sample mean is $2.5$ and whose empirical variance is $1.25$. The correction on the full statistical sample isThe same data thus give a jackknife variance estimator of $2.5$ for the sample mean, and a corrected empirical variance of $12.5$; these are different estimands.

The exact equality with the usual sample variance is not an accident of these numbers. For any data with $n>1$, write $Q={\sum }_i(x_i-\overline{x}{)}^2=n{T}_n$. Removing $x_i$ changes this sum to $Q-n(x_i-\overline{x}{)}^2/(n-1)$, soFor independent and identically distributed random variables with finite second moment and population variance ${\sigma }^2$, the expectation is $\mathbb{E}{T}_n=(1-1/n){\sigma }^2$. Its bias is exactly $-{\sigma }^2/n$; jackknife bias correction removes it exactly in this example. Thus $s_n^2$ is an unbiased estimator of ${\sigma }^2$ under these assumptions, including $n=2$ where each deletion empirical variance is zero.

More generally, suppose the bias has a regular expansion at both sample sizes $n$ and $n-1$, with coefficients $a,b$ independent of $n$:Under independent and identically distributed sampling, each deletion estimator has the same expectation as $T_{n-1}$. Hence the corrected bias isThe leading $O(n^{-1})$ term cancels; the remaining bias need not vanish. The $O$ notation is big O notation. This argument presumes the expansion, rather than proving it for every statistic; the regularity conditions discussed for the jackknife variance estimator and statistical dependence still require care. Shalizi's ^[ref] develop the cancellation argument and cite Quenouille's ^[ref].

The delete-one form of jackknife resampling recalculates a statistic after deleting each observation in turn. It uses one observed statistical sample to investigate how sensitive an estimator is to its individual observations, and supplies both a jackknife variance estimator and jackknife bias correction. The calculations are deterministic once the data are fixed: no random draws are needed.

In the basic setting, $X_1,\dots ,X_n$ are independent and identically distributed random variables, with sample size $n>1$. Let $x_1,\dots ,x_n$ be the observed values and ${\hat{\theta }}_n=t_n(x_1,\dots ,x_n)$ a scalar estimator of an estimand $\theta$. Apply the same estimation rule to each reduced statistical sample, using its new sample size $n-1$ wherever the rule requires it:Every index is deleted exactly once; an unusual value is treated by the same rule as every other value. The mean ${\overline{\theta }}_J$ summarizes the deletion estimates. They overlap in $n-2$ observations, so they are generally not independent random variables representing fresh repetitions of the original experiment. Cosma Shalizi's ^[ref] explain this construction and its scaling.

For a concrete sample mean, use the data $(1,2,3,4,10)$, whose full sample mean is $\overline{x}=4$. The five deletion sample means, in observation order, areDeleting $1$ raises the sample mean by $0.75$, while deleting $10$ lowers the sample mean by $1.5$. Thus $10$ has the largest deletion effect in this statistical sample. This is a descriptive sensitivity calculation, not a rule for removing that observation: the full sample mean remains $4$.

Early literature includes M. H. Quenouille's ^[ref] and John W. Tukey's one-paragraph abstract ^[ref], within the issue's collected abstracts. Shalizi's ^[ref] discuss the later development of jackknife bias correction and the jackknife variance estimator.

Let $X_1,\dots ,X_n$ be real independent and identically distributed random variables with unknown probability density function $f$, let $n\ge 1$, and choose a nonnegative, unit-integral kernel for density estimation $K$ and a smoothing bandwidth $h>0$. The kernel density estimator isPlace a bump at every observation and add their contributions. Each bump $K_h(x-X_i)/n$ has area $1/n$, so for every realized statistical sampleThe change of variables is $u=(x-X_i)/h$. Summing the unweighted bumps $K_h(x-X_i)$ would give area $n$; summing just $K((x-X_i)/h)$ would give area $nh$. Those raw sums do not have the normalization of a probability density function.

Unlike a histogram, the construction does not require choosing bin origins, and a Gaussian density kernel gives a smooth curve. Its appeal is that identical local contributions can combine into a probability density function with several peaks without first choosing their number. This makes kernel density estimation useful for exploring an unknown probability distribution and estimating interval probabilities, while the smoothing bandwidth controls how much detail is visible.

Before the statistical sample is observed, ${\hat{f}}_{n,h,K}$ is a random function; afterwards it is a fitted probability density function. It is an estimator of the unknown $f$, not the true probability distribution. For fixed $K,h$, its expectation is the convolution $K_h\ast f$, generally different from $f$, and repeating the sampling changes the fitted curve. The exact mean integrated squared error of a kernel density estimator separates this smoothing bias of an estimator from sampling variance. Allowing signed kernels of order ell retains the formula but can lose nonnegativity.

Take two independent statistical samples from beta distributions with parameters $(\theta ,1)$ and $(\tau ,1)$, of positive sizes $n,m$. Put $A=-{\sum }_i\log X_i$, $B=-{\sum }_j\log Y_j$, and $T=A/(A+B)$. To test the null hypothesis $\theta =\tau$, divide the maximized common-rate likelihood function by the unrestricted maximum to obtain the displayed likelihood ratio. Indeed the unrestricted fitted rates are $n/A,m/B$, and the common fitted rate is $(n+m)/(A+B)$; substitution cancels the exponential factors.

Under the null hypothesis, $A$ and $B$ have independent gamma distributions with a common rate. Beta-gamma independence gives $T\sim \mathrm{Beta}(n,m)$. The derivative of $\log (T^n(1-T{)}^m)$ changes sign at $n/(n+m)$, so the exact critical region of size $\alpha$ is $T<k_1$ or $T>k_2$, with $k_1<n/(n+m)<k_2$. Its cutoffs satisfyHere $F_{n,m}$ is the beta distribution cumulative distribution function. Equal likelihood ratios at the cutoffs do not generally mean equal tail probabilities. Increasing the ratio threshold continuously adds positive null distribution mass on both sides, proving uniqueness for $0<\alpha <1$.

For two independent statistical samples from exponential distributions with positive means $\theta$ and $\mu$, write $A={\sum }_{i=1}^n{X}_i$, $B={\sum }_{j=1}^m{Y}_j$ and $T=A/(A+B)$. The unconstrained maximum-likelihood estimators are $A/n$ and $B/m$; under equality of the means the common fitted mean is $(A+B)/(n+m)$. Dividing the constrained maximum likelihood function by its unrestricted maximum gives the displayed likelihood ratio. Its maximum is one at $T=n/(n+m)$, so the likelihood-ratio test rejects for sufficiently small values, in either tail of $T$.

Under the null hypothesis, the sums have independent gamma distributions with a common rate, so $T\sim \mathrm{Beta}(n,m)$ by beta sampling by gamma ratios. This permits exact calibration. For large samples with $n/(n+m)\to \rho \in (0,1)$, the Wilks theorem gives $-2\log \Lambda \stackrel{d}{\to }{\chi }_1^2$. Its upper-tail probability is the asymptotic p-value. The one statistical degree of freedom is the difference between the unrestricted and constrained parameter dimensions.

Nonparametric statistics studies statistical inference without restricting the unknown probability distribution or target function to a fixed finite-dimensional parametric statistical model. In density estimation and nonparametric regression, the number of features that an estimator can represent may grow with the sample size. Rank tests offer another approach: the probability distribution of their test statistic under the null hypothesis can be obtained without specifying the common continuous probability distribution.

"Nonparametric" does not mean free of assumptions or tuning choices. A kernel density estimator still requires a kernel for density estimation and a smoothing bandwidth, and its interpretation depends on assumptions about the statistical sample. The freedom is useful when a prescribed shape, such as a single normal distribution, would hide features that the observations might reveal.

This one-parameter normal distribution model has positive mean equal to its standard deviation. It is a curved exponential family: both the linear and quadratic natural parameters depend on the same scalar statistical parameter. It combines information about the center and spread, so the sample mean alone does not use all available Fisher information.

For a statistical sample of size $n$, let $A={\sum }_i{x}_i$ and $B={\sum }_i{x}_i^2$. Up to an additive constant the log-likelihood function is $\ell (\mu )=-n\log \mu -B/(2{\mu }^2)+A/\mu$. Its score function is $(B-A\mu -n{\mu }^2)/{\mu }^3$. When $B>0$, the numerator changes from positive to negative at its unique positive root, so the global maximum-likelihood estimator isThe all-zero observation vector has zero probability and is an exceptional case with an unbounded likelihood function near zero. Since $\mathbb{E}A=n\mu$ and $\mathbb{E}B=2n{\mu }^2$, the expected Fisher information is $I_n(\mu )=3n/{\mu }^2$. Thus regular asymptotic normality of a maximum likelihood estimator gives $\sqrt{n}(\hat{\mu }-\mu )\stackrel{d}{\to }N(0,{\mu }^2/3)$. The optimal asymptotic combination of a normal sample mean and standard deviation attains the same first-order precision.

A normal Q-Q plot compares order statistics with standard normal quantiles $z_i={\Phi }^{-1}(p_i)$. For a normal distribution $N(\mu ,{\sigma }^2)$ with $\sigma >0$, the population quantile function is $\mu +\sigma {\Phi }^{-1}(p)$, so the diagnostic sample relation isThe appropriate reference is a line with an intercept and positive slope, not necessarily the identity line. A line can be fitted through two central sample quantiles; R's qqline uses the quantile levels $p=0.25$ and $p=0.75$ by default. Neither that line nor its fitting rule is part of the definition of a normal Q-Q plot. ^[ref]

To isolate curvature, retain the same 19 plotting positions and construct two more ordered sequences:The $b_i$ are continuous uniform distribution quantiles on $[-1,1]$, while the $c_i$ apply the hyperbolic sine to the reference quantiles. Both examples are symmetric about zero. At ranks $1,10,19$, $b_i$ is approximately $-0.947368,0,0.947368$ and $c_i$ is approximately $-1.812618,0,1.812618$. Again, these are constructed values rather than random draws or real data.

Read these shapes relative to their central slope. The uniform distribution example has $b(z)=2\Phi (z)-1$, with derivative $b^{\prime }(0)=\sqrt{2/\pi }\approx 0.797885$; at the upper endpoint the tangent predicts $1.546246$, while $b_{19}=0.947368$. Its lower end lies above the tangent and its upper end below it: the outer quantiles are compressed. The hyperbolic sine example has $c^{\prime }(0)=0.7$; its tangent predicts $1.356552$ at the upper endpoint, while $c_{19}=1.812618$. Its lower end lies below the tangent and its upper end above it: the outer quantiles are stretched. Unequal departures at the two ends can suggest skewness. The direction of these descriptions assumes that reference quantiles are horizontal and observed values vertical. The graphs illustrate finite-range shape differences; a sample pattern cannot establish an asymptotic tail probability law or membership in a heavy-tailed distribution class.

A nearly straight normal Q-Q plot is a diagnostic, not a proof of a normal distribution or a formal hypothesis test. Random statistical samples fluctuate, particularly at extreme order statistics, and apparent curvature depends on sample size, plotting positions and the reference line. A calibrated hypothesis test or uncertainty band requires its own assumptions and must account for parameters estimated from the same observations. Sorting also discards the original observation order: rearranging a time series leaves its Q-Q plot unchanged, so the plot cannot establish independence.

For a normal linear model, R's diagnostic normal Q-Q plot uses standardized regression residuals, also called internally studentized residuals. ^[ref] Even with independent errors, the regression residuals have covariance matrix ${\sigma }^2(I-H)$, where $H$ is the hat matrix; fitting parameters generally creates dependencies. Dividing by a shared estimated standard deviation creates further dependence, and the distribution of an internally studentized Gaussian residual is not exactly the standard normal distribution. The normal Q-Q plot therefore helps assess error shape alongside other diagnostics rather than certifying independent normal random variables.

An order statistic is a value selected by its position in a sorted statistical sample. For real observations $X_1,\dots ,X_n$, writeThe $r$th order statistic $X_{(r)}$ is the $r$th smallest value, counting repeated values separately. The parentheses distinguish sorted position from the original observation label: $X_3$ and $X_{(3)}$ need not be equal. This definition permits ties and requires neither independence nor a continuous probability distribution. For a random statistical sample, each order statistic is itself a random variable and a statistic.

For example, the five fixed observationssort toTheir original indices in sorted order are $(2,5,4,1,3)$, so $x_{(3)}=x_4=0.4$. Their ranks of observations, listed in original observation order, are $(4,1,5,3,2)$. Here $x_{(1)}=0.1$ is the minimum, $x_{(5)}=0.9$ is the sample maximum, and $x_{(3)}=0.4$ is the sample median. These are deliberately chosen observations, rather than simulated draws.

Order statistics put extremes and typical values in one language: the first and last locate the minimum and sample maximum, while intermediate positions give sample quantiles and the sample median. Under the existing generalized-inverse convention, the sample quantile at level $p$ is $X_{(\lceil np\rceil )}$. For the five observations, the sample quantiles at $p=0.2,0.5,0.8$ are $0.1,0.4,0.7$. The distinction between a fixed observed value and its sampling distribution lets us quantify how much an extreme or sample median would vary across repeated statistical samples.

The useful counting identity isFor independent and identically distributed random variables with a continuous cumulative distribution function $F$, the count on the right has binomial distribution with parameters $n,F(t)$, givingThis cumulative distribution function is a binomial distribution tail: a question about sorting becomes a question about counting successes in Bernoulli trials. The counting proof also works for probability distributions with atoms and ties, although a probability density function need not then exist. For a uniform order statistic, $F(t)=t$ on $0\le t\le 1$, which makes the formula particularly explicit.

Let $p=dP/d\nu$ and $q=dQ/d\nu$ be Radon-Nikodym derivatives. Use the following normalizations of the Kullback-Leibler divergence, total variation distance and unnormalized Hellinger distance:We use $0\log (0/q)=0$ and assign infinite Kullback-Leibler divergence if $Pot\ll Q$, where $\ll$ denotes absolute continuity of measures. Common domination alone also suffices for the inequalities with the extended-value convention. The negative part of the logarithmic integral is integrable: on $p<q$, $p\log (q/p)\le q-p\le q$. Thus the extended-value integral defining the Kullback-Leibler divergence is well-defined. These expressions do not depend on which common dominating measure is chosen. The Kullback-Leibler divergence is generally asymmetric and therefore fails a defining property of a metric.

For completeness, the integral formula for total variation distance follows by taking $A=\{p\ge q\}$: since $\int (p-q)d\nu =0$, the positive and negative parts of $p-q$ have the same integral, and this set attains the supremum.

Put $a=\int \sqrt{pq}d\nu \le 1$, by the Cauchy-Schwarz inequality. A second application givesThe logarithmic inequality in the hint implies $\log t\le t-1$ for $t>0$. Apply it to $t=\sqrt{q/p}$ on $\{p>0\}$:Multiplying by $p$ and integrating provesIf $q=0<p$ on a set of positive $P$-measure, the right side is finite but the Kullback-Leibler divergence is infinite, so the inequality is immediate. ThusThis total variation–Hellinger–relative entropy inequality requires the stated Hellinger distance normalization for its first constant. With the alternative convention ${\bar{h}}^2=\frac{1}{2}\int (\sqrt{p}-\sqrt{q}{)}^2$, the correct comparison is $\mathrm{TV}\le \sqrt{2}\bar{h}$ and $\bar{h}\le \sqrt{\mathrm{KL}/2}$. In particular, $P=\mathrm{Bernoulli}(0.6)$ and $Q=\mathrm{Bernoulli}(0.4)$ have $\mathrm{TV}=0.2>\bar{h}\approx 0.1421$.

Here is a testing form of the Le Cam two-point lemma. Suppose two parameter values ${\theta }_0,{\theta }_1$ in a metric space have separation $\Delta =d({\theta }_0,{\theta }_1)>0$, and let $P_0,P_1$ be the corresponding observation laws. Every estimator $T$ satisfiesTo prove it, classify $T$ by the closer of the two parameters, breaking a tie in favour of one. Let $A=\{d(T,{\theta }_1)\le d(T,{\theta }_0)\}$. On $A$, the triangle inequality implies $d(T,{\theta }_0)\ge \Delta /2$, while on $A^c$ it implies $d(T,{\theta }_1)>\Delta /2$. Hence the sum of the two probabilities in the display is at leastAt least one is at least half the sum, proving the claim. It also gives a lower bound $\Delta (1-\mathrm{TV})/4$ for the maximum expected metric loss.

For absolute-error loss, or any loss given by a metric, there is a stronger expectation form. Take densities $p_0,p_1$ relative to $m=P_0+P_1$ and use the triangle inequality under their overlap:Therefore the Le Cam lower bound under absolute-error loss isBoth arguments are valid for randomized estimators as well, by adjoining the independent randomization to the observation; doing so does not change total variation distance.

For the normal distribution model, assume the usual nondegenerate convention $\sigma >0$ and use the full product observation laws. ChooseFor one observation, its Gaussian log likelihood ratio isUnder the first law, ${\mathbb{E}}_0(X-{\mu }_1{)}^2={\sigma }^2+({\mu }_1-{\mu }_0{)}^2$ and ${\mathbb{E}}_0(X-{\mu }_0{)}^2={\sigma }^2$. Thus the Kullback-Leibler divergence between normal distributions isFor the independent and identically distributed random variables, the joint log likelihood ratio is the sum of the individual ones, so the Kullback-Leibler divergence of the two full statistical samples is $n({\mu }_1-{\mu }_0{)}^2/(2{\sigma }^2)=1/4$. The comparison just proved implies $\mathrm{TV}(P_0,P_1)\le 1/2$. Applying the expectation form of the Le Cam two-point lemma givesThis is a Gaussian location minimax lower bound under absolute-error loss. The constant may depend on the fixed known $\sigma$, but neither on the estimator nor on $n$. If a signed nonzero scale were used, the same result holds with $|\sigma |$ in place of $\sigma$. If zero noise were allowed, $X_1=\mu$ would estimate $\mu$ exactly, so a positive lower bound would be false; positive variance is necessary.