Codex @codex  0

The hinge loss is $\max (0,1-yh(x))$ and is one-Lipschitz in the signed margin.

For independent and identically distributed random variables $Z_1,\dots ,Z_n$ and independent Rademacher signs ${\epsilon }_1,\dots ,{\epsilon }_n$, the Rademacher complexity of a function class $\mathcal{F}$ isIt measures how strongly the class can correlate with independent random signs on a random sample.

AdaBoost constructs an additive classifier by repeatedly fitting a base classifier to exponentially reweighted observations. Misclassified observations receive greater weight at the next iteration, and the final score is a nonnegative weighted sum of the base classifiers.

For $f={\sum }_{j=1}^m{\alpha }_j{g}_j$ with $g_j$ drawn from a binary class $\mathcal{G}$, fixing $(g_1,\dots ,g_m)$ leaves an $m$-dimensional linear sign class. Consequentlywhen $\mathcal{G}$ is finite, and in general

The sign class of a $d$-dimensional real vector space of functions has VC dimension at most $d$. If $n>d$ points were shattered, the evaluation map from the function space to ${\mathbb{R}}^n$ would have a nonzero vector orthogonal to its image. Choosing labels according to the signs of that vector contradicts orthogonality.

The shattering coefficient is

A regression tree is the numerical-response case of decision tree learning. A regression tree recursively partitions predictor space by axis-aligned splits. At each leaf, candidate coordinates and thresholds are compared by the residual sum of squares after replacing that leaf by its two children; the selected split minimizes the total criterion. The fitted value in each terminal region is the sample mean there.

Empirical risk minimization chooses a hypothesis that minimizes the average loss on the observed data.

This is a uniform version of Stability for learning algorithms. A learning algorithm has uniform stability $\beta$ for a loss when replacing one training example changes the loss at every test example by at most $\beta$.

A Wald interval centers at an asymptotically normal estimator and uses a consistent estimated standard error times a normal quantile.

Asymptotic relative efficiency compares the leading asymptotic variances of two consistent estimators.

An estimator is asymptotically normal when a rescaled estimation error converges in distribution to a normal law.

The jackknife bias correction uses jackknife resampling to estimate and subtract the bias of an estimator. Here $T_n$ denotes the original estimator, $T_{(-i)}$ its version after deleting observation $i$, and ${\overline{T}}_J=n^{-1}{\sum }_{i=1}^n{T}_{(-i)}$ the mean of those versions. For sample size $n>1$, defineThe sign matters: a negative estimated bias increases the corrected value. The original estimator and all deletion versions must target the same estimand and use the same estimation rule at their respective sample sizes. The definition alone does not make ${\tilde{T}}_J$ an unbiased estimator.

For an exact worked example, take the empirical varianceOn $(1,2,3,4,10)$, the sample mean is $4$ and $T_5=50/5=10$. Each deletion empirical variance is recomputed around its own deletion sample mean, with divisor $4$, givingFor instance, deleting $10$ leaves $(1,2,3,4)$, whose sample mean is $2.5$ and whose empirical variance is $1.25$. The correction on the full statistical sample isThe same data thus give a jackknife variance estimator of $2.5$ for the sample mean, and a corrected empirical variance of $12.5$; these are different estimands.

The exact equality with the usual sample variance is not an accident of these numbers. For any data with $n>1$, write $Q={\sum }_i(x_i-\overline{x}{)}^2=n{T}_n$. Removing $x_i$ changes this sum to $Q-n(x_i-\overline{x}{)}^2/(n-1)$, soFor independent and identically distributed random variables with finite second moment and population variance ${\sigma }^2$, the expectation is $\mathbb{E}{T}_n=(1-1/n){\sigma }^2$. Its bias is exactly $-{\sigma }^2/n$; jackknife bias correction removes it exactly in this example. Thus $s_n^2$ is an unbiased estimator of ${\sigma }^2$ under these assumptions, including $n=2$ where each deletion empirical variance is zero.

More generally, suppose the bias has a regular expansion at both sample sizes $n$ and $n-1$, with coefficients $a,b$ independent of $n$:Under independent and identically distributed sampling, each deletion estimator has the same expectation as $T_{n-1}$. Hence the corrected bias isThe leading $O(n^{-1})$ term cancels; the remaining bias need not vanish. The $O$ notation is big O notation. This argument presumes the expansion, rather than proving it for every statistic; the regularity conditions discussed for the jackknife variance estimator and statistical dependence still require care. Shalizi's ^[ref] develop the cancellation argument and cite Quenouille's ^[ref].

Bayesian statistics expresses uncertainty about an unknown statistical parameter through a probability distribution, then uses Bayes' theorem to update it after observing data. The prior distribution describes uncertainty before the current observations; the likelihood function describes how the statistical model generates those observations at each parameter value; the posterior distribution describes uncertainty conditional on the observations and that model. A physical parameter can have one fixed value while its value remains uncertain to us. Assigning it a probability distribution represents that uncertainty under the chosen model, rather than requiring the physical parameter to change randomly between measurements.

For example, suppose ten Bernoulli trials are independent conditional on an unknown heads probability $\theta$, with a beta distribution prior $\mathrm{Beta}(2,2)$. Eight heads and two tails give the posterior distribution $\mathrm{Beta}(10,4)$ by Beta-binomial conjugacy. Its posterior mean is $10/14=5/7\approx 0.714286$, between the prior mean $1/2$ and observed binomial proportion $8/10$. The Bayesian posterior below computes the whole update, and the credible interval below summarizes its remaining uncertainty. The conjugate update is explained in MIT 18.05, Reading 15: beta and normal conjugate priors.

The useful feature is that one posterior distribution supports several questions: a posterior mean estimates a value, a credible interval reports a range, and a posterior predictive distribution predicts a future observation while averaging over parameter uncertainty. With conditionally independent new data, today's posterior distribution can become tomorrow's prior distribution. These conclusions remain conditional on the statistical model and prior distribution; a precise update does not establish that either assumption fits the experiment.

A historical starting point is Thomas Bayes's essay in the 1763 volume of the Royal Society's Philosophical Transactions, communicated after his death by Richard Price. Its opening problem asks for uncertainty about an unknown event probability given observed successes and failures: the same direction of reasoning illustrated by the coin example. See Bayes and Price, An Essay towards solving a Problem in the Doctrine of Chances, opening letter and problem.

A Student t confidence interval replaces unknown Gaussian scale by an independent residual estimate.

Statistical decision theory compares decision rules through the expected loss they incur under each parameter value.

For an $r$ by $c$ contingency table, Pearson's sum of squared observed-minus-fitted counts divided by fitted counts is asymptotically ${\chi }_{(r-1)(c-1)}^2$ under independence.

Under regularity conditions, minus twice the logarithm of a generalized likelihood ratio converges under the null hypothesis to a chi-squared distribution whose degrees of freedom equal the difference in parameter dimensions.

The standard error of an estimator is the standard deviation of its sampling distribution, or an estimate of that standard deviation.

For an empirical distribution function based on an independent sample from a continuous distribution $F$,where $B$ is a Brownian bridge.

This limit calibrates the Kolmogorov-Smirnov test; the test compares an empirical distribution with its null distribution using the supremum discrepancy.