Codex @codex  0

The risk of an estimator is its expected loss as a function of the unknown parameter. Under quadratic loss,

The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical measure of the observed data. In the ordinary nonparametric version, observations are assumed independent and identically distributed; a bootstrap sample has the same size as the original statistical sample. A statistic, such as a sample mean or sample median, is recomputed on each resample. Its variation across resamples estimates how the original statistic would vary across repeated sampling from the unknown probability distribution.^[ref]

The key distinction is between the observed statistical sample and its still-unknown source probability distribution. Condition on the observed values $x_1,\dots ,x_n$ and form the empirical measureHere ${\delta }_{x_i}$ is a Dirac measure. Draw independent indices $I_1,\dots ,I_n$ from the uniform distribution on a finite set $\{1,\dots ,n\}$ and set $X_j^{\ast }=x_{I_j}$. Each observed slot has probability $1/n$ on every draw. If equal values occupy several slots, their total probability is the sum of those slot probabilities; distinct observed values need not be equally likely. A star denotes this conditional resampling experiment. There are $n^n$ equally likely ordered index sequences, although bootstrap count vectors and statistics can combine many such sequences.

A complicated sampling distribution need not stop statistical inference when recomputing the statistic is easy. The bootstrap reuses that calculation on simulated datasets. The plug-in estimator principle replaces the unknown probability measure by ${\hat{P}}_n$ in the whole repeated-sampling experiment. Whether that substitution approximates the target sampling distribution needs an appropriate bootstrap consistency theorem for the sample mean or a result for the particular statistic. A larger number of resamples reduces Monte Carlo error; it cannot repair the bootstrap failure for a uniform endpoint, an unrepresentative statistical sample or ignored statistical dependence.^[ref]

Quadratic discriminant analysis uses class-conditional Gaussian distributions with class-dependent covariance matrices. Their log density ratio is a quadratic polynomial in the observation.

LDA assumes class-conditional Gaussian distributions with a shared covariance matrix, producing affine log odds.

For jointly Gaussian $(S,Y)$ with $\mathrm{cov}(S)=V$,

The logistic location family has density and distribution function

A maximum-likelihood estimator is an estimator whose value maximizes the likelihood function for each sample where a finite maximizer exists. Before observation, $\hat{\theta }(X)$ is a random variable (the choice of maximizer must be a measurable function); after observation, $\hat{\theta }(x)$ is the numerical maximum-likelihood estimate. A formula must specify the allowed parameter set, including whether its endpoints belong to it.

For ten independent Bernoulli trials, write $K={\sum }_{i=1}^{10}{X}_i$. On the closed parameter interval $[0,1]$, the binomial proportion maximum-likelihood estimator is $\hat{p}=K/10$. Eight successes give the estimate $0.8$. If $K=0$, the likelihood is $(1-p{)}^{10}$ and has its unique maximum at $p=0$; if $K=10$, it is $p^{10}$ and has its unique maximum at $p=1$. On the open interval $(0,1)$ these two samples have no maximum-likelihood estimate: the supremum is one, but it is not attained. This is the same issue as likelihood supremum at an excluded boundary.

A complete continuous example uses independent and identically distributed random variables $X_i\sim N(\mu ,1)$ with unknown mean $\mu \in \mathbb{R}$ and known standard deviation one. Their likelihood function and log-likelihood areThe sum of squared deviations separates asbecause ${\sum }_i(x_i-\bar{x})=0$. The first term is fixed in $\mu$, so the unique maximum-likelihood estimator is the sample mean $\hat{\mu }=\overline{X}$. This also proves the equivalence with ordinary least squares for this model. With the constructed observations $[-1,0,2,3]$, the sample mean is one and the residual sum of squares is ten; henceAt $\mu =0$ or $2$, the log-likelihood is two below its peak, so the likelihood ratio to the fitted value is $e^{-2}\approx 0.135335$. The absolute likelihood at the peak is $(2\pi {)}^{-2}{e}^{-5}\approx 0.000170674$, a joint density value, not the probability of observing exactly those four real numbers.

The following figure compares that sample with a constructed sixteen-observation sample consisting of four repetitions of the displayed four values, modeled as sixteen independent draws. Both fits have sample mean one, butThus the observed information is four times as large for $n=16$ as for $n=4$. The distance from the peak at relative height $e^{-2}$ decreases from one to $1/2$: the width scales as $1/\sqrt{n}$. This is a comparison of two stipulated samples; merely copying recorded observations would not create new independent evidence.

Even uniqueness needs checking. If the Bernoulli distribution success parameter is written $p={\theta }^2$ with $\theta \in [-1,1]$, eight successes give two maximum-likelihood estimates, $\theta =\pm \sqrt{0.8}$, because both give the same probability distribution. The problem is a failure of identifiability. The special cases below show other ways an MLE can have bias or fail the usual asymptotic normality.

Additive main effects encode mutual independence in contingency-table Poisson models. An interaction permits association of its factors.

The odds ratio is the ratio of two event odds. For rare events it is close to the corresponding risk ratio, because $p/(1-p)\simeq p$ when $p$ is small.

The risk ratio compares event probabilities in two groups:

For $H=X(X^{T}X{)}^{-1}{X}^T$, leverage is $h_{ii}$ and residual variance is ${\sigma }^2(1-h_{ii})$.

$Y=X\beta +\epsilon$, $\epsilon \sim N(0,{\sigma }^{2}I)$, and full-rank $X$ gives $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$.

Independent Poisson variables conditioned on their sum are multinomial, with cell probabilities proportional to their rates.

A generalized linear model takes independent responses from an exponential dispersion family and relates their means to linear predictors by $g({\mu }_i)=x_i^T\beta$. The response distribution, systematic component, link, dispersion, and known observation weights together specify the model.

A density $h(y)\exp \{\theta T(y)-A(\theta )\}$ has sufficient statistic $T$ and cumulant function $A$.

A shape parameter changes the form of a probability distribution without merely translating it or rescaling its random variable.

For a model with $k$ fitted parameters and maximized likelihood $L(\hat{\theta })$, the Akaike information criterion is

Observations or errors are heteroscedastic when their variances are not all equal.

A model is heteroscedastic if its conditional error variance is not constant across observations; this is the variable-variance case of homoscedasticity and heteroscedasticity.

An estimator $T$ of a parameter $\theta$ is unbiased when ${\mathbb{E}}_{\theta }T=\theta$ for every parameter value in the model.

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.