M-estimator

An M-estimator chooses a parameter by minimizing an empirical objective:Here $\rho$ is a loss function. For scalar location, squared-error loss gives the sample mean, absolute-error loss gives a sample median, and Huber loss gives a Huber location estimator.

At an interior minimum of a differentiable objective, the estimating equation is ${\sum }_i\psi (x_i,\hat{\theta })=0$, where $\psi ={\partial }_{\theta }\rho$. A root need not be a global minimizer for a nonconvex objective: zero is a stationary point of $\rho (\theta )=({\theta }^2-1{)}^2$, but the minima are at $\theta =\pm 1$. For a differentiable convex function, a zero derivative is sufficient for global minimality.

Thus the loss-minimization and estimating-equation formulations require an explicit relation between their solutions. The sandwich variance of an M-estimator describes regular estimating-equation roots; choosing a robust loss does not itself establish the regularity or uniqueness needed for that calculation. Peter J. Huber's original location criterion is a central historical example.^[ref]

For an estimating equation with population root $\theta =T(F)$, let $A={\mathbb{E}}_F{\partial }_{\theta }\psi (X,\theta )$ be nonsingular and $B={\mathbb{E}}_F[\psi {\psi }^T]$. Differentiating the contaminated population equation gives influence function $-A^{-1}\psi (x,\theta )$. A consistent sample root with differentiable local expansion satisfies $\sqrt{n}(\hat{\theta }-\theta )=-A^{-1}{n}^{-1/2}{\sum }_i\psi (X_i,\theta )+o_p(1)$, so the central limit theorem gives the displayed covariance. In the scalar case this is $\mathbb{E}{\psi }^2/A^2$.

On a compact parameter set, a continuous deterministic objective with a unique minimum has a positive separation gap outside each neighbourhood of its minimizer. Uniform convergence in probability of the random objectives bounds the deterministic objective difference at any attained random minimizer by twice the supremum discrepancy. This proves statistical consistency of each measurable minimizing selection.

Uniform convergence in probability of empirical criteria on a compact set proves consistency for minimizers in that set; it does not confine a minimizer over a larger domain. A bounded identifiable regression function can approach its true value again at infinity. For median regression, $a(\theta )={\theta }^2/(1+{\theta }^4)$ has unique zero at zero, but a small positive fitted sample median $c$ has a second solution ${\theta }^2=(1+\sqrt{1-4c^2})/(2c)$. This solution diverges as $c\downarrow 0$, although uniform convergence on any fixed compact interval holds. Constraining the estimator, or proving suitable global separation and localization, repairs the argument.

A Scale M-estimator estimates the spread of centered data by solvingfor a chosen function $\psi$. If the center is estimated, residuals $x_i-\hat{\mu }$ replace $x_i$. The function and its normalization specify the population scale being estimated; an arbitrary choice need not have a positive or unique root.

For known center zero and at least one nonzero observation, choosing $\psi (u)=u^2-1$ gives $t=\sqrt{n^{-1}{\sum }_i{x}_i^2}$, the root mean square. This example is highly sensitive to an outlier. Bounded choices of $\psi$ can instead limit an observation's contribution to the estimating equation, but their influence function and breakdown point still require analysis.

If $\psi$ is an even function and the positive root is unique, multiplying the data by $c\ne 0$ multiplies the root by $|c|$: substitute $t_c=|c|t$ into the equation. This is scale equivariance, the expected response to changing measurement units.