M-estimator

An M-estimator minimizes an empirical objective ${\sum }_i\rho (x_i,\theta )$ or, when differentiable, solves an estimating equation ${\sum }_i\psi (x_i,\theta )=0$.

For an estimating equation with population root $\theta =T(F)$, let $A={\mathbb{E}}_F{\partial }_{\theta }\psi (X,\theta )$ be nonsingular and $B={\mathbb{E}}_F[\psi {\psi }^T]$. Differentiating the contaminated population equation gives influence function $-A^{-1}\psi (x,\theta )$. A consistent sample root with differentiable local expansion satisfies $\sqrt{n}(\hat{\theta }-\theta )=-A^{-1}{n}^{-1/2}{\sum }_i\psi (X_i,\theta )+o_p(1)$, so the central limit theorem gives the displayed covariance. In the scalar case this is $\mathbb{E}{\psi }^2/A^2$.

On a compact parameter set, a continuous deterministic objective with a unique minimum has a positive separation gap outside each neighbourhood of its minimizer. Uniform convergence in probability of the random objectives bounds the deterministic objective difference at any attained random minimizer by twice the supremum discrepancy. This proves statistical consistency of each measurable minimizing selection.

Uniform convergence in probability of empirical criteria on a compact set proves consistency for minimizers in that set; it does not confine a minimizer over a larger domain. A bounded identifiable regression function can approach its true value again at infinity. For median regression, $a(\theta )={\theta }^2/(1+{\theta }^4)$ has unique zero at zero, but a small positive fitted sample median $c$ has a second solution ${\theta }^2=(1+\sqrt{1-4c^2})/(2c)$. This solution diverges as $c\downarrow 0$, although uniform convergence on any fixed compact interval holds. Constraining the estimator, or proving suitable global separation and localization, repairs the argument.

A scale M-estimator solves an equation of the formfor a positive scale $t$.