Huber location estimator (${\hat{\theta }}_{k,s}$)

The Huber location estimator solves ${\sum }_i{\psi }_k(x_i-\theta )=0$, where ${\psi }_k$ clips its argument to $[-k,k]$.

More generally, fix a known scale $s>0$ and a dimensionless threshold $k>0$ before fitting. For observations $x_1,\dots ,x_n$, a Huber location estimator is a minimizer of the summed Huber loss, equivalently a root of its estimating equation:Here ${\rho }_k$ is the Huber loss and ${\psi }_k$ the Huber score. The physical clipping distance is $ks$, in the observations' units; the initial formula uses $s=1$. A large observation keeps contributing a bounded signed score rather than being discarded. The original clipped location criterion appears in Peter J. Huber's 1964 paper.^[ref]

For the original data $(0,1,2,3,4,40)$, take $s=1$ and $k=2$. At $\theta =2.5$, the standardized differences and their Huber scores areTheir sum is zero, so the Huber location estimator is $2.5$. Four scores remain in the central linear region, making the root unique. By comparison, the sample mean is $50/6\simeq 8.3333$, and the sample median, using the midpoint of the two middle observations, is $(2+3)/2=2.5$. Agreement with the sample median is a feature of these data, rather than an identity between the two estimators.

The geometry explains both existence and possible nonuniqueness. The Huber loss is a convex function, so $Q_{k,s}$ is a convex function of $\theta$ and its derivative is $-s^{-1}{\sum }_i{\psi }_k((x_i-\theta )/s)$. For $n\ge 1$, $Q_{k,s}(\theta )$ tends to infinity as $|\theta |$ grows, hence attains a minimum; its zero derivative characterizes every minimizer. Between clipping transitions, the summed score has slope $-m/s$, where $m$ is the number of observations strictly within $ks$ of $\theta$. A root with $m>0$ is unique. If $m=0$, a flat interval is possible: for data $(0,10)$ with $s=1,k=2$, every $\theta \in [2,8]$ is a minimizer. A complete single-valued estimator then needs a selection rule, such as the interval's midpoint.

A bounded Huber score limits one observation's contribution to the estimating equation. The influence function of a Huber location estimator additionally divides by the population's central mass, so the size of its bound depends on the reference probability distribution. Clipping does not by itself make the bias of an estimator zero or prove a universal breakdown point; the separate local influence versus replacement breakdown argument specifies what fixed scale permits. Estimating $s$ from the same data changes the procedure and requires separate analysis.

Hold $(0,1,2,3,4)$ fixed and move only the sixth observation $M$. For $M\ge 5$, the sample mean is $(10+M)/6$, the sample median is $2.5$, and the Huber location estimator with fixed $s=1,k=2$ stays at $2.5$. Once $M-2.5\ge 2$, its Huber score is already $2$; moving it farther changes the Huber loss but leaves the root unchanged.

For a closer view of the transition, the exact Huber location estimator over $0\le M\le 40$ isTo verify the first branch, when $0\le M\le 2$ the score of $4$ is saturated and the other five scores sum with it to $8+M-5\theta$. When $2\le M\le 4.5$, the score of $0$ is saturated instead, giving the same expression. At $M=4.5$, the moving observation saturates too, and the equation becomes $10-4\theta =0$. The piecewise formula is continuous at the transition.

The rising sample mean comes from an unbounded linear score, the flat distant-outlier response from a bounded Huber score. This is a finite-data response curve with a specified scale and threshold, rather than a population influence function or a general claim about arbitrary numbers of outliers.

The Huber score clips a residual at a fixed threshold:For $k>0$, the Huber score is the derivative of the Huber loss:It is a continuous odd function and a monotone function, with Lipschitz continuity and $|{\psi }_k(u)|\le k$. In a Huber location estimator, its argument is the standardized difference $u=(x-\theta )/s$. Its derivative is $1$ inside the clipping interval and $0$ outside; that derivative is undefined at the two endpoints. The score itself remains continuous there. This estimating score need not be the score function of a separately specified statistical model, nor is it already a normalized influence function.

For standard normal distribution location estimation, an influence function $h$ obeys $\mathbb{E}[Zh(Z)]=1$. Under $|h|\le C$, minimize $\mathbb{E}{h}^2$ by projecting $ax$ onto $[-C,C]$, choosing $a$ to meet that identity. For $C>\sqrt{\pi /2}$ this gives the displayed Huber score normalized by $2\Phi (K)-1$, with $C=K/(2\Phi (K)-1)$. Pointwise minimality of $h^2-2axh$ proves global optimality after integration. At $C=\sqrt{\pi /2}$ the only feasible normalized function is $C\mathrm{sgn}(x)$ almost everywhere, the influence function of the sample median. The endpoint is a limit of rescaled clipped scores, not the identically zero score obtained by substituting $K=0$ literally.

The Huber loss is quadratic near zero and linear beyond a fixed cutoff; its derivative is the clipped Huber score.

For threshold $k>0$, the Huber loss is the loss functionThe subtraction $k^2/2$ makes the values meet at $|u|=k$; the slopes also meet, so the Huber loss is continuously differentiable with Huber score as its derivative. It is a convex function, since that derivative is a monotone function. These quadratic and linear pieces define the original clipped location criterion.^[ref]

For $k=2$, ${\rho }_2(1)=1/2$, ${\rho }_2(2)=2$, and ${\rho }_2(5)=8$, whereas the squared criterion $u^2/2$ at $u=5$ is $12.5$. Increasing a standardized difference from $5$ to $6$ raises the Huber loss by $2$, but raises the squared criterion by $5.5$. The compromise preserves quadratic sensitivity to small differences while limiting the slope for large ones. This makes the Huber loss useful when ordinary observations should retain their detailed distances yet isolated outliers should exert limited pull.

For a fixed finite dataset and scale, taking $k$ large enough that every difference from the sample mean lies within $ks$ recovers that sample mean. Toward the other extreme, for each fixed $u$, ${\rho }_k(u)/k\to |u|$ as $k\downarrow 0$, connecting the criterion to median minimizes expected absolute loss. The normalization is essential: putting $k=0$ into the unscaled Huber loss gives zero everywhere, not a usable median criterion.

A Huber loss applied to a discrete gradient combines quadratic penalization of small slopes with linear penalization of large slopes. This interpolates between squared-gradient smoothing and total variation denoising, allowing smooth regions while retaining an edge-preserving large-gradient regime. The threshold should be chosen consistently with the signal's units and discretization. It can reduce staircasing in total variation denoising without guaranteeing its elimination.