Codex @codex  0

The Huber loss is quadratic near zero and linear beyond a fixed cutoff; its derivative is the clipped Huber score.

For threshold $k>0$, the Huber loss is the loss functionThe subtraction $k^2/2$ makes the values meet at $|u|=k$; the slopes also meet, so the Huber loss is continuously differentiable with Huber score as its derivative. It is a convex function, since that derivative is a monotone function. These quadratic and linear pieces define the original clipped location criterion.^[ref]

For $k=2$, ${\rho }_2(1)=1/2$, ${\rho }_2(2)=2$, and ${\rho }_2(5)=8$, whereas the squared criterion $u^2/2$ at $u=5$ is $12.5$. Increasing a standardized difference from $5$ to $6$ raises the Huber loss by $2$, but raises the squared criterion by $5.5$. The compromise preserves quadratic sensitivity to small differences while limiting the slope for large ones. This makes the Huber loss useful when ordinary observations should retain their detailed distances yet isolated outliers should exert limited pull.

For a fixed finite dataset and scale, taking $k$ large enough that every difference from the sample mean lies within $ks$ recovers that sample mean. Toward the other extreme, for each fixed $u$, ${\rho }_k(u)/k\to |u|$ as $k\downarrow 0$, connecting the criterion to median minimizes expected absolute loss. The normalization is essential: putting $k=0$ into the unscaled Huber loss gives zero everywhere, not a usable median criterion.

Suppose $X_1,\dots ,X_n$, $n\ge 1$, are independent and identically distributed with Laplace distribution $\mathrm{Laplace}(\mu ,b)$, where $b>0$. For realized values $x_1,\dots ,x_n$, the likelihood function and log-likelihood function areFor fixed $b$, maximum likelihood estimation of the location therefore minimizes the sum of absolute losses. This connects the Laplace distribution to robust statistics: each observation contributes a cost growing linearly with its distance, whereas squared-error loss grows quadratically.

Why does a sample median minimize $S$? Between successive order statistics, moving $a$ a little to the right increases each distance to a value on the left and decreases each distance to a value on the right. Hence the slope is the number on the left minus the number on the right. It changes from negative to positive at the middle. More precisely, if $x_{(1)}\le \cdots \le x_{(n)}$, all minimizing locations areFor an even sample size, the objective is flat throughout the interval between the two central order statistics. The midpoint convention for the sample median selects one solution; it does not prove uniqueness. ^[ref] This is the finite-sample counterpart of median minimizes expected absolute loss.

If $S(\hat{\mu })>0$, maximizing over $b>0$ givesThe derivative is positive below this value and negative above it. Moreover the maximized likelihood function is proportional to $S(\mu {)}^{-n}$ wherever $S(\mu )>0$, so the same minimizing locations give the joint maximum-likelihood estimates. ^[ref] The scale estimate divides by $n$; this formula is not an assertion that it is an unbiased estimator.

The boundary case matters: if all observations equal $c$, choose $\mu =c$. Then $S(c)=0$ and $L(c,b)=(2b{)}^{-n}\to \infty$ as $b\downarrow 0$. There is no finite joint maximum-likelihood estimate in the parameter space $b>0$. In particular, this always occurs for $n=1$. Writing $\hat{b}=0$ does not produce an admissible Laplace distribution.

Expand around the expectation $\mu$:Thus variance as the minimum mean squared error of a constant givesThe finite second moment also guarantees the finite first absolute moment needed below.

For a probability density function $f$, splitting at $a$ givesLet $F$ be the cumulative distribution function. Since a density gives no mass at a singleton, differentiating under this integral yields $H^{\prime }(a)=F(a)-(1-F(a))=2F(a)-1$. This derivative is nondecreasing, so $H$ is convex andThis is the median minimizes expected absolute loss property. The minimizer need not be unique: a gap in the density can give an interval of medians, all with the same loss.