Weighted least squares

Weighted least squares minimizes ${\sum }_i{w}_i(Y_i-X_i^T\beta {)}^2$; inverse-variance weights give the efficient estimator under known heteroscedasticity.

Consider the squared error of approximating $x^n$, $n\ge 1$, by a polynomial $q$ under a measure containing $dx/x$ on $(0,1)$. This error is finite exactly when $q(0)=0$. A nonzero constant term makes the integrand asymptotic to $q(0{)}^2/x$, which is not Lebesgue integrable; a vanishing constant term makes $x^n-q(x)=O(x)$, giving an integrable $O(x)$ integrand.

Consequently the full space of polynomials of degree at most $n-1$ is not contained in the weighted L2 space. Its admissible subspace is $\mathrm{span}\{x,\dots ,x^{n-1}\}$, or zero when $n=1$. This finite-dimensional subspace has a unique orthogonal projection of $x^n$, hence a unique best approximation. Indeed its Gram matrix is positive definite: a nonzero polynomial cannot vanish on an interval, so its squared norm is positive. Thus a singular weight can invalidate the proposed full approximation space while still allowing a unique finite-error minimizer in its admissible part. As usual, weighted L2 spaces identify functions equal almost everywhere.

The constant minimizing ${\int }_{-1}^1{x}^{2n}(x^2-c{)}^{2}dx$ is the weighted meanThis follows by differentiating the quadratic error or requiring the residual to be orthogonal to constants. The normalized weight $(2n+1)x^{2n}/2$ assigns mass $r^{2n+1}$ to $[-r,r]$ for every $0<r<1$. As $n$ increases it concentrates at the endpoints, where $x^2=1$, explaining the limiting best constant. Although the weight vanishes at zero, its integral against the square of a nonzero continuous function is positive, so the associated weighted inner product is positive definite.

For a proposed gamma-transition graph, fix the ground-state energy and fit the remaining level energies with a linear incidence matrix. Loops give consistency constraints and correlated excitation-energy errors. Different transition graphs can have the same minimized residuals if they rearrange the same energy-sum identities; a small chi-squared is not then a unique level assignment.

Independent observations with known errors add a positive-semidefinite Fisher information matrix $G$ to the existing information $F$. Writing $F+G=F^{1/2}(I+F^{-1/2}G{F}^{-1/2})F^{1/2}$ shows its inverse covariance cannot increase in any parameter direction. Improvement can be zero along an uninformative direction. This statement does not assume that data-dependent residual variance estimates or model systematic errors must decrease.

When frequency and phase are known, sinusoidal amplitude and offset fitting is a two-parameter linear regression. For equal known errors $\sigma$, profiling out the offset gives $K=\sum (s_i-\bar{s})(y_i-\bar{y})/\sum (s_i-\bar{s}{)}^2$ and $\mathrm{Var}(K)={\sigma }^2/\sum (s_i-\bar{s}{)}^2$. Identical phases cannot separate amplitude from offset. Unknown phase instead requires independent sine and cosine coefficients.

In variable projection, a prediction has the form $\mathbf{G}(\mathbf{q})\mathbf{m}$, nonlinear in $\mathbf{q}$ but linear in $\mathbf{m}$. For each trial $\mathbf{q}$, solve the inner weighted least squares problem for $\mathbf{m}$, then optimize the resulting residual over $\mathbf{q}$. This reduces the nonlinear dimension without pretending that the whole inverse problem is linear.

Generalized least squares minimizes $(Y-X\beta {)}^T{\Omega }^{-1}(Y-X\beta )$ for a known covariance shape $\Omega$.

Whitening multiplies a model by a square root of its precision matrix so that the transformed errors have identity covariance.