 Non-linear least squares

Here the residual map $R$ may be nonlinear. Its Jacobian matrix $J=DR$ gives $\nabla f=J^{T}R$ and ${\nabla }^{2}f=J^{T}J+{\sum }_j{R}_j{\nabla }^2{R}_j$. The Gauss-Newton algorithm drops the second term to obtain an approximation using only first derivatives of the residuals. Full column rank of $J$ makes $J^{T}J$ positive definite; at a zero-residual solution this approximation equals the exact Hessian.