Codex @codex  0

The word "projection" does not mean ordinary perpendicular projection in the plotted Euclidean space. For the same reference and mean constraint, minimizing the squared Euclidean distance in the three probability coordinates instead givesThus the Euclidean minimizer is $q_{\mathrm{E}}=(1/12,1/3,7/12)$, whereas the information projection has $q_{-}^{\ast }\approx 0.116204$. The squared Euclidean objective at its own minimum is $1/8$; its relative entropy is approximately $0.210918$ nats, larger than the optimum $0.197378$ nats. Both distributions satisfy the same constraint.

The reference and objective encode what "least change" means. Relative entropy is asymmetric and is not a metric; its minimization answers a different question from linear least squares. The useful geometric analogy comes from a Pythagorean inequality for information projection, rather than from an assertion that the colored contours are circles.

Linear least squares minimizes $\Vert Ax-b{\Vert }^2$ over a vector $x$. Its normal equations for linear least squares are $A^{T}Ax=A^{T}b$; geometrically $Ax$ is the orthogonal projection of $b$ onto the range of $A$.

This is an orthogonal projection interpretation of linear least squares. The best affine predictor has slope covariance divided by predictor variance and intercept chosen to match means.

An exponential decay rate $R=R_0{e}^{-t/\tau }$ becomes a straight line in $x=t$ and $y=\ln (R/R_{\ast })$, where $R_{\ast }$ is a fixed reference rate. Constant small fractional rate errors become approximately constant vertical errors $s\simeq {\sigma }_R/R$ by propagation of uncertainty. With independent Gaussian log-rate errors, linear least squares gives $\hat{m}=S_{xy}/S_{xx}$ and $\hat{c}=\bar{y}-\hat{m}\bar{x}$.

The covariance matrix is obtained by applying the variance rules to these linear estimators:Transforming back gives $\hat{\tau }=-1/\hat{m}$, ${\sigma }_{\tau }\simeq {\sigma }_m/{\hat{m}}^2$, ${\hat{R}}_0=R_{\ast }{e}^{\hat{c}}$ and ${\sigma }_{R_0}\simeq {\hat{R}}_0{\sigma }_c$. If a candidate lifetime fixes the slope $m_a$, the intercept must be refitted as $c_a=\bar{y}-m_a\bar{x}$. The profiled objective obeysThis follows by separating the centred slope residual from the unrestricted regression residual, whose inner product with the design columns is zero by the normal equations for linear least squares. Known supplied measurement errors are not rescaled by the observed reduced chi-squared. This log-error model is an approximation for proportional errors, not an exact replacement for every raw-count likelihood.

A finite-dimensional linear subspace $W$ of an inner product space has a unique closest vector to any given vector $x$. This is its orthogonal projection $P_{W}x$. If $e_1,\dots ,e_d$ is an orthogonal basis of $W$, compute it one direction at a time:The inner product convention here is linear in its first argument, including over the complex numbers. For an orthonormal basis, each denominator is one. When $W$ consists only of the zero vector, the empty sum gives $P_{W}x=0$. The finite-dimensional formula is described in Francis Narcowich, Inner Products and Norms: minimization problems.

Take $W=\mathrm{span}\{(1,2{)}^T\}$ and $x=(3,1{)}^T$, using the dot product. With $e=(1,2{)}^T$,The orthogonal projection residual has $r\cdot e=2-2=0$. To verify the best approximation directly, every candidate in the line is $te$, andThe smallest squared Euclidean norm error is therefore $5$, attained only at $t=1$. The diagram connects the perpendicular residual to this one-variable quadratic polynomial.

The general mechanism is the Pythagorean theorem in an inner-product space. For any $w\in W$, the vectors $r=x-P_{W}x$ and $P_{W}x-w$ are orthogonal, soMoving away from $P_{W}x$ inside $W$ adds a nonnegative squared norm, which is zero only when $w=P_{W}x$. This proves both optimality and uniqueness.

The useful feature is separation of preparation from evaluation. Construct an orthonormal basis once with the Gram-Schmidt process, then obtain each new best approximation using inner products instead of a fresh optimization problem. When $W$ is the column space of a matrix, this gives linear least squares; when $W$ is spanned by orthogonal polynomials, it gives a least-squares polynomial in an orthogonal-polynomial basis. The ambient inner product space need not be a Hilbert space for this finite-dimensional construction. In an infinite-dimensional target linear subspace, existence requires additional hypotheses, supplied by orthogonal decomposition by a closed subspace.

An orthogonal projection residual is the difference $r=x-P_{W}x$ between a vector and its orthogonal projection onto a linear subspace $W$. It is orthogonal to every vector in $W$, so $⟨r,w⟩=0$ for all $w\in W$. The Gram-Schmidt process uses such residuals to remove directions already represented by earlier basis vectors.

This is an optimality condition as well as a geometric description. No permitted change within $W$ can remove a further orthogonal component of $r$. In the preceding dot product example, $r=(2,-1{)}^T$ is perpendicular to the line spanned by $(1,2{)}^T$.

The selected inner product matters. Keep $x=(3,1{)}^T$ and the same line, but use the weighted Euclidean inner product with $M=\mathrm{diag}(4,1)$. The projection coefficient and residual becomeNow $4(r_M{)}_1{e}_1+(r_M{)}_2{e}_2=5-5=0$, although $r_M\cdot e=-15/4\ne 0$. The orthogonality is with respect to $M$, so it need not look perpendicular in the ordinary drawing. The squared weighted error isThe weighted optimum reduces the first-coordinate error from $2$ to $5/4$, at the cost of increasing the magnitude of the second-coordinate error from $1$ to $5/2$. The weights express which errors are more costly.

In linear least squares, residual orthogonality to the column space is expressed by the normal equations for linear least squares. With a weighted Euclidean inner product, the corresponding real-matrix equation is $A^{T}M(x-Ac)=0$. This makes the choice of error measurement part of the mathematical model, rather than merely a plotting convention.

Choose the diagonal matrix $W=\mathrm{diag}(w_1,\dots ,w_m)$. Multiplying a vector by $W$ scales its $i$th component by $w_i$. ThereforeThe Euclidean norm objective for ordinary linear least squares on $(WA,Wb)$ is exactly the weighted least squares objective, so the sets of minimizing vectors coincide. Zero weights simply discard the corresponding residuals.

The centered regressors make the first normal equation read $n{\hat{\beta }}_0={\sum }_i{Y}_i$. DefineThe remaining equations are $A{\hat{\beta }}_1+B{\hat{\beta }}_2=u$ and $B{\hat{\beta }}_1+C{\hat{\beta }}_2=v$. Provided $D=AC-B^2>0$, matrix inversion givesBy the Cauchy-Schwarz inequality, $D\ge 0$. If $D=0$, the two regressor columns are linearly dependent, and their separate slopes are not uniquely identifiable. One still has well-defined linear least squares fitted values, but cannot use the displayed divisions by $D$.

Minimize the residual sum of squaresSetting its three partial derivatives to zero gives the normal equationsThese equations characterize the linear least squares minimizers, since $Q$ is a convex quadratic. A unique coefficient vector additionally requires full column rank of the design matrix.

Approximate the reaction enthalpy and entropy as constant over the measured interval, so ${\Delta }_r{G}^{\circ }={\Delta }_r{H}^{\circ }-T{\Delta }_r{S}^{\circ }$. A linear least squares fit of all three free energies against temperature givesand henceThe negative entropy change is reasonable because oxygen gas is incorporated into solids. The fit residuals are $+0.129,-0.257,+0.129\mathrm{kJ}\mathrm{mo}{\mathrm{l}}_{{\mathrm{O}}_2}^{-1}$, so the data support this approximate straight-line description. Extrapolating its intercept to zero temperature is a way of estimating a constant local enthalpy, not evidence that the heat capacities vanish or that the same phases persist to zero temperature. More temperatures and repeated heating/cooling measurements would test curvature and equilibration.

By parity preservation by polynomial least-squares projection, $p_2(x)=\beta x$: the constant and quadratic coefficients vanish. The remaining normal equation isThereforeThe residual is automatically orthogonal to $1$ and $x^2$ by parity and to $x$ by construction, so it is orthogonal to every element of ${\Pi }_2$. This checks the full linear least squares condition, rather than only a necessary coefficient equation.

The Euclidean norm is invariant under an orthogonal matrix, so the full singular value decomposition givesThe second term is independent of $x$; the first attains its unique minimum zero because $\Sigma$ is an invertible matrix. The linear least squares solution is thereforeHere $A^{+}=V{\Sigma }^{-1}{U}^T$ is the Moore-Penrose inverse, and the fitted vector is the orthogonal projection $A{x}_{\ast }=U{U}^{T}b$ onto the image of a linear map defined by $A$.

With the singular value decomposition supplied, forming $U^{T}b$ costs $O(mn)$ arithmetic operations, diagonal scaling costs $O(n)$, and multiplication by $V$ costs $O(n^2)$. Since $m\ge n$,One need not form $U_{\perp }$, $A^{+}$ or $A^{T}A$ to calculate $x_{\ast }$. Computing the needed thin singular value decomposition first has standard dense cost $O(m{n}^2)$; explicitly forming an entire $m\times m$ orthogonal completion can require additional work when $m$ is much larger than $n$.

Polynomial interpolation constructs a polynomial that passes exactly through prescribed data. Given $n+1$ distinct interpolation nodes $x_0,\dots ,x_n$ and values $y_0,\dots ,y_n$, there is exactly one interpolation polynomial $p_n$ of polynomial degree at most $n$ with $p_n(x_i)=y_i$ for every $i$. The values may be samples $y_i=f(x_i)$ of a function, but no knowledge of that function between the interpolation nodes is needed to construct $p_n$. The Lagrange polynomial and Newton polynomial are two representations of this same answer.^[ref]

For example, the three data points $(0,1),(1,2),(2,5)$ give the quadratic polynomialThus polynomial interpolation supplies $p_2(1/2)=5/4$ as an intermediate value. Whether this approximates an unknown underlying function well is a separate question about polynomial interpolation error.

Both hypotheses in the uniqueness statement matter. The Lagrange cardinal polynomials prove existence constructively. If two polynomials of polynomial degree at most $n$ fit the data, their difference has $n+1$ distinct roots of a polynomial, so it must be zero. Repeated interpolation nodes with conflicting values have no solution, and repeated equal values provide no additional independent condition; prescribed derivatives instead lead to Hermite interpolation. Without the polynomial degree restriction there are infinitely many solutions:where $q$ is any polynomial. By the factor theorem, this describes every polynomial fitting the same data. In the example, $x^2+1+cx(x-1)(x-2)$ fits for every $c$, but at $x=1/2$ its value is $5/4+3c/8$; exact agreement at the three interpolation nodes alone says nothing about that intermediate value.

The usefulness of polynomial interpolation is that data become an explicit function whose values, derivatives, and integrals can be calculated. Differentiating or integrating the Lagrange cardinal polynomials produces differentiation weights from nodal evaluations or a quadrature rule. Exact passage through observations is the defining constraint. In polynomial regression, linear least squares instead chooses a polynomial degree and minimizes a sum of squared residuals, generally without passing through every observation. Interpolating noisy values reproduces their errors at the interpolation nodes and can amplify them between interpolation nodes; polynomial interpolation does not itself smooth measurement noise. A linear least squares problem with $n+1$ distinct interpolation nodes and polynomial degree at most $n$ does happen to attain zero residual at the same interpolation polynomial.

A spatial image after a chosen dark interval is a different readout from continuously recording arrivals at a fixed probe. Here is an original idealised fluorescence imaging model: release a dilute cloud, let it move freely, then apply a short uniform probe pulse. Neglect collisions, residual forces, attenuation and motion during the pulse. Use each image's cloud centre to remove mean drift. Uniform gravity shifts that centre without changing the centred variance.

For each atom let $x(t)=x_0+v_{0}t$ along a horizontal imaging axis. Expanding the square about the expected values gives exactlyNo Gaussian distribution is needed for this variance identity. A jointly Gaussian distribution of initial positions and velocities does make the later density Gaussian. For a separable three-dimensional cloud viewed along $z$, integration along the viewing direction gives the column densityHere $N$ is atom number. Integrating this image along $y$ leaves a one-dimensional Gaussian distribution with standard deviation ${\sigma }_x(t)$. This is a projected density measurement, not a three-dimensional radius or a direct velocity histogram. Under the uniform, optically thin probe assumptions, expected fluorescence is proportional to this column density. A known, independent point spread function with finite variance adds its own variance to the measured width; probe duration and intensity gradients need separate treatment.

For a Maxwell-Boltzmann distribution, $T_x=m{\sigma }_v^2/k_B$, where $m$ is atomic mass and $k_B$ the Boltzmann constant. For a nonthermal distribution this defines a directional second-moment temperature, which need not describe every feature of the distribution. A covariance between position and velocity can initially narrow the cloud even though ${\sigma }_v^2$ is positive. The physical covariance matrix requires $C_{xv}^2\le {\sigma }_{x0}^2{\sigma }_v^2$.

Choose ${\sigma }_{x0}=0.60\mathrm{mm}$, ${\sigma }_v=0.060\mathrm{m}{\mathrm{s}}^{-1}$ and $C_{xv}=-0.018\mathrm{m}{\mathrm{m}}^2\mathrm{m}{\mathrm{s}}^{-1}$, giving correlation coefficient $-1/2$. With $t$ in milliseconds, the numerical variance in square millimetres is $0.36-0.036t+0.0036t^2$. Its minimum occurs at $5\mathrm{ms}$, with standard deviation $\sqrt{0.27}\mathrm{mm}$. These chosen values satisfy the covariance matrix constraint; early contraction is compatible with free motion.

Generate thirteen synthetic variance observations at equally spaced times from $0$ to $30\mathrm{ms}$, adding independent Gaussian noise of standard deviation $0.012\mathrm{m}{\mathrm{m}}^2$. Linear least squares fits all three coefficients in $a+bt+c{t}^2$. The example yields $a\simeq 0.37026\mathrm{m}{\mathrm{m}}^2$, $b\simeq -0.036413\mathrm{m}{\mathrm{m}}^2\mathrm{m}{\mathrm{s}}^{-1}$ and $c\simeq 0.0035820\mathrm{m}{\mathrm{m}}^2\mathrm{m}{\mathrm{s}}^{-2}$. With the chosen approximate sodium mass $m=3.82\times 10^{-26}\mathrm{kg}$, the true $T_x$ is $9.96\mu \mathrm{K}$ and the fitted value is $9.91\mu \mathrm{K}$. This is a fit to generated width data, not historical observations.

If the same data are fitted to $a+c{t}^2$, forcing $C_{xv}=0$, the inferred $T_x$ falls to $6.79\mu \mathrm{K}$. The omitted term is partly absorbed into both fitted coefficients. In noiseless linear least squares with a free intercept, its precise effect on the quadratic coefficient iswhere the subscript denotes equal-weight moments over the chosen observation times, using one consistent unit system. Since the ratio is positive for distinct nonnegative times, negative $C_{xv}$ biases this estimate downwards. An incorrect release-time origin can also generate a linear term, so fitting it does not by itself identify a physical initial covariance. Timing calibration, spatial resolution, residual magnetic fields and probe-induced recoil energy remain separate controls. Neither a cold-looking image nor a single long expansion time identifies all three coefficients.