= Information projection and Euclidean projection
The word "projection" does not mean ordinary perpendicular projection in the plotted <Euclidean space>. For the same reference and <mean> constraint, minimizing the squared <Euclidean distance> in the three <probability> coordinates instead gives
$$
\sum_i(q_i-1/3)^2=6x^2-x+\frac16,
\qquad x_{\mathrm E}=\frac1{12}.
$$
Thus the Euclidean minimizer is $q_{\mathrm E}=(1/12,1/3,7/12)$, whereas the <information projection> has $q_-^*\approx0.116204$. The squared Euclidean objective at its own minimum is $1/8$; its <relative entropy> is approximately $0.210918$ <nats>, larger than the optimum $0.197378$ <nats>. Both distributions satisfy the same constraint.
\Image[/information-projection-relative-entropy-versus-least-squares.png]
{title=Relative entropy and least squares select different laws under the same mean constraint}
{description=Both plots use the feasible laws $q=(x,1/2-2x,1/2+x)$ and the same uniform reference. The left plot minimizes <relative entropy>; the right plot minimizes squared <Euclidean distance> in the three probability coordinates. Their different marked minima explain why the choice of objective matters.}
The reference and objective encode what "least change" means. <Relative entropy> is asymmetric and is not a <metric>; its minimization answers a different question from <linear least squares>. The useful geometric analogy comes from a <Pythagorean inequality for information projection>, rather than from an assertion that the colored contours are circles.
Back to article page
