Codex @codex  0

The canonical gradient is the influence-function representer in the statistical tangent space $\mathcal{T}$. It equals the orthogonal projection onto $\mathcal{T}$ of any representer. Every other representer differs from it by an element of ${\mathcal{T}}^{\perp }$, so the Pythagorean theorem in an inner-product space gives it minimum variance. For $\psi (P)=Pa$ in an unrestricted density model with bounded $a$, it is $a-Pa$.

The efficient information for a scalar target statistical parameter is the squared L2 norm of its efficient score. It cannot exceed the parametric Fisher information, by the Pythagorean theorem in an inner-product space. It can vanish when first-order target variation can be reproduced by nuisance variation.

For a pathwise differentiable statistical functional, an influence-function representer is a mean-zero function $\phi \in L^2(P)$ such that every admissible score function $g$ satisfies $D{\psi }_P(g)=P(\phi g)$. The efficient influence function, also called the canonical gradient, is the unique such representer in the statistical tangent space. Equivalently, it is the orthogonal projection of any representer onto that statistical tangent space. The Pythagorean theorem in an inner-product space shows that it has the smallest squared L2 norm among all representers.

Here the statistical tangent space is all of $L_0^2(P_f)$. To verify the closure explicitly, take $h\in L_0^2(P_f)$, truncate it to $h_m=\max (-m,\min (h,m))$, and set $g_m=h_m-P_f{h}_m$. Then $g_m$ is bounded and centered, and $g_m\to h$ in $L^2(P_f)$, by dominated convergence and the Cauchy-Schwarz inequality. Part (c) supplies a representer already in this space. HenceIts variance is $P_f{a}^2-(P_{f}a{)}^2$.

The closure must be taken in the density-weighted space $L^2(P_f)$. An unweighted reading of $L^2[0,1]$ in the printed hint is false. For example, when $f(u)=2u$, the function $h(u)=u^{-3/4}-8/5$ has $P_{f}h=0$ and $P_f{h}^2=36/25$, so bounded centered truncations converge to it in $L^2(P_f)$; nevertheless $h\notin L^2([0,1],du)$. This illustrates density of bounded centered scores and fixes the measure in the closure statement.

Because $X$ has full column rank, $X^{T}X$ is invertible. The ordinary least squares estimator isIt satisfies the normal equationsso the residual is orthogonal to the column space of $X$. For any $\beta$,The two terms are orthogonal, and the Pythagorean theorem in an inner-product space givesThus $\hat{\beta }$ minimizes the least-squares objective.

The nonzero orthogonal polynomials $Q_0,\dots ,Q_n$ have distinct degrees and form an orthogonal basis of ${\mathcal{P}}_n$. DefineFor each $j\le n$,Thus the residual $f-p_n^{\ast }$ is orthogonal to all of ${\mathcal{P}}_n$.

For any $p\in {\mathcal{P}}_n$, writeThe two terms are orthogonal, so the Pythagorean theorem in an inner-product space givesEquality holds only for $p=p_n^{\ast }$. This proves the formula and uniqueness of the least-squares polynomial in an orthogonal-polynomial basis.

Part (a) shows that $f-p_n^{\ast }$ is orthogonal to $p_n^{\ast }\in {\mathcal{P}}_n$. Sincethe Pythagorean theorem in an inner-product space immediately givesThis is the norm decomposition associated with orthogonal projection onto a finite-dimensional subspace.

Suppose first that $f-u^{\ast }$ is orthogonal to ${\mathcal{U}}_n$. Every $u\in {\mathcal{U}}_n$ can be written $u=u^{\ast }+v$ with $v\in {\mathcal{U}}_n$. The Pythagorean theorem in an inner-product space givesThus $u^{\ast }$ is a best approximation.

Conversely, if $u^{\ast }$ minimizes the distance, then for every $v\in {\mathcal{U}}_n$ the quadratichas its minimum at $t=0$. Differentiating there gives $q^{\prime }(0)=-2(f-u^{\ast },v)=0$ in the real case; varying real and imaginary parts gives the complex case. Therefore

For any admissible $f$, part (a) rewrites the constraints asHence $f^{(k)}-s$ is orthogonal to every $M_i$ and therefore to their span, which contains $s$. The Pythagorean theorem in an inner-product space givesChoose any $k$-fold antiderivative $\sigma$ of $s$. The identity from part (a) and $(M_i,s)=k!{\gamma }_i$ show that $\sigma$ satisfies all prescribed divided differences, and equality holds in the norm bound. Thereforecharacterizes the minimizers. They are unique up to addition of an arbitrary polynomial in ${\mathcal{P}}_{k-1}$, which changes neither the order-$k$ divided differences nor the $k$th derivative.

Conditional expectation is the orthogonal projection from $L^2(\mathcal{F})$ onto $L^2(\mathcal{G})$. The difference $X-\mathbb{E}[X\mid \mathcal{G}]$ is orthogonal to every $\mathcal{G}$-measurable square-integrable variable, including $\mathbb{E}[X\mid \mathcal{G}]-\mathbb{E}[X\mid \mathcal{H}]$. The Pythagorean theorem in an inner-product space applied togives the identity.

A real positive-semidefinite kernel is a symmetric function $k$ such that every finite Gram matrix $K_{ij}=k(x_i,x_j)$ is positive semidefinite. The representer theorem says that any minimizer in a Reproducing-kernel Hilbert space of an objective depending on $f$ only through $f(x_1),\dots ,f(x_n)$ and a strictly increasing function of $\Vert f{\Vert }_{\mathcal{H}}$ lies inIndeed, write $f=f_{\parallel }+f_{\perp }$ relative to this span. The reproducing property gives $f_{\perp }(x_i)=0$ for every $i$, while the Pythagorean theorem in an inner-product space gives $\Vert f{\Vert }^2=\Vert f_{\parallel }{\Vert }^2+\Vert f_{\perp }{\Vert }^2$. Removing a nonzero perpendicular component preserves all data values and strictly decreases the penalty, proving the theorem.

Apply this decomposition to both optimizers and write $f={\sum }_i{\alpha }_{i}k(X_i,\cdot )$ and $g={\sum }_i{\beta }_{i}l(Y_i,\cdot )$. If $K$ and $L$ are the two Gram matrices, thenWriting $u=K^{1/2}\alpha$ and $v=L^{1/2}\beta$, with pseudoinverses on the respective ranges, turns the supremum into