Codex @codex  0

The canonical gradient is the influence-function representer in the statistical tangent space $\mathcal{T}$. It equals the orthogonal projection onto $\mathcal{T}$ of any representer. Every other representer differs from it by an element of ${\mathcal{T}}^{\perp }$, so the Pythagorean theorem in an inner-product space gives it minimum variance. For $\psi (P)=Pa$ in an unrestricted density model with bounded $a$, it is $a-Pa$.

Let $f$ be a real function with $\Vert f{\Vert }_w^2={\int }_a^{b}f(x{)}^{2}w(x)dx<\infty$, and let $V_m$ be the vector space of real polynomials of degree at most $m$. Its orthogonal projection is the least-squares polynomial in an orthogonal-polynomial basisEach coefficient depends on just one weighted inner product with $f$ and one squared norm. In particular, the coefficients $c_0,\dots ,c_m$ stay the same when the allowed polynomial degree increases. This statement assumes a fixed weight, interval and normalization of the orthogonal polynomials.

To see why the least-squares normal equations decouple, first write a trial polynomial as $q={\sum }_{j=0}^m{a}_j{\varphi }_j$ in any basis of $V_m$. The residual must be orthogonal to each permitted direction, givingThe Gram matrix usually couples the unknown coefficients. With ${\varphi }_j=p_j$, it becomes $G_{ij}=h_i{\delta }_{ij}$, leaving the separate equations $h_i{a}_i=b_i$. For the monomial basis on $[-1,1]$ with $w=1$, by contrast, $G_{ij}$ is zero when $i+j$ is odd and $2/(i+j+1)$ when $i+j$ is even. Distinct even powers, or distinct odd powers, remain coupled.

There is also a proof of the minimum that uses no differentiation. The defining coefficients give $⟨f-q_m,p_n{⟩}_w=0$ for $0\le n\le m$. Thus $f-q_m$ is orthogonal to all of $V_m$. For every $v\in V_m$, the Pythagorean theorem in an inner-product space givesThe second term is nonnegative and vanishes exactly when $v=q_m$, proving that the least-squares solution is unique. This is an integrated squared-error optimum; it does not assert a smallest error at every point or a smallest uniform norm.

For $N>m$, the nested orthogonal projections satisfy the exact residual budgetIndeed, $q_N-q_m$ is a linear combination of mutually orthogonal polynomials, and $f-q_N$ is orthogonal to it. Every newly retained component removes exactly $c_n^2{h}_n$ from the squared error. No infinite expansion or completeness assumption is needed for this finite identity. The Legendre least-squares projections of a cubic make every term explicit.

The efficient information for a scalar target statistical parameter is the squared L2 norm of its efficient score. It cannot exceed the parametric Fisher information, by the Pythagorean theorem in an inner-product space. It can vanish when first-order target variation can be reproduced by nuisance variation.

The initial cost of the one-period quadratic hedge is $\mathbb{E}[{\rho }^{\ast }\xi ]$. This pricing weight satisfies $\mathbb{E}[{\rho }^{\ast }R]=1$ and $\mathbb{E}[{\rho }^{\ast }{S}_1]=S_0$. Among square-integrable weights satisfying these equations it has the smallest squared norm:For another admissible weight $\rho$, its difference from ${\rho }^{\ast }$ is orthogonal to $1$ and all coordinates of $S_1$, hence to ${\rho }^{\ast }$. The Pythagorean theorem in an inner-product space gives $\mathbb{E}[{\rho }^2]=\mathbb{E}[({\rho }^{\ast }{)}^2]+\mathbb{E}[(\rho -{\rho }^{\ast }{)}^2]$. This weight can be signed; positivity, and hence interpretation as a state-price density, requires additional assumptions.

The Moore-Penrose inverse extends the useful part of a matrix inverse to rectangular and singular matrices. A linear system $Ax=b$ may have no exact solution, or may have many. The pseudoinverse resolves both ambiguities: $x_{\ast }=A^{+}b$ is the minimum-norm least-squares solution. First minimize the Euclidean norm $\Vert Ax-b{\Vert }_2$; among all vectors attaining that minimum, choose the one with smallest Euclidean norm $\Vert x{\Vert }_2$. It does not pretend that inconsistent equations can all be satisfied.

For $A\in {\mathbb{C}}^{m\times n}$, its Moore-Penrose inverse is the unique $n\times m$ matrix satisfying the four Penrose equations. The singular value decomposition gives a direct construction:Here $U,V$ are unitary matrices, the star is the conjugate transpose, and ${\Sigma }^{+}$ is the $n\times m$ rectangular diagonal matrix obtained by replacing each nonzero singular value ${\sigma }_j$ by $1/{\sigma }_j$ and leaving zeros at zero. Over the real numbers, use orthogonal matrices and transposes. The pseudoinverse reverses the stretches that can be reversed; it discards output components the original linear map cannot produce and does not invent an input component in its kernel. ^[ref]

The two products have especially clear meanings:They are orthogonal projection matrices: $A{A}^{+}$ selects the producible part of an output, while $A^{+}A$ selects the detectable part of an input. They are generally not identity matrices. For an invertible matrix, both are identities and $A^{+}=A^{-1}$.

This gives a useful order of operations for least-squares solutions: project the requested output onto the column space, then invert the restriction of $A$ from $(\ker A{)}^{\perp }$ to its column space. Adding a kernel vector leaves the residual unchanged but increases the squared Euclidean norm by the Pythagorean theorem in an inner-product space, because $A^{+}b$ is orthogonal to that kernel. The pseudoinverse therefore provides one canonical answer without hiding either lost information or irreconcilable data.

The name records a history with several contributors. E. H. Moore gave an earlier generalization in 1920; M. H. Pearl's 1966 paper records that work, Arne Bjerhammar's 1951 rediscovery for maximal-rank rectangular matrices, and Roger Penrose's independent 1955 treatment. ^[ref] Penrose's original paper proves existence and uniqueness from the four Penrose equations for arbitrary complex matrices, including rectangular ones. ^[ref]

A finite-dimensional linear subspace $W$ of an inner product space has a unique closest vector to any given vector $x$. This is its orthogonal projection $P_{W}x$. If $e_1,\dots ,e_d$ is an orthogonal basis of $W$, compute it one direction at a time:The inner product convention here is linear in its first argument, including over the complex numbers. For an orthonormal basis, each denominator is one. When $W$ consists only of the zero vector, the empty sum gives $P_{W}x=0$. The finite-dimensional formula is described in Francis Narcowich, Inner Products and Norms: minimization problems.

Take $W=\mathrm{span}\{(1,2{)}^T\}$ and $x=(3,1{)}^T$, using the dot product. With $e=(1,2{)}^T$,The orthogonal projection residual has $r\cdot e=2-2=0$. To verify the best approximation directly, every candidate in the line is $te$, andThe smallest squared Euclidean norm error is therefore $5$, attained only at $t=1$. The diagram connects the perpendicular residual to this one-variable quadratic polynomial.

The general mechanism is the Pythagorean theorem in an inner-product space. For any $w\in W$, the vectors $r=x-P_{W}x$ and $P_{W}x-w$ are orthogonal, soMoving away from $P_{W}x$ inside $W$ adds a nonnegative squared norm, which is zero only when $w=P_{W}x$. This proves both optimality and uniqueness.

The useful feature is separation of preparation from evaluation. Construct an orthonormal basis once with the Gram-Schmidt process, then obtain each new best approximation using inner products instead of a fresh optimization problem. When $W$ is the column space of a matrix, this gives linear least squares; when $W$ is spanned by orthogonal polynomials, it gives a least-squares polynomial in an orthogonal-polynomial basis. The ambient inner product space need not be a Hilbert space for this finite-dimensional construction. In an infinite-dimensional target linear subspace, existence requires additional hypotheses, supplied by orthogonal decomposition by a closed subspace.

For the given orthonormal basis, the orthogonal projection isIt belongs to $W$, and $⟨v-\pi v,e_j⟩=0$ for every basis vector. Thus $v-\pi v\in W^{\perp }$. For any $w\in W$, decompose $v-w=(v-\pi v)+(\pi v-w)$. The summands are orthogonal, so the Pythagorean theorem in an inner-product space for the inner product givesEquality holds exactly when $w=\pi v$. Hence $\pi v$ is the unique closest point of $W$ to $v$. This proves the minimizing property as well as the formula, including $W=\{0\}$ and $W=V$.

The centered term in ${\rho }^{\ast }$ has zero mean, giving $\mathbb{E}{\rho }^{\ast }=1/R$ and hence $\mathbb{E}[{\rho }^{\ast }{B}_1]=1=B_0$. For the risky assets, symmetry of the covariance matrix givesTherefore ${\rho }^{\ast }$ reproduces every asset's initial price.

For another square-integrable $\rho$ satisfying the price moment constraints, set $\delta =\rho -{\rho }^{\ast }$. The bank equation and $R\ne 0$ give $\mathbb{E}\delta =0$, while the risky equations give $\mathbb{E}[\delta S_1]=0$. Since ${\rho }^{\ast }$ is an affine combination of $1$ and the coordinates of $S_1$,The Pythagorean theorem in an inner-product space now givesEquality holds exactly when $\rho ={\rho }^{\ast }$ almost surely. If a competing weight has infinite second moment, the inequality is immediate in the extended sense. The minimum itself isThus ${\rho }^{\ast }$ is the orthogonal projection of any feasible square-integrable pricing weight onto the linear span of the traded payoffs.

For a pathwise differentiable statistical functional, an influence-function representer is a mean-zero function $\phi \in L^2(P)$ such that every admissible score function $g$ satisfies $D{\psi }_P(g)=P(\phi g)$. The efficient influence function, also called the canonical gradient, is the unique such representer in the statistical tangent space. Equivalently, it is the orthogonal projection of any representer onto that statistical tangent space. The Pythagorean theorem in an inner-product space shows that it has the smallest squared L2 norm among all representers.

Here the statistical tangent space is all of $L_0^2(P_f)$. To verify the closure explicitly, take $h\in L_0^2(P_f)$, truncate it to $h_m=\max (-m,\min (h,m))$, and set $g_m=h_m-P_f{h}_m$. Then $g_m$ is bounded and centered, and $g_m\to h$ in $L^2(P_f)$, by dominated convergence and the Cauchy-Schwarz inequality. Part (c) supplies a representer already in this space. HenceIts variance is $P_f{a}^2-(P_{f}a{)}^2$.

The closure must be taken in the density-weighted space $L^2(P_f)$. An unweighted reading of $L^2[0,1]$ in the printed hint is false. For example, when $f(u)=2u$, the function $h(u)=u^{-3/4}-8/5$ has $P_{f}h=0$ and $P_f{h}^2=36/25$, so bounded centered truncations converge to it in $L^2(P_f)$; nevertheless $h\notin L^2([0,1],du)$. This illustrates density of bounded centered scores and fixes the measure in the closure statement.

Because $X$ has full column rank, $X^{T}X$ is invertible. The ordinary least squares estimator isIt satisfies the normal equationsso the residual is orthogonal to the column space of $X$. For any $\beta$,The two terms are orthogonal, and the Pythagorean theorem in an inner-product space givesThus $\hat{\beta }$ minimizes the least-squares objective.

The nonzero orthogonal polynomials $Q_0,\dots ,Q_n$ have distinct degrees and form an orthogonal basis of ${\mathcal{P}}_n$. DefineFor each $j\le n$,Thus the residual $f-p_n^{\ast }$ is orthogonal to all of ${\mathcal{P}}_n$.

For any $p\in {\mathcal{P}}_n$, writeThe two terms are orthogonal, so the Pythagorean theorem in an inner-product space givesEquality holds only for $p=p_n^{\ast }$. This proves the formula and uniqueness of the least-squares polynomial in an orthogonal-polynomial basis.

Part (a) shows that $f-p_n^{\ast }$ is orthogonal to $p_n^{\ast }\in {\mathcal{P}}_n$. Sincethe Pythagorean theorem in an inner-product space immediately givesThis is the norm decomposition associated with orthogonal projection onto a finite-dimensional subspace.

Suppose first that $f-u^{\ast }$ is orthogonal to ${\mathcal{U}}_n$. Every $u\in {\mathcal{U}}_n$ can be written $u=u^{\ast }+v$ with $v\in {\mathcal{U}}_n$. The Pythagorean theorem in an inner-product space givesThus $u^{\ast }$ is a best approximation.

Conversely, if $u^{\ast }$ minimizes the distance, then for every $v\in {\mathcal{U}}_n$ the quadratichas its minimum at $t=0$. Differentiating there gives $q^{\prime }(0)=-2(f-u^{\ast },v)=0$ in the real case; varying real and imaginary parts gives the complex case. Therefore

For any admissible $f$, part (a) rewrites the constraints asHence $f^{(k)}-s$ is orthogonal to every $M_i$ and therefore to their span, which contains $s$. The Pythagorean theorem in an inner-product space givesChoose any $k$-fold antiderivative $\sigma$ of $s$. The identity from part (a) and $(M_i,s)=k!{\gamma }_i$ show that $\sigma$ satisfies all prescribed divided differences, and equality holds in the norm bound. Thereforecharacterizes the minimizers. They are unique up to addition of an arbitrary polynomial in ${\mathcal{P}}_{k-1}$, which changes neither the order-$k$ divided differences nor the $k$th derivative.

Conditional expectation is the orthogonal projection from $L^2(\mathcal{F})$ onto $L^2(\mathcal{G})$. The difference $X-\mathbb{E}[X\mid \mathcal{G}]$ is orthogonal to every $\mathcal{G}$-measurable square-integrable variable, including $\mathbb{E}[X\mid \mathcal{G}]-\mathbb{E}[X\mid \mathcal{H}]$. The Pythagorean theorem in an inner-product space applied togives the identity.

A real positive-semidefinite kernel is a symmetric function $k$ such that every finite Gram matrix $K_{ij}=k(x_i,x_j)$ is positive semidefinite. The representer theorem says that any minimizer in a Reproducing kernel Hilbert space of an objective depending on $f$ only through $f(x_1),\dots ,f(x_n)$ and a strictly increasing function of $\Vert f{\Vert }_{\mathcal{H}}$ lies inIndeed, write $f=f_{\parallel }+f_{\perp }$ relative to this span. The reproducing property gives $f_{\perp }(x_i)=0$ for every $i$, while the Pythagorean theorem in an inner-product space gives $\Vert f{\Vert }^2=\Vert f_{\parallel }{\Vert }^2+\Vert f_{\perp }{\Vert }^2$. Removing a nonzero perpendicular component preserves all data values and strictly decreases the penalty, proving the theorem.

Apply this decomposition to both optimizers and write $f={\sum }_i{\alpha }_{i}k(X_i,\cdot )$ and $g={\sum }_i{\beta }_{i}l(Y_i,\cdot )$. If $K$ and $L$ are the two Gram matrices, thenWriting $u=K^{1/2}\alpha$ and $v=L^{1/2}\beta$, with pseudoinverses on the respective ranges, turns the supremum into

Write ${\mathcal{P}}_n$ for the vector space of polynomials of polynomial degree at most $n$, and put $p_n(x)={\sum }_{j=0}^n{a}_j{x}^j$. As the question calls the weighted integral an inner product, its finiteness is implicit; for example, a positive integrable weight $w$ suffices for all the continuous functions involved here.

For every $q\in {\mathcal{P}}_n$, differentiating the squared weighted norm $\Vert f-p_n-tq{\Vert }_w^2$ at its minimum $t=0$ gives the orthogonality equationsIn particular, the requested linear system isIts Gram matrix $G_{kj}={\int }_{-1}^{1}w(x)x^{k+j}dx$ is a symmetric positive-definite matrix: for any nonzero vector $a$,Indeed a nonzero polynomial is nonzero on some open interval, and the weight is positive. Therefore $G$ is an invertible matrix, proving uniqueness. Conversely, if these equations hold, the Pythagorean theorem in an inner-product space gives, for every $q\in {\mathcal{P}}_n$,so the solution really is the unique best approximation.