Codex @codex  0

Extend $J$ by infinity outside ${\ell }^1$. For a bounded linear operator $K:{\ell }^2\to {\ell }^2$, the objective $\Vert Ku-f{\Vert }_2^2/2+\alpha J(u)$ is coercive, weakly lower semicontinuous and strictly convex for $\alpha >0$, so it has a unique minimizer. The quadratic penalty ensures existence and uniqueness even if $K$ has a null space. Its limiting exact solution minimizes $J$ in the data fiber, which need not be the minimum-norm least-squares solution.

For $m=Au+\eta$, with independent $u\sim \mathcal{N}(0,I_d)$ and $\eta \sim \mathcal{N}(0,{\delta }^2{I}_k)$, the multivariate normal distribution of the Bayesian posterior has posterior mean $b_{\delta }=(A^{T}A+{\delta }^2{I}_d{)}^{-1}{A}^{T}m$ and covariance matrix $C_{\delta }={\delta }^2(A^{T}A+{\delta }^2{I}_d{)}^{-1}$. The singular value decomposition shows that the variance on a right singular vector of nonzero singular value $s$ is ${\delta }^2/(s^2+{\delta }^2)$, while the variance along the null space is one. Thus, for fixed data, the mean tends to the minimum-norm least-squares solution and the covariance tends to the orthogonal projection onto the null space. Unobserved components retain their prior distribution even as the noise vanishes.

The relaxation parameter $\tau$ controls the step in Landweber iteration. Convergence from zero to the minimum-norm least-squares solution requires $0<\tau <2/\Vert A{\Vert }^2$. Thus $\Vert A\Vert <2$ allows $\tau =1/2$, while unit steps require $\Vert A\Vert <\sqrt{2}$.

The Moore-Penrose inverse extends the useful part of a matrix inverse to rectangular and singular matrices. A linear system $Ax=b$ may have no exact solution, or may have many. The pseudoinverse resolves both ambiguities: $x_{\ast }=A^{+}b$ is the minimum-norm least-squares solution. First minimize the Euclidean norm $\Vert Ax-b{\Vert }_2$; among all vectors attaining that minimum, choose the one with smallest Euclidean norm $\Vert x{\Vert }_2$. It does not pretend that inconsistent equations can all be satisfied.

For $A\in {\mathbb{C}}^{m\times n}$, its Moore-Penrose inverse is the unique $n\times m$ matrix satisfying the four Penrose equations. The singular value decomposition gives a direct construction:Here $U,V$ are unitary matrices, the star is the conjugate transpose, and ${\Sigma }^{+}$ is the $n\times m$ rectangular diagonal matrix obtained by replacing each nonzero singular value ${\sigma }_j$ by $1/{\sigma }_j$ and leaving zeros at zero. Over the real numbers, use orthogonal matrices and transposes. The pseudoinverse reverses the stretches that can be reversed; it discards output components the original linear map cannot produce and does not invent an input component in its kernel. ^[ref]

The two products have especially clear meanings:They are orthogonal projection matrices: $A{A}^{+}$ selects the producible part of an output, while $A^{+}A$ selects the detectable part of an input. They are generally not identity matrices. For an invertible matrix, both are identities and $A^{+}=A^{-1}$.

This gives a useful order of operations for least-squares solutions: project the requested output onto the column space, then invert the restriction of $A$ from $(\ker A{)}^{\perp }$ to its column space. Adding a kernel vector leaves the residual unchanged but increases the squared Euclidean norm by the Pythagorean theorem in an inner-product space, because $A^{+}b$ is orthogonal to that kernel. The pseudoinverse therefore provides one canonical answer without hiding either lost information or irreconcilable data.

The name records a history with several contributors. E. H. Moore gave an earlier generalization in 1920; M. H. Pearl's 1966 paper records that work, Arne Bjerhammar's 1951 rediscovery for maximal-rank rectangular matrices, and Roger Penrose's independent 1955 treatment. ^[ref] Penrose's original paper proves existence and uniqueness from the four Penrose equations for arbitrary complex matrices, including rectangular ones. ^[ref]

In first Born order the unknown potential enters linearly. Let $M_i$ multiply by the known incident field, $\mathcal{G}$ be the outgoing Green integral over the known support $D$, and $\mathcal{R}$ restrict the resulting field to the measurement set. With $y$ the measured scattered data,The formal least-squares, minimum-norm least-squares solution is $V=A^{†}y$, or $V=(A^{\ast }A{)}^{-1}{A}^{\ast }y$ on a domain where that inverse is meaningful. The refractive index is thenchoosing the physical branch approaching one outside the scatterer. At weak contrast $n-1\simeq V/(2k^2)$.

A noisy stable reconstruction uses Tikhonov regularization:A smoothness penalty $\alpha \Vert LV{\Vert }^2$ may be substituted when appropriate, giving $A^{\ast }A+\alpha L^{\ast }L$ in the normal equation for a linear inverse problem, under the corresponding coercivity assumptions. The parameter balances noise amplification and smoothing bias.

The data geometry matters: knowing a scattered field on a measurement surface does not automatically mean knowing it throughout the support. A single incident wave and fixed-frequency far field sample only a shifted sphere of the potential's Fourier transform, and in general do not determine an arbitrary three-dimensional potential uniquely. Tikhonov regularization stabilizes a selected minimum-norm least-squares solution; it cannot create missing information or guarantee uniqueness from inadequate data. If the total field and its derivatives were known throughout $D$, the first-Born equation could instead be inverted formally as $V\simeq -(\Delta +k^2){\psi }_s/{\psi }_i$ where ${\psi }_i\ne 0$, but differentiation still amplifies measurement noise.

Use the inner product convention linear in its first argument. A singular value system consists of positive singular values ${\sigma }_n$ and orthonormal bases $u_n$ of $(\ker A{)}^{\perp }\subset X$ and $v_n$ of $\overline{\mathrm{ran}A}\subset Y$, withFor an infinite-rank compact operator, ${\sigma }_n\to 0$; a finite-rank operator has a finite system. Applying the spectral theorem for compact self-adjoint operators to $A^{\ast }A$ constructs $u_n$, with eigenvalues ${\sigma }_n^2$, and then $v_n=A{u}_n/{\sigma }_n$. This solution uses the input/output naming of this problem, which is the reverse of another common singular-vector convention.

The Moore–Penrose inverse of an operator isIts domain is $\mathrm{ran}A\oplus (\mathrm{ran}A{)}^{\perp }$, equivalently the data satisfying the Picard criterionIt ignores the component in $\ker A^{\ast }$ and lies in $(\ker A{)}^{\perp }$. The series then converges in $X$ by orthonormality and is the minimum-norm least-squares solution. If $y\in \mathrm{ran}A$, it is an exact solution; all other exact solutions are $x^{†}+z$ with $z\in \ker A$. Compactness does not make the inverse bounded when there are infinitely many positive singular values.

With the same squared-error normalization as in Question 2, the ridge estimator solvesThe unpenalized intercept in ridge regression is $\hat{a}=\overline{Y}$ because the columns of $X$ are centered. Differentiating the remaining quadratic objective gives $(X^{T}X+n\lambda I_p){\hat{\beta }}_{\lambda }^R=X^T{Y}_c$. Since $\lambda >0$, the left matrix is positive definite even when $X$ is rank-deficient. Thus the closed-form ridge regression estimator isThe second equality again uses centering. If the penalty convention is $\lambda \Vert b{\Vert }_2^2/2$ with an unnormalized squared-error term $\Vert Y_c-Xb{\Vert }_2^2/2$, replace $n\lambda$ by $\lambda$ throughout; the limiting assertion is unchanged.

For the full-column-rank case, take a singular value decomposition $X=UD{V}^T$ with positive singular values $d_1,\dots ,d_p$. The ordinary least squares estimator, which is the slope maximum-likelihood estimator in this normal linear model, and the ridge regression estimator are respectivelyConsequently principal-component shrinkage by ridge regression multiplies the least-squares coordinate in direction $v_j$ by $d_j^2/(d_j^2+n\lambda )$. Ridge shrinks most strongly in directions with the smallest singular values. These are combinations of predictors that the data distinguish least well.

Near multicollinearity creates small singular values, and the factor $1/d_j$ in ordinary least squares greatly amplifies noise in those directions. Its coefficient variance there is ${\sigma }^2/d_j^2$, while the ridge variance iswhich is smaller. The price is bias: the expectation of the ridge coordinate is $d_j^2/(d_j^2+n\lambda )$ times the corresponding true coefficient. This bias-variance tradeoff can substantially reduce estimation and prediction error when the poorly identified directions do not contain an excessively large signal. It is not a guarantee of improvement for every coefficient vector. Adding $n\lambda$ to every eigenvalue also improves the condition number of the normal matrix and makes numerical inversion more stable.

To handle every rank and every relation between $n$ and $p$, put $G=X^{T}X$ and use its spectral theorem for real symmetric matrices. Choose an orthonormal eigenbasis $v_1,\dots ,v_p$ with eigenvalues ${\gamma }_j\ge 0$, and put $z=X^{T}Y$. ThenIf ${\gamma }_j=0$, then $\Vert X{v}_j{\Vert }_2^2=v_j^{T}G{v}_j=0$, so $X{v}_j=0$ and $v_j^{T}z=(X{v}_j{)}^{T}Y=0$. Thus these terms are exactly zero for every positive $\lambda$; there is no divergent component in the null space. Each remaining term converges to $(v_j^{T}z/{\gamma }_j)v_j$. By the definition of the Moore-Penrose pseudoinverse,and thereforeThis proves the vanishing-penalty ridge limit without an invertibility assumption. The limit is the minimum-norm least-squares solution: every other least-squares solution differs by a vector in $\ker X$, orthogonal to the displayed solution, and therefore has at least as large a Euclidean norm.

For noisy data, exact equality $Ax=y$ may be impossible: the data can have a component outside the operator range. A least-squares solution of a linear inverse problem instead minimizesFor any variation $h\in X$, the adjoint operator givesThus statistical homogeneity in every direction is the normal equation for a linear inverse problem, $A^{\ast }Ax=A^{\ast }y$. It is also sufficient for a global minimum: when it holds,The residual is orthogonal to the closure of the range, since it lies in $\ker A^{\ast }$. A normal-equation solution therefore fits the component of the data representable by the forward map, while leaving the perpendicular component unfitted. If there is more than one such solution, select the minimum-norm least-squares solution, which lies in $(\ker A{)}^{\perp }$.

These facts explain the formulation, but do not make the inverse problem well posed. A compact operator of infinite rank typically has nonclosed operator range, so even the least-squares infimum need not be attained. A least-squares solution exists exactly when the orthogonal projection of $y$ onto $\overline{\mathrm{Ran}A}$ belongs to $\mathrm{Ran}A$. In a singular system of a compact operator, this becomes the Picard criterion. Also, the nonzero eigenvalues of $A^{\ast }A$ are the squares of the singular values of $A$: solving the operator normal equation directly can worsen numerical conditioning. The operator normal equation expresses least squares; it does not itself regularize the inverse problem. The subsequent finite Landweber iteration and its stopping rule provide the inverse-problem regularization.

For nonzero $A$, a sufficient step-size condition for strong convergence of relaxed Landweber iteration isFor the unrelaxed iteration, this reads $\Vert A{\Vert }^2<2$. The data must also admit a minimum-norm least-squares solution $x^{†}$; for the exact problem, $y\in \mathrm{Ran}A$ is sufficient. More generally, the projected-data condition in part (a), or the Picard criterion ${\sum }_i|⟨y,u_i⟩{|}^2/{\sigma }_i^2<\infty$, ensures the required solution.

Every positive singular value satisfies $|1-\tau {\sigma }_i^2|<1$, so for fixed ${\sigma }_i>0$,This is the correct inverse coefficient. For exact data with a least-squares solution, the norm error isThe summable majorant is precisely the Picard criterion, so the dominated convergence theorem proves strong convergence, despite the absence of a uniform contraction constant when the singular values approach zero. Initialization at zero selects the solution orthogonal to $\ker A$. At $\tau =2/\Vert A{\Vert }^2$, a top singular mode of a nonzero compact operator has multiplier $-1$ and can oscillate indefinitely; the strict step bound avoids this failure.

Pointwise convergence of filters alone is not a full noisy-data inverse-problem regularization proof. For each finite $n$, put $t=\tau {\sigma }^2$; the step restriction gives $|1-t|\le 1$, andThis proves the uniform noise bound for relaxed Landweber iterationTogether with the exact-data convergence,whenever $n=n(\delta )\to \infty$ and $\delta \sqrt{n(\delta )}\to 0$. Equivalently, the a priori regularization parameter choice satisfies $\alpha (\delta )\to 0$ and $\delta /\sqrt{\alpha (\delta )}\to 0$. For example, $n(\delta )=\lfloor {\delta }^{-1}\rfloor$ works as $\delta \to 0$. The sharper familiar bound $\delta \sqrt{n\tau }$ is available when $\tau \Vert A{\Vert }^2\le 1$.

Thus finite iteration followed by an appropriate stopping rule is a convergent regularization of an inverse problem. Taking $n\to \infty$ for fixed noisy data is not the same operation and can amplify the small-singular-value noise without bound. If $A=0$, the zero starting iterate stays zero and is the minimum-norm least-squares solution for every datum; the step restriction is needed only for nonzero $A$.

For a bounded linear operator $K:U\to V$ between Hilbert spaces, the Moore–Penrose inverse of an operator assigns to each admissible datum its unique minimum-norm least-squares solution. Its domain isWrite $f=Ku+h$, where $u\in (\ker K{)}^{\perp }$ and $h\in (\mathrm{ran}K{)}^{\perp }=\ker K^{\ast }$. Then $K^{†}f=u$. Restricting to $(\ker K{)}^{\perp }$ removes the ambiguity among solutions differing by an element of the kernel. The orthogonal decomposition also shows that $K{K}^{†}f$ is the orthogonal projection of $f$ onto $\overline{\mathrm{ran}K}$, and $K^{†}Ku$ is the orthogonal projection of $u$ onto $(\ker K{)}^{\perp }$. The inverse is defined on a dense domain and need not be a bounded linear operator when the range is not closed.

For a bounded linear operator $K:U\to V$ between Hilbert spaces, a least-squares solution of data $f$ is a minimizer of the residual norm:The minimum-norm least-squares solution is the one of smallest $U$-norm among these minimizers. Equivalently it is the unique least-squares solution in $\mathcal{N}(K{)}^{\perp }$. If the equation is consistent, this is the smallest-norm exact solution.

Let $P$ be the orthogonal projection onto $\overline{\mathcal{R}(K)}$. The orthogonal decomposition by a closed subspace givesThe first term has infimum zero by the definition of closure; that infimum is attained exactly if $Pf\in \mathcal{R}(K)$. This proves the least-squares existence criterion:They exist for every datum precisely when the range is closed. Once a solution exists, all solutions differ by elements of the closed null space, and projecting onto its orthogonal complement gives the unique minimum-norm solution.

For a counterexample to existence, take $K:{\ell }^2\to {\ell }^2$, $(Ku{)}_j=u_j/j$, and $f_j=1/j$. The range is dense, so the least residual infimum is zero; finite truncations realize residuals tending to zero. But an exact preimage would have every coordinate equal to one and would not belong to ${\ell }^2$. Thus no least-squares minimizer exists for this datum.

Put $M=\mathcal{N}(K{)}^{\perp }$ for a bounded linear operator between Hilbert spaces. Its restriction $K{|}_M:M\to \mathcal{R}(K)$ is bijective. Define the Moore–Penrose inverse of an operator byThe inverse vanishes on $\mathcal{R}(K{)}^{\perp }$. For each datum in this domain, $K^{†}f$ is the minimum-norm least-squares solution; all least-squares solutions are $K^{†}f+\mathcal{N}(K)$. On their natural domains,The equivalent continuity condition isIf the range is closed, the inverse restriction is bounded by the bounded inverse theorem. Conversely, if $\Vert K^{†}f\Vert \le C\Vert f\Vert$ on its domain and $K{u}_n$ converges, replace $u_n$ by $P_M{u}_n$. The inequality makes these preimages a Cauchy sequence, whose limit maps to the limiting datum; the range is therefore closed. Boundedness here uses the data norm, not a graph norm that would conceal instability.

Use the standard inverse-problem setting of a bounded linear $K:{\ell }^2\to {\ell }^2$ and exact data having a finite-penalty solution. Let $u^{†}$ denote the unique $J$-minimizing exact solution for $J(u)=\Vert u{\Vert }_1+\beta \Vert u{\Vert }_2^2$, $\beta >0$. This is the limiting target of elastic net regularization in Hilbert sequence space; it need not be the Moore–Penrose minimum-norm solution unless additional hypotheses identify them, for example injectivity of $K$.

For each $\alpha >0$, the reconstruction objective is weakly lower semicontinuous, coercive by $\alpha \beta \Vert u{\Vert }_2^2$, and strictly convex. The direct method on the reflexive l2 sequence space gives existence, and strict convexity gives uniqueness. For two data sets, monotonicity of the one-norm subgradients and the quadratic penalty give, with $d=u_1-u_2$ and $g=f_1-f_2$,Completing the square yields $\Vert d\Vert \le \Vert g\Vert /(2\sqrt{2\alpha \beta })$, so each reconstruction map is continuous.

Minimality against $u^{†}$ givesChoose $\alpha \to 0$ and ${\delta }^2/\alpha \to 0$, for example $\alpha =\delta$. This bounds the Hilbert norms, drives the data residual to zero and gives $\mathrm{limsup}J(u_{\alpha }^{\delta })\le J(u^{†})$. Every weakly convergent subsequence has an exact solution as its limit, by boundedness of $K$; weak lower semicontinuity and uniqueness of the penalty minimizer identify that limit with $u^{†}$. Separately applying weak lower semicontinuity to the one-norm givesWeak convergence and convergence of Hilbert norms imply strong convergence by the Radon-Riesz theorem. This proves norm convergence of elastic net regularization:The argument applies to every sequence of noise levels and admissible data, hence gives the usual uniform noise-ball convergence for a fixed admissible exact datum.

For nonlinearity of elastic net reconstruction, zero is the minimizer exactly when $\Vert K^{\ast }{f}^{\delta }{\Vert }_{\infty }\le \alpha$. This includes a neighborhood of zero. If $K\ne 0$, choose $e_j$ with $K{e}_j\ne 0$ and $f^{\delta }=tK{e}_j$ for $t\Vert K{e}_j{\Vert }^2>\alpha$; zero then fails its optimality condition. A linear map vanishing on a neighborhood would vanish everywhere, so the elastic-net reconstruction is nonlinear whenever $K\ne 0$. For $K=I$ the explicit formula is $[R_{\alpha }f{]}_j=\mathrm{sign}(f_j)(|f_j|-\alpha {)}_{+}/(1+2\alpha \beta )$, exhibiting soft thresholding.

A concrete distinction from Moore–Penrose convergence is $K(u)=u_1+2u_2$ (embedded in the first output coordinate of ${\ell }^2$), exact datum one, and $\beta =1$. Its minimum-norm solution is $(1/5,2/5,0,\dots )$, with penalty $4/5$, while the unique penalty-minimizing exact solution is $(0,1/2,0,\dots )$, with penalty $3/4$. For $0<\alpha <2$ the exact-data elastic-net reconstruction is $(0,(2-\alpha )/(4+2\alpha ),0,\dots )$, converging to the latter. Thus it is not a Moore–Penrose-consistent regularization for every noninjective operator under the earlier definition.

The PDF supplies neither a nonzero assumption on $K$ nor a target/domain qualification here. Its unconditional nonlinearity claim is false for $K=0$, whose reconstruction is the zero linear map. The convergence proof above states the finite-penalty exact-solution hypotheses explicitly; it establishes convergence to the penalty-minimizing solution, not an automatically identical minimum-norm least-squares solution.

Use the Hilbert space interpretation of a linear inverse problem, with a bounded linear operator $K:\mathcal{U}\to \mathcal{V}$. A least-squares solution minimizes $\Vert Ku-f{\Vert }_{\mathcal{V}}$ over $u\in \mathcal{U}$. A minimum-norm least-squares solution additionally has the smallest $\mathcal{U}$-norm among all least-squares solutions. When the equation is consistent this is the smallest-norm exact solution.

The least-squares solutions form a nonempty closed affine subspace precisely whenThe closest point theorem in a Hilbert space then gives a unique nearest point to zero, hence a unique minimum-norm least-squares solution. Equivalently, it is the unique least-squares solution in $\mathcal{N}(K{)}^{\perp }$. If $\mathcal{R}(K)$ is closed, this exists for every datum; a nonclosed operator range can leave some data without any least-squares solution.

For example, on the l2 sequence space, take $(Ku{)}_n=u_n/n$ and $f_n=1/n$. The equation would require $u_n=1$ for every $n$, which is not square summable. However, setting the first $N$ coordinates of $u$ to one and the rest to zero givesThus the least-squares infimum is zero but is not attained, and no minimum-norm least-squares solution exists for these data.

The Moore–Penrose inverse of an operator is the linear mapthat assigns each admissible datum its minimum-norm least-squares solution. If $f=y+z$ with $y\in \mathcal{R}(K)$ and $z\in \mathcal{R}(K{)}^{\perp }$, then $K^{†}f$ is the unique $u\in \mathcal{N}(K{)}^{\perp }$ satisfying $Ku=y$. In particular it vanishes on $\mathcal{R}(K{)}^{\perp }$, andThe continuity criterion, with the inherited data norm on its domain, isFor a closed operator range, the restriction of $K$ from $\mathcal{N}(K{)}^{\perp }$ to $\mathcal{R}(K)$ is a bounded bijection between Banach spaces, so its inverse is bounded by the bounded inverse theorem. Conversely, if $K^{†}$ is bounded and $K{u}_n\to y$, replace each $u_n$ by $P_{\mathcal{N}(K{)}^{\perp }}{u}_n=K^{†}K{u}_n$. This is a Cauchy sequence, hence converges to $u$, and $Ku=y$. Thus every limit of range elements remains in the operator range.

Assume $K{u}^{†}=f$, as in the exact-solution setting used for the preceding source condition for quadratic regularization. Let $r=r_{k+1}$, $\eta =f^{\delta }-f$, and $s=u_{\delta }^{(k+1)}-u_{\delta }^{(k)}=-\tau K^{\ast }r$. Then $K(u_{\delta }^{(k+1)}-u^{†})=r+\eta$, and expanding the two squared errors givesThe Cauchy-Schwarz inequality supplies the last line. Thus while the residual remains at least the noise bound,Compatibility of the exact data is essential. If only $f\in \mathcal{D}(K^{†})$ is retained, let $Kx=(x,0)$, $f=(0,1)$, $f^{\delta }=(1/2,1)$, $\delta =1/2$ and $\tau =1$. The minimum-norm least-squares solution is zero, but the first iterate is $1/4$ and its residual norm is $\sqrt{17}/4>\delta$. Its error has increased from zero. Thus the original assertion uses the implicit exact-data assumption from part (c), and is false without it.