For and labels , with squared-norm penalty normalized as , the kernel soft-margin support vector machine solves
Here is the hinge loss building block, and is column of the symmetric kernel matrix. The prediction is
with a fixed class choice, for example , when the score is zero. Other positive rescalings of the penalty change numerical coefficients in the optimality equation but not the zero-coefficient conclusion.
For a finite convex function , its subdifferential at is
For , affine composition preserves the defining convexity inequality: for ,
If , the subgradient inequality gives , proving .
Conversely, let and first suppose . For every , the function is constant along . Applying the subgradient inequality for both signs of shows . Thus is in the span of , or with . Substituting gives for every real . Hence . If , is constant and ; since finite convex on all of has a nonempty subdifferential at , as well. This handles the case in which the suggested orthogonal projection would be undefined. Therefore
This proves the subdifferential under scalar affine composition identity with all its degenerate cases.
Let be a fitted support-vector-machine margin. The subdifferential of the hinge loss yields choices
Applying the affine-composition identity and the subdifferential sum rule to the convex objective, its optimum satisfies
The second equation is the intercept condition. Since is invertible, multiplying the first by gives the kernel support-vector coefficient from hinge activity formula
In particular Invertibility is needed for this statement about the chosen coefficients: for with two positive labels, and give zero objective and both margins equal to two, while both coefficients are nonzero. A singular kernel matrix allows coefficient changes in the null space without changing the fitted function.
At prediction time, the sum can omit all zero coefficients, requiring only the number of evaluations of the positive-definite kernel corresponding to retained support vectors rather than all . This need not remove the cost of forming or solving with a dense training kernel matrix. Every misclassified point has , including a score-zero tie assigned to the opposite class, so and . Thus many classification errors imply many active support vectors and little prediction-time sparsity benefit. Correctly classified points inside the unit margin can also have nonzero coefficients; a point exactly on the margin need not have a nonzero coefficient.
For a bounded linear operator between Hilbert spaces, a least-squares solution of data is a minimizer of the residual norm:
The minimum-norm least-squares solution is the one of smallest -norm among these minimizers. Equivalently it is the unique least-squares solution in . If the equation is consistent, this is the smallest-norm exact solution.
Let be the orthogonal projection onto . The orthogonal decomposition by a closed subspace gives
The first term has infimum zero by the definition of closure; that infimum is attained exactly if . This proves the least-squares existence criterion:
They exist for every datum precisely when the range is closed. Once a solution exists, all solutions differ by elements of the closed null space, and projecting onto its orthogonal complement gives the unique minimum-norm solution.
For a counterexample to existence, take , , and . The range is dense, so the least residual infimum is zero; finite truncations realize residuals tending to zero. But an exact preimage would have every coordinate equal to one and would not belong to . Thus no least-squares minimizer exists for this datum.
The temporal integration by parts terms are
The terminal states are free and the objective has no explicit terminal-velocity dependence. Therefore
The factor two follows from the displayed objective without a prefactor. The initial scalar field perturbation is fixed, so and there is no independently prescribed adjoint-scalar condition at . Its value there is obtained by backward integration. Independent variation of the initial velocity state gives .
Variation of the candidate in the energy-augmented functional gives the fixed-energy initial-condition optimality condition. For the usual divergence-free no-slip control space , write for its orthogonal projection, the Leray-Helmholtz projection in the usual incompressible velocity space. This is the standard formal control condition when pressure is determined by incompressible flow and normal momentum. If the independently imposed direct pressure flux is retained as an additional constraint, the admissible control variations must also satisfy its compatibility conditions along the trajectory; they need not fill the usual space . The standard projection formula alone does not prove stationarity for that more restricted problem. Then
For the natural solenoidal adjoint space, . Equivalently, its component tangent to the sphere in a normed vector space of fixed kinetic energy vanishes:
Here is needed for the sphere in a normed vector space to be regular and for division by . The multiplier is real and can have either sign; normalizing the initial adjoint with a prescribed positive sign is not a general necessary condition for minimization. If , the only feasible initial velocity is zero and this tangent formula for the sphere in a normed vector space is inapplicable. First-order stationarity alone also allows maxima or saddles; a local minimum requires the appropriate nonnegative constrained second variation.
The spatial boundary terms vanish with periodic adjoints in , homogeneous no-slip boundary conditions at , and there. To see this, velocity variations vanish at the wall while their normal derivatives need not, so the viscous boundary term forces the adjoint velocity to vanish. Scalar variations have zero normal derivative but free values, forcing the adjoint scalar's normal derivative to vanish. Pressure variation gives adjoint incompressible flow; no independent terminal or initial datum is assigned to . Its additive time-dependent constant may be fixed by a zero spatial mean.
In particular, a homogeneous Neumann condition for the direct pressure does not by variational transposition require . For a smooth adjoint solution, the wall-normal adjoint momentum equation instead supplies
at the flat walls, because the direct scalar normal derivative is zero there. A zero adjoint-pressure derivative would be an additional compatibility choice, valid only when this right-hand side vanishes.
There are also literal direct-data compatibility qualifications in the printed setup. The initial total scalar field is , whose wall derivative is . It is extraordinarily small but mathematically not zero. Thus exact initial scalar data and exact zero wall flux are incompatible for a classical solution smooth at . One may use a parabolic mild solution of an abstract Cauchy problem with the boundary condition enforced for , or replace the initial profile by an exactly compatible smooth one; these are conventions, not an equality . The reference scalar field is also not a stationary profile of the diffusion equation at finite : the full term must be retained.
The pressure compatibility at a no-slip wall is similarly important. Direct normal momentum gives . The separately imposed zero pressure derivative requires this right side to vanish. It is not automatic for arbitrary fixed-energy initial velocities. For example, the divergence-free no-slip field generated by the periodic stream function with has , and , while . With adjusted to any positive energy, no classical solution smooth up to the initial wall can satisfy that extra pressure condition. The displayed adjoints are the formal necessary equations along admissible smooth trajectories; a well-posed physical formulation normally determines pressure from incompressible flow and normal momentum, rather than imposing independent homogeneous pressure flux for every control.
Fix the ordering of the orthonormal basis and write , taking indices from . The linear N-term approximation retains the first coefficients:
Its index set is independent of , so the map is an orthogonal projection and is linear.
For the nonlinear N-term approximation, let be any set of indices with greatest , resolving ties arbitrarily and padding with zero coefficients if necessary. Since , such a selection exists. Then
The Parseval identity shows that for any fixed index set the displayed coefficients minimize the Hilbert space error. Choosing the largest squared magnitudes minimizes the omitted sum over every set of at most indices. This is the best N-term approximation; dependence of the selected indices on makes the operation nonlinear.
The orthogonal projections of the Gaussian noise onto the column space of and its orthogonal complement are independent. Consequently and the residual sum of squares are independent, and
Thus under the alternative,
This is the noncentral t-distribution. The rejection region is fixed by the design, and the chosen level, while this probability distribution depends on only through . Therefore the statistical power does not depend on the other regression coefficients, with the design and fixed.
Decompose into its components parallel and orthogonal to the unit vector . Its nearest point in the hyperplane is the orthogonal component, since the two components' squared distances add. Thus
This is a unique orthogonal projection; for the hyperplane is the singleton zero and the formula still applies.
The only noninvertible case is . Then is the orthogonal projection onto , so
These dimensions agree with the rank-nullity theorem.
Use the usual unital-subalgebra convention . The spectral theorem for a commutative operator algebra states that the character space , equipped with the Gelfand topology, is a compact Hausdorff space, and there is a unique regular projection-valued measure on its Borel sigma-algebra satisfying
where is the Gelfand transform. The map is an isometric unital star-isomorphism from onto by the Commutative Gelfand--Naimark theorem.
More explicitly, every is an orthogonal projection, , , and for disjoint Borel sets ,
with convergence in the norm of the Hilbert space. Regularity means that each scalar spectral measure is a regular Borel measure. The integral identity then implies . No separability assumption on is needed.
If the spectrum has one point, the preceding part gives , and the linear span of any nonzero vector is a closed proper invariant subspace because .
Otherwise choose distinct points and disjoint nonempty relatively open neighborhoods . The open-set result gives and , while
Thus . Set . As the range of an orthogonal projection, is closed, and the preceding observations give . The Borel functional calculus for a normal operator gives , so
In fact both and its orthogonal complement are invariant.
Write . Every least-squares solution is for some , while . Decompose using orthogonal projections. Then the Pythagorean identity gives
The first term is independent of , and the second vanishes at exactly one point. Hence the unique nearest least-squares solution is
Geometrically this is the orthogonal projection of onto the closed affine subspace of least-squares solutions.
The great circle containing the pointing and celestial pole is the intersection of the unit sphere with the meridian plane . The orthogonal projection of onto that plane, normalized to unit length, is
Within the local meridian chart , its declination is therefore . Outside that chart, the same construction gives an oriented great circle parameter; the foot can lie on the opposite right-ascension half of the meridian, and need not be its ordinary declination.
To verify the right angle geometrically, the tangent to the great circle from toward is proportional to . Since , that tangent has only a component. The tangent from toward the pole, , lies in . Their orthogonality proves the claimed spherical right angle. Thus within the local meridian chart, is the declination of the perpendicular foot on the pointing meridian.
There are antipodal perpendicular feet on the complete great circle. On the local meridian chart, the telescope field selects the foot near the pointing; the cotangent equation alone determines only modulo . At a coincident foot or pole, the corresponding angle is understood by continuity rather than by a nonzero tangent vector.
Let be the orthogonal projection onto . The dilation property and the orthonormal basis property give the orthonormal basis of . The periodized nonnegative function
has period one and , by Tonelli theorem. For a function with compact support contained in a bounded interval , Cauchy-Schwarz inequality and the Parseval identity give
as . The last step uses absolute continuity of the integral of a locally function, since the interval's length tends to zero. If , then
Functions with compact support are dense in , so . Consequently
This proof needs neither the density axiom nor any compact support assumption on the scaling function.
At zero, the partial products are . Their assumed convergence and force and . Uniform convergence on compact sets makes continuous. The identity with gives with squared norm . Define by the inverse Fourier transform in . The Plancherel theorem and the other given integral identities give
Define as the closed span of . These functions form an orthonormal basis of , and dyadic dilation gives the scale axiom. Shifting the infinite product gives
so the finite Fourier series of gives a scaling refinement equation and . The coarse-scale projection argument of part 1(b) proves the trivial-intersection axiom.
For density, take a function with bounded Fourier support. The MRA projection Fourier identity gives, for large ,
Uniform convergence of to one on that support proves the limit. Since is an orthogonal projection, . Such functions are dense in , giving
Thus all the multiresolution analysis axioms hold. Under the strong convergence and orthogonality assumptions supplied here, the additional nonvanishing condition is not needed in this last verification; it is useful when establishing those assumptions from a filter.
Write . A linear N-term approximation fixes the indices independently of , normally the first in a prescribed ordering:
A best N-term approximation chooses the indices using : retain coefficients of largest absolute value, resolving ties arbitrarily, and set . The Parseval identity shows why this choice is optimal:
For any fixed index set, the orthogonal projection coefficients minimize the error; optimizing the set then means discarding the smallest squared coefficients. Thus “linear” requires a specified ordering, while “nonlinear” refers to the data-dependent selection.
The starting point of a conforming finite element method is a weak formulation in a Hilbert space . For example, a homogeneous Dirichlet boundary condition on a bounded Lipschitz domain gives . One seeks
where is a bounded bilinear form, , and is a bounded linear functional. A conforming finite element space is finite dimensional and is typically built from functions that are polynomial on each element of a finite element mesh. With a basis , the Galerkin method imposes the same identity only for , giving
The stiffness matrix is sparse when basis supports overlap only locally. Local element integrals assemble its entries; the mass matrix represents an term. This separates the choice of trial functions from the variational principle determining their coefficients.
The fundamental well-posedness theorem is the Lax-Milgram theorem: if for some , there is a unique solution for each , with . The same theorem on gives a unique discrete solution. The constants must be independent of the mesh for uniform stability of a numerical method. Boundary conditions are built into the trial space when they are essential constraints; flux conditions arise through boundary terms in integration by parts and are natural constraints in the weak formulation.
When is symmetric and a coercive bilinear form, the Ritz method minimizes
over . Its first variation is exactly the Galerkin method, so Ritz minimization and Galerkin testing coincide for symmetric coercive problems. The energy norm turns the approximation into an orthogonal projection. Indeed, subtracting the exact and discrete weak equations gives Galerkin orthogonality, . For every the Pythagorean identity becomes
so
This exact best-approximation property is the central advantage of the Ritz method.
Symmetry is unnecessary for the Galerkin method. For any bounded, coercive form, the Céa lemma gives
To prove it, put . Galerkin orthogonality yields , so coercivity and boundedness give . Divide when and take the infimum. Thus approximation capability plus uniform coercivity gives convergence. In contrast, minimizing for a nonsymmetric form differentiates its symmetric part and generally does not solve the original weak equation.
A concrete symmetric example is the one-dimensional reaction-diffusion problem from Question 5. Continuous piecewise-linear hat functions give the diffusion stiffness matrix with diagonal and neighbouring entries , and the mass matrix with diagonal and neighbouring entries . Therefore and . Its positive-definite matrix is the coordinate form of the continuous energy. The exact smooth solution makes this an explicit test of the method, not just an abstract existence result.
For a nonsymmetric example, take positive , constant real , and
Its form is . Since , it has coercivity constant in the full norm, although it is not symmetric when . The Galerkin method remains well posed. On the uniform hat basis, , where , and . The skew-symmetric matrix cancels from , proving nonsingularity. A Ritz energy would omit precisely this advection contribution. Small may nevertheless make the approximation constant large and produce poorly resolved layers; algebraic solvability is not an accuracy guarantee.
Quantitative convergence uses a finite element interpolation estimate. On a shape-regular mesh, for piecewise-linear conforming elements and in the usual low-dimensional setting,
The Céa lemma therefore gives an energy error. The Aubin–Nitsche duality argument improves the L2 norm error when the dual elliptic problem has regularity: solve and use
Consequently
with constants depending on the solution and regularity bounds. The second estimate is conditional on dual regularity; corners or insufficient data regularity can reduce the rate.
The same Galerkin method handles evolution equations. For the homogeneous heat equation, the semidiscrete finite element heat equation is . Because is a positive-definite matrix and is a positive semidefinite matrix,
The decreasing quadratic form is exactly the L2 norm squared of the finite-element function. This separates spatial stability of a numerical method from the subsequent time integrator, just as in the Schrödinger equation example. Conformity, coercivity and approximation estimates together explain convergence.
Give the uniform probability measure and let be the indicator function of the edge set. For partitions and , define the energy
Because conditional expectation is an orthogonal projection in L2 space, .
Suppose the pair of partitions is not -regular. For each irregular choose witnesses and with
Refine each by all sets and each by all sets . If the old partitions have cells, the refined ones have at most cells.
The Pythagorean theorem for the two nested conditional-expectation projections gives
Inside an irregular , the witness rectangle occupies at least an proportion and the mean of the displayed difference over it has magnitude greater than . The Cauchy-Schwarz inequality therefore gives an energy gain greater than
from that pair. Since irregular pairs have total weight greater than , the complete refinement raises the energy by more than .
Energy is at most one, so after at most refinements the process stops at an -regular pair of partitions. Iterating the cell-count bounds and from a bounded number of times produces a finite independent of . This proves the Bipartite Szemerédi regularity lemma.
Let be its positive-negative decomposition. Since , the positive and negative parts have equal trace, namely . For every positive contraction ,
Here both products have nonnegative trace, and bounds the first term. Equality is attained by the orthogonal projection onto the positive spectral subspace of , since and . Thus the variational characterization of trace distance is
Let be the orthogonal projection onto the good linear subspace , and let be a prepared unit vector with . For , define
The amplitude amplification theorem states that, if the reflection operators about the initial state and the good subspace can be implemented, the iteration
satisfies
To prove it, the two reflections preserve the linear span of . In this ordered orthonormal basis, direct multiplication gives
Multiplying this rotation matrix by replaces by , proving the formula by mathematical induction.
For known , choose to be a nearest nonnegative integer to . Then , so measurement finds the good subspace with probability at least , using iterations. For , measurement of the initial state already has constant success probability. If success is certain; if these reflections cannot create any good component. If , the initial-state reflection is , implemented using quantum state preparation and its inverse quantum circuit.
Figure 1. . Each iteration adds the angle in the good-bad plane. The probability rises near one and then falls again, so the stopping time matters. Details.
For an eigenvector with eigenvalue ,
By part (a), every vector orthogonal to is a linear combination of such eigenvectors, so vanishes on . On the remaining eigenspace,
The orthogonal projection onto is . It follows that
Put . For , the identity reads
and therefore
Because is the number of common neighbours, these two constants prove that is strongly regular. This is the converse half of the three-eigenvalue characterization of a connected strongly regular graph.