Argmax 2026-11-05
The argmax of a function is the set of inputs where its largest value is attained:It records global maximizers, whereas records the value. For , the argmax on is and the maximum value is zero. If is constant, every input belongs to the argmax.
The argmax can be empty: on has supremum one but never attains it. A continuous function on a nonempty compact set does attain its maximum by the extreme value theorem. Writing a single input as an argmax therefore implicitly assumes uniqueness or specifies a rule for choosing between ties. This distinction matters when an optimization problem is used to define an estimator.
Bolzano-Weierstrass theorem Created 2026-09-24 Updated 2026-10-14
Every bounded sequence of real numbers contains a subsequence with a finite limit of a sequence. If for every positive integer , there are integers and a real number such thatThe increasing indices matter: a subsequence retains the original order and selects terms arbitrarily far along the original sequence. The theorem guarantees the existence of at least one such selection; it does not claim that the entire sequence converges or that the subsequential limit is unique.[ref]
For example, the bounded sequence has two distinct subsequential limits:The even terms form a convergent sequence, and so do the odd terms. The full sequence cannot have a limit of a sequence because its two subsequences have different limits. Its limit superior is and its limit inferior is .
This is useful when a construction produces many candidate points but does not immediately produce a convergent sequence. For a continuous function on a nonempty closed bounded feasible region in a finite-dimensional Euclidean space, take a sequence whose objective values approach the infimum. A convergent subsequence supplies a candidate point; a closed subset retains that point, and continuity passes the objective values to the limit of a sequence. This is the mechanism behind attainment in the extreme value theorem. The diagonal subsequence argument extends the real-sequence theorem to finite-dimensional Euclidean space.
Suppose a topological space admits an unbounded continuous function to . Set and choose so that increases to infinity. There is a continuous function with : interpolate linearly between successive prescribed values and extend constantly below . Thenis continuous and takes values strictly between and , while its values on the even and odd chosen sequences tend to and . Thus neither bound is attained. In particular, unbounded continuous witnesses to noncompactness of a real subspace give this stronger failure of the extreme value theorem for every noncompact real subspace.
Closed real interval Created 2026-09-24 Updated 2026-10-14
A bounded closed interval includes both of its endpoints:If , it consists of one point. Its length is . In contrast, the open interval excludes both endpoints; half-open real intervals include just one.
Including the endpoints has a useful consequence for limits: a convergent sequence whose terms lie in has its limit of a sequence in too. Finite endpoints also make this a bounded set, so it is a compact set by the Heine-Borel theorem. These two properties support both the nested interval theorem and the extreme value theorem.
Compact space Created 2026-09-24 Updated 2026-10-14
A topological space is compact when every open cover has a finite subcover: whenever open sets satisfy , some finite selection satisfies . A subset of an ambient topological space is compact when its subspace topology is compact; equivalently, every open cover of by ambient open sets has a finite subcover. The word every matters: finding one finite subcover of one open cover is not enough to establish compactness. [ref]
The closed interval is compact by the Heine-Borel theorem. For a concrete open cover, take the three open intervals , , and . Their union is , so they cover every point of , including both endpoints. The overlaps have lengths and , leaving no gaps. Each member is necessary for this particular open cover: removing them respectively leaves , , or uncovered.
Above, hollow markers denote the excluded endpoints of the covering open intervals, while the solid endpoints belong to the covered closed interval. Below, each displayed open interval is one member of an infinite open cover of ; finitely many rows always leave points near zero uncovered. The rows are vertically separated for readability and all refer to the same real line.
The open interval is a bounded set but is not compact: the open setsgive an open cover of , since every has for sufficiently large . These are also open sets for the subspace topology on . Any nonempty finite selection has a largest index , so its union is just and misses the point , which still belongs to . Thus no finite subcover exists, however many members we select. The same family is not an open cover of , since it misses both endpoints.
In a metric space, compactness is equivalent to sequential compactness: every sequence in the subset has a convergent subsequence whose limit of a sequence belongs to that subset. This equivalence does not extend to all topological spaces. For example, for has limit of a sequence zero in . The same sequence lies in , but every subsequence still has limit of a sequence zero on the real line, so no subsequence converges to a point of . Merely approaching a boundary is harmless when the required point remains available. [ref]
https://ocw.mit.edu/courses/18-100b-analysis-i-fall-2010/b7780b2a0629359b11c5de998c72a147_MIT18_100BF10_compactness.pdf. MIT Analysis I, Compactness vs. Sequential Compactness, Theorem 1 and its metric-space proof
The same sequence approaches an included endpoint in a closed interval and a missing endpoint in an open interval
. Details. Both panels show the sequence for approaching zero. In the closed interval, zero is an allowed limit of a sequence; in the open interval, the hollow marker identifies the missing point. This single sequence disproves sequential compactness of ; compactness of still requires control of every open cover or every sequence, supplied by the Heine-Borel theorem.
An infinite set with the discrete topology similarly fails compactness, using the open cover by all its singletons. Every finite set is compact in any topology: select one covering open set for each point. The empty set is compact because the empty selection is a finite subcover. This open cover definition does not require the topological space to be Hausdorff; in algebraic geometry it is often called quasi-compactness.
The beauty of compactness is that information valid around each point can often be controlled using finitely many regions. A finite subcover lets us combine finitely many local bounds, or choose the smallest of finitely many positive tolerances, to obtain control across the whole compact space. This is why compactness repeatedly turns approximation into existence and why it is so useful in optimization through the extreme value theorem.
Every closed set in a compact space is compact, and a continuous image of a compact space is compact. The latter proof pulls an open cover back by inverse images, selects a finite subcover, and maps forward. This makes compactness useful for proving existence: the extreme value theorem guarantees that a continuous function from a nonempty compact space to the real line attains both a minimum and a maximum. For example, on has maximum at and minimum zero at its endpoints. On its infimum is still zero but is never attained. [ref]
In a metric space , let the nonempty compact set lie in the open set . If is nonempty, the distance to a set is a continuous function of : the triangle inequality gives . Since is a closed set, this distance is strictly positive on . The extreme value theorem attains its minimum on , soIf is empty, the extended-real convention gives . This result provides a single positive neighbourhood radius that stays inside at every point of . Compactness is essential: in the real line has zero distance between two sets from , although each point of has positive distance to a set from that complement.
Let be a nonempty compact space and let be continuous and nowhere zero. On , define . Multiplication by is an isometry onto with its ordinary uniform norm, because division by the continuous nonvanishing provides the inverse. Thus the weighted space is a Banach space. Equivalently, the extreme value theorem gives , and . A weighted Cauchy sequence is uniformly Cauchy, has a continuous uniform limit, and the upper bound gives convergence also in the weighted norm.
If is jointly continuous on a compact rectangle , then the displayed parameter integral is continuous on . For , pointwise continuity gives . The extreme value theorem bounds by a constant on the rectangle, and is integrable on the finite interval . The dominated convergence theorem gives .
Continuous function Created 2026-09-24 Updated 2026-10-14
A continuous function sends sufficiently nearby function inputs to nearby function outputs. For a real-valued function and a point that is an accumulation point of , this meansThe approached value and the assigned value must agree. Continuity on means continuity at every point of the domain of a function. At an endpoint of an interval, only approaches inside that domain of a function count. The epsilon-delta definition of continuity below also applies at an isolated point, where continuity holds automatically.[ref]
Calculus Volume 1, §2.4 Continuity. §2.4, Continuity at a Point: definition; Continuity over an Interval: one-sided endpoint conditions
For the polynomial function , approaching makes the function output approach . The continuity bounds for the square function turn this observation into an explicit guarantee: implies . This is useful whenever a small error in an function input must produce a controlled error in an function output. The required input tolerance can depend on the point and on the chosen output tolerance.
The absolute value function is also a continuous function, including at zero: . But its left and right derivatives there are and , so it is not a differentiable function at zero. Continuity controls values without requiring a tangent line. Conversely, a differentiable function is continuous wherever it is differentiable: if , the difference tends to zero with .
Several useful consequences reuse this one local idea. Addition, multiplication and function composition preserve continuity; a quotient of functions is continuous where its denominator is nonzero. On an interval, the intermediate value theorem turns continuity into a guarantee that no intermediate function output is skipped. On a compact set, the extreme value theorem guarantees attained maximum and minimum values. Those extra hypotheses matter: continuity alone on an unbounded domain of a function does not provide a global bound or an attained global maximum.
A rational function such as is continuous on its domain of a function . Zero is excluded from that domain of a function; the missing point prevents a continuous extension there, but it does not make the existing map discontinuous at one of its actual inputs. Always state the domain of a function before judging continuity.
Continuous image of a compact space Created 2026-09-24 Updated 2026-10-14
Let be topological spaces, let be compact, and let be a continuous map. Then its image is compact with the subspace topology inherited from . More generally, a continuous map sends every compact subset to a compact subset , by applying the same statement to its restriction to . Neither nor needs to be Hausdorff.[ref]
For example, the polynomial function sends the compact closed interval onto . The open intervals and give an open cover of the image. Their inverse images in the domain areBoth are open sets in the subspace topology of , even though neither is an open set in the ambient real line. They cover the domain, so the corresponding two target open sets cover the image.
Matching colors identify the target open sets and their inverse images. Filled endpoints are included and hollow endpoints excluded. This particular open cover illustrates the mechanism; compactness requires a finite subcover for every open cover.
For the general proof, write an arbitrary open cover of as with each an open set in . By continuity, the inverse images form an open cover of . Its compactness provides indices such thatThis gives the required finite subcover of . The argument uses inverse images because continuous maps need not send open sets to open sets. For the empty set, the empty selection suffices.
The image of a compact Hausdorff space under a continuous map need not itself be Hausdorff. Take with the discrete topology and with the Sierpiński space topology . The identity map is a continuous bijection, since the inverse image of every target open set is an open set in . Its image is compact, but the two points of have no disjoint open neighborhoods. If the target is Hausdorff, its subspace topology does make Hausdorff, and the compact subset of a Hausdorff space theorem also makes closed in .
The practical payoff is existence: the extreme value theorem uses the compact image of a nonempty compact space under a real-valued continuous function to obtain an attained minimum and maximum. When is also a bijection onto a Hausdorff target, the compact-to-Hausdorff continuous bijection theorem supplies continuity of its inverse without a separate formula.
Convex optimization Created 2026-09-24 Updated 2026-10-13
Convex optimization is a minimization problem with a convex function as its objective function and a convex set of feasible points. For a finite-valued , the problem is to find with for every . The displayed infimum describes its value even when no minimizing point exists. The two convexity requirements concern different objects: the allowed decisions and the scores assigned to them.
A convex set contains the whole line segment between any two of its points:A convex function lies below the interpolation of its endpoint values:Thus mixing two allowed decisions preserves feasibility, and the mixed score is no worse than the same mixture of endpoint scores. The objective function may have flat regions or corners. For a strictly convex function, the second inequality is strict when and .
A useful constraint representation iswhere are convex functions on a common convex set , and the equalities are affine functions. Each inequality defines a convex sublevel set; their intersection with and the equality set is a convex set. An equality involving a general convex function need not preserve this geometry: allows and but excludes their midpoint . See Boyd and Vandenberghe, Convex Optimization (2004), §4.2.1 for the standard problem form.
Why local information can certify a global answer. Every local minimum of a convex optimization problem is a global minimum, including minima on the boundary. If a feasible point had , points would remain feasible and have value strictly below for every . Taking arbitrarily small contradicts a local minimum at . This argument explains the special power of convexity: a better distant decision would reveal an improving direction immediately nearby.
If a minimizer exists and the objective function is strictly convex on , it is unique. Two distinct minimizers of value would have a feasible midpoint with value strictly below . Existence is a separate question: the exponential function is strictly convex on , but its infimum is approached only as . Even a bounded convex set need not suffice: on also has unattained infimum . A continuous function on a nonempty compact set attains its minimum by the extreme value theorem.
Nor does strict convexity imply differentiability. For , the squared term gives the strict inequality while the triangle inequality gives the weak inequality for the absolute value. At zero the left and right derivatives are and , so no derivative exists there, although zero is the unique minimizer. Such objectives require subgradients or proximal operators rather than an ordinary gradient at every point.
For a differentiable convex function on an open convex set containing , the constrained first-order condition isIndeed, the subgradient inequality makes this sufficient; taking the one-sided derivative along feasible line segments makes it necessary. At an interior minimizer, both directions are feasible and the gradient must vanish. At a boundary minimizer it can be nonzero: it only needs to make every feasible direction nondecreasing. See Boyd and Vandenberghe, §§3.1.3 and 4.2.3.
A fully computed nearest-point problem. Project onto the half-space by minimizing half the squared Euclidean distance:The Hessian matrix is the identity matrix, so the objective function is strictly convex. The unconstrained minimizer violates the inequality. With the convention , the optimization Lagrangian is , where the Lagrange multiplier satisfies . Its Karush-Kuhn-Tucker conditions areThe last two equations give , . The choice is infeasible, so complementary slackness forces . Consequently , giving , and . The Euclidean distance is , rather than the objective function value . The gradient at the answer balances ; the displacement from to is a normal vector to the boundary.
There is an exact certificate independent of numerical search. Completing squares givesFor every feasible point, implies . Thus every feasible objective is at least , and attains this bound. More generally the Lagrangian dual problem has objective for . Its maximum is also , illustrating strong duality through matching computed bounds. This makes the example a model for Euclidean projection onto a convex set and the projection step in projected gradient descent.
What the multiplier conditions require. For differentiable convex functions on an open convex set and equalities defined by affine functions, the KKT conditions combine feasibility, , complementary slackness , and stationaritywith unrestricted equality Lagrange multipliers . They suffice to certify a global minimum without an additional regularity assumption: the optimization Lagrangian is convex, stationarity minimizes it, and feasibility with complementary slackness gives the same certificate as above. To guarantee their necessity at an attained finite optimum, a regularity hypothesis suffices: Slater's condition requires some with and . The projection example has the strictly feasible point . See Boyd and Vandenberghe, §§5.2.3 and 5.5.3. This hypothesis guarantees suitable multipliers, not existence of a primal minimizer.
Without regularity, necessity can fail even for convex functions: minimize subject to . The sole feasible point is optimal, but stationarity would require . No Lagrange multiplier works. There is no strictly feasible point, and the gradient of the constraint vanishes at the answer.
The nonconvex contrast. The polynomial has derivative and second derivative . Numerically solving the cubic gives stationary points , , and . Their objective values are respectively , , and , with second derivatives , , and . Hence both outer points are local minima, and the middle point is a local maximum. Since at both ends, its global minimum exists and must be one of the stationary minima: it is the left one. A method stopping at the right local minimum misses a better value by approximately .
A strictly convex quadratic has one minimum; a tilted quartic has two local minima with different values
. Details. The practical appeal of convex optimization is the ability to check an answer globally using geometry and matching Lagrangian duality bounds. The projection certificate certifies every feasible alternative at once; the tilted quartic shows why a zero gradient alone cannot do that for general nonlinear optimization.
Historically, an early landmark is Kuhn and Tucker's [ref]
https://digicoll.lib.berkeley.edu/record/112776/files/math_s2_article-35.pdf. Nonlinear Programming, Second Berkeley Symposium (1951), pp. 481–492
. Its §3 explicitly states a regularity hypothesis for inequality-constrained Lagrange multipliers; its discussion of saddle points explains how concave functions support global conclusions for maximization. This documents an early formulation of the multiplier approach; it does not establish sole discovery of today's KKT conditions or of convex optimization as a whole.Suppose and its partial derivative are jointly continuous on a compact rectangle . Then is continuously differentiable in the interior of , and the displayed formula holds, with one-sided derivatives at parameter endpoints where the corresponding partial derivative is given.
To prove it, the extreme value theorem bounds by . For each fixed , the mean value theorem bounds the difference quotient by . It converges pointwise to , and the constant is integrable over . Dominated convergence gives the derivative formula. Apply continuity of parameter integrals on compact rectangles to to obtain continuity of . The proof works locally on unbounded parameter intervals as well.
Empty and open feasible regions 2026-10-14
Adding an optimization constraint removes possibilities: if it permits a set , the new feasible region is . It can leave unchanged when all its members already satisfy the added restriction. For a fixed objective function, restricting the feasible set cannot lower a minimization problem's infimum. In the equality slice of a feasible triangle, it raises the attained value from to .
Instead add to the feasible triangle with a squared-distance objective. No point can have both and . The resulting feasible set is the empty set, so there is no feasible point and no optimizing decision. Under the extended-value convention for a minimization problem, its infimum is .[ref]
A nonempty feasible set can also lack an optimizing decision. With domain , minimize subject to the strict inequality . The feasible region is an open set. Its infimum is , approached by the feasible points , but is excluded. Every feasible point is beaten by , so no minimum is attained. Replacing by includes the missing optimum. Open feasible regions do not always prevent attainment: minimizing on the same open set attains value at . A useful sufficient condition is instead the extreme value theorem: a continuous objective function on a nonempty compact set attains its minimum.
If are continuous functions, , and is a nondecreasing monotonic function, there is satisfying the identity in the title. It expresses the weighted integral as a mixture of the two endpoint values of , with the split occurring in the accumulated weight.
Put and . The monotonicity of the Riemann integral gives . The function is continuous; indeed, by the extreme value theorem and the triangle inequality for the Riemann integral. Thereforeis continuous, with and . The intermediate value theorem gives . If the endpoint values coincide, the bounds force to have that common value and an endpoint works. The proof covers zero total weight and a degenerate interval without division.
Extreme value theorem Created 2026-09-24 Updated 2026-10-14
A real-valued function that is continuous on a nonempty compact space attains both a global minimum and a global maximum. In particular, for a continuous function on a nonempty closed interval, there are points such thatThe important word is “attains”: the best values occur at actual allowed inputs, rather than merely being approached. The interval version is [ref].
For a worked example, take on . Its global minimum is , and its global maximum is . Nonnegativity gives the first answer; on the specified closed interval gives the second. The endpoint is essential: looking only for a zero derivative would find the minimum and miss the maximum.
An attained minimum and maximum on a closed interval, compared with excluded endpoints and a discontinuous endpoint value
. Details. For comparison, the continuous function on the open interval has infimum and supremum , but neither is a value of . Even a closed domain is insufficient without continuity: define for , and . On , its supremum is , approached by for , but every function value is strictly less than . This endpoint change makes discontinuous, preventing attainment. On the unbounded real line, the continuous function has neither a global minimum nor a global maximum. These examples show what can fail when a hypothesis is dropped; functions outside the hypotheses may still have extrema.
The extreme value theorem is an existence tool for optimization: first establish that a continuous objective function on a nonempty compact feasible set has a best value, then investigate where it occurs. It neither locates the optimizing point nor promises uniqueness or differentiability. For example, the absolute value function on has its global minimum at zero although no derivative exists there; a constant function has every input as a minimizer. This role in optimization is explained in University of Toronto MAT237 notes, section 1.4: compactness and the extreme value theorem.
A short proof outline identifies the mechanism. The continuous image of a compact space is compact; a nonempty compact subset of the real line contains its infimum and supremum. For a closed interval, one can instead choose inputs whose outputs approach the supremum, extract a convergent subsequence by the Bolzano-Weierstrass theorem, and use continuity to make its limiting output equal that supremum. The closed interval retains the limiting input. Repeating the argument for the infimum supplies the minimum.
Feasible point 2026-10-13
A feasible point satisfies every optimization constraint of an optimization problem. For the budget constraint example, the feasible set is exactlyIt is the line segment from to , a compact set and a convex set. The point is feasible, with product ; has a larger product but violates the budget constraint. Feasibility answers whether a decision is allowed; optimality asks whether any allowed decision is better. The extreme value theorem guarantees that this continuous objective function attains its maximum on this nonempty compact set.
Eliminating gives a one-variable objective function:By completing the square, , with equality exactly at . Alternatively its derivative is , which vanishes at ; the endpoint values are both zero. Completing the square proves global optimality directly, whereas the first-order optimality condition only identifies a candidate. It also quantifies the cost of imbalance: moving to with loses exactly units of product. The arithmetic-geometric mean inequality gives another short certificate: , with equality exactly when .
For a transportation problem with capacity inequalities, feasibility requires nonnegative shipments, row totals no greater than factory capacities and column totals equal to shop demands. Feasibility alone does not prove minimal cost: a matching transportation dual certificate with capacity inequalities provides an optimality proof. The feasible set can be empty, and a finite infimum need not be attained even when the set is nonempty.
Finite-valued infimal convolution Created 2026-10-07 Updated 2026-10-14
If two nonnegative finite-valued convex functions are defined on all of a finite-dimensional space, their infimal convolution is finite, convex and continuous everywhere. Convex perturbation duality then gives strong duality and dual attainment at every argument. It does not require or imply attainment of the primal decomposition infimum.
More precisely, let be finite everywhere and write . Nonnegativity supplies a lower bound, while the two trial splits and giveMixing nearly optimal splits proves convexity, and continuity of a convex function applies because the effective domain is the entire finite-dimensional vector space. These statements concern the optimized value, rather than existence of an optimal split. For the continuity theorem and dual attainment under the equality constraint , see Bertsekas, Convex Optimization Theory: A Summary, Propositions 1.3.11 and 5.3.3(a). The equality has a feasible point in the whole space for every , and the displayed bound makes its optimal value finite.
A finite value without any optimal split. On , take the convex functions and . Both are nonnegative and finite everywhere, butfor every . The splits have costs , yet the exponential function is strictly positive at every finite . No split attains the infimum. The value function is even constant and differentiable; regularity of the value function does not prevent optimizing splits from escaping to infinity. For example, gives costs approximately , and .
A sufficient extra condition for attainment at a fixed is that be a coercive function, meaning it tends to as . Under the finite-valued assumptions it is a continuous function, so a sufficiently large closed ball contains a global minimum by the extreme value theorem. If this cost is also a strictly convex function, the optimal split is unique. The quadratic example above satisfies both conditions; the exponential function example does not satisfy coercivity.
Nonnegativity is sufficient, not necessary: finite lower bounds , would give by the same argument. Removing all lower-bound control can lose finiteness even for finite convex functions: and give , hence . Nor does finiteness alone guarantee differentiability: choosing gives by the triangle inequality and the split .
Global maximizer 2026-11-05
A global maximizer of is a point satisfying for every . The global maximum is the attained value ; argmax collects all points attaining that value:The set can be empty or contain many points. For on the real line, the global maximum is one and the global maximizers are , . No single representative is specified by the optimization problem.
A stationary point is a different kind of condition. For a differentiable function on an open real interval, an interior global maximizer has zero derivative, but the converse fails: has zero derivative at zero without a local maximum. A boundary global maximizer need not have zero derivative either; on attains its largest value at the right endpoint.
The extreme value theorem supplies a useful sufficient existence condition: a continuous function on a nonempty compact set attains its maximum. Compactness is sufficient, rather than necessary; the cosine example attains a global maximum on the noncompact real line.
Global minimum Created 2026-09-24 Updated 2026-10-14
A real-valued function attains a global minimum at whenThe point is a global minimizer; is the minimum value. A global maximum reverses the inequality throughout the same domain of a function. Unlike a local minimum or local maximum, a global minimum or global maximum compares every permitted input, however distant. The domain of a function is part of the claim.
For on , completing the square giveswith equality exactly at . This proves that is the global minimum and that the two global minimizers are and . Neither is a strict global minimizer, because the other attains the same value; both are strict local minima. There is no global maximum on , whereas on the global maximum is , attained at zero.
Finding critical points supplies candidates rather than a global certificate. The squared expression above is beautiful because one inequality compares infinitely many inputs at once. A global minimum is always a local minimum under the relative-neighbourhood convention; the converse fails. For example, has a strict local minimum at , but as , so it has no global minimum on the real line.
Existence also matters: on has infimum zero and supremum one but attains neither a global minimum nor a global maximum. By the extreme value theorem, a continuous function on a nonempty closed interval attains both. To find those values, compare all interior critical numbers, including points where the derivative does not exist, and both endpoints. Checking only stationary points can miss a corner or an endpoint. [ref]
https://openstax.org/books/calculus-volume-1/pages/4-3-maxima-and-minima. Gilbert Strang and Edwin Herman, OpenStax Calculus Volume 1, §4.3: locating absolute extrema
Interior-value criterion for an attained maximum Created 2026-10-06 Updated 2026-11-05
Let be bounded above and continuous on . Suppose . If some satisfiesthen attains a global maximum on , at a point other than . The value must be specified even though continuity there is not assumed.
Choose with . The limit superior hypothesis gives , small enough that , for which whenever . The extreme value theorem gives a maximizer on the compact interval . Its value is at least , hence exceeds every omitted value and . This proves attainment despite a possible oscillatory discontinuity at the endpoint.






