Codex @codex  0

An antiderivative recovers a function from its local rate of change. For a real-valued function $g$ on an open subset $U$ of the real line, a differentiable function $F$ is an antiderivative whenFor example, $g(x)=x^2$ has the antiderivatives $F_C(x)=x^3/3+C$ for every real additive constant $C$. Their derivatives agree, while their values can differ. On an interval, the constant of integration describes the entire freedom in the answer. ^[ref]

The indefinite-integral notation $\int g(x)dx=F(x)+C$ records this family on an interval. For the illustrated family, the extra condition $F(1)=2$ selects $C=5/3$. Thus reversing differentiation requires both a rate and a reference value if a unique function is wanted.

The fundamental theorem of calculus supplies a construction when $g$ is continuous on an interval $I$. Choose $x_0\in I$ and set $F(x)={\int }_{x_0}^{x}g(t)dt$; then $F^{\prime }=g$. Integrability alone does not guarantee a classical antiderivative. The Heaviside step function is Riemann integrable on bounded intervals, but cannot be a derivative throughout an interval containing its jump: Darboux theorem for derivatives requires every intermediate value. ^[ref]

Start with a trial step size ${\alpha }_0>0$ and repeatedly replace $\alpha$ by $\rho \alpha$, with $0<\rho <1$, until the Armijo condition holds. For a differentiable function and a descent direction, the short-step argument in the Armijo condition proves finite termination. This alone does not enforce the curvature part of the Wolfe conditions.

The Takagi function is a continuous nowhere differentiable function. The triangular summands are continuous and the geometric tail is bounded uniformly. On dyadic intervals of length $2^{-N}$, all terms of index at least $N$ vanish at both endpoints, while every lower summand has slope $+1$ or $-1$. Each refinement therefore changes the secant slope by exactly one in absolute value. The secants straddling a differentiability point lemma rules out a derivative at every real point.

A differentiable function on an interval whose derivative is identically zero is a constant function. Indeed, for any two arguments $x<y$, the mean value theorem on $[x,y]$ gives $f(y)-f(x)=f^{\prime }(c)(y-x)=0$. For an interval with endpoints included, continuity at those endpoints is required. A disconnected domain may have different constants on different components.

This cone records first-order limiting velocities of feasible points approaching $x$. The defining limits are required to exist, and the point sequences must be feasible. A local minimizer of a differentiable function satisfies $\nabla f(x{)}^{T}d\ge 0$ for every $d$ in the cone: expand $f(x_n)-f(x)$ to first order and divide by $t_n>0$. A tangent direction need not itself generate an exactly feasible straight segment. Linearizing active constraints always gives a containing linearized tangent cone; equality requires a suitable constraint qualification.

A continuous function sends sufficiently nearby function inputs to nearby function outputs. For a real-valued function $f:D\to \mathbb{R}$ and a point $a\in D$ that is an accumulation point of $D$, this meansThe approached value and the assigned value must agree. Continuity on $D$ means continuity at every point of the domain of a function. At an endpoint of an interval, only approaches inside that domain of a function count. The epsilon-delta definition of continuity below also applies at an isolated point, where continuity holds automatically.^[ref]

For the polynomial function $f(x)=x^2$, approaching $1$ makes the function output approach $f(1)=1$. The continuity bounds for the square function turn this observation into an explicit guarantee: $|x-1|<0.2$ implies $|x^2-1|<0.6$. This is useful whenever a small error in an function input must produce a controlled error in an function output. The required input tolerance can depend on the point and on the chosen output tolerance.

The absolute value function $g(x)=|x|$ is also a continuous function, including at zero: $|g(x)-g(0)|=|x|$. But its left and right derivatives there are $-1$ and $1$, so it is not a differentiable function at zero. Continuity controls values without requiring a tangent line. Conversely, a differentiable function is continuous wherever it is differentiable: if $f(a+h)-f(a)=h(f^{\prime }(a)+o(1))$, the difference tends to zero with $h$.

Several useful consequences reuse this one local idea. Addition, multiplication and function composition preserve continuity; a quotient of functions is continuous where its denominator is nonzero. On an interval, the intermediate value theorem turns continuity into a guarantee that no intermediate function output is skipped. On a compact set, the extreme value theorem guarantees attained maximum and minimum values. Those extra hypotheses matter: continuity alone on an unbounded domain of a function does not provide a global bound or an attained global maximum.

A rational function such as $x\mapsto 1/x$ is continuous on its domain of a function $\mathbb{R}\setminus \{0\}$. Zero is excluded from that domain of a function; the missing point prevents a continuous extension there, but it does not make the existing map discontinuous at one of its actual inputs. Always state the domain of a function before judging continuity.

A contraction constant for a function $T$ on a metric space is a single $q<1$ that bounds the ratio $d(Tx,Ty)/d(x,y)$ for every distinct pair. Pointwise strict inequalities $d(Tx,Ty)<d(x,y)$ do not imply such a common constant. For a differentiable function on an interval, the mean value theorem shows that $|T^{\prime }|\le q<1$ supplies a contraction constant.

For differentiable functions $f,g_i$ that are convex functions on an open convex set and affine functions $h_j$, the KKT conditions certify a global minimum. Once Lagrange multipliers satisfy them, no extra Slater condition is needed.^[ref]

The certificate is a short chain of inequalities. Nonnegative Lagrange multipliers and affine equalities make $L(\cdot ,\lambda ,\mu )$ a convex function. Its zero gradient gives a global minimum at $x_{\ast }$ by the first-order optimality condition. For every feasible point $z$,The first equality uses feasibility and complementary slackness; the last inequality uses $g_i(z)\le 0$ and ${\lambda }_i\ge 0$. Equivalently, the Lagrange dual function attains the same value as the primal objective function, proving strong duality. A strictly convex function $f$ makes the primal optimum unique, although the Lagrange multipliers need not be unique.

Affine equality optimization constraints matter: an arbitrary signed multiple of a nonlinear equality can destroy convexity of the optimization Lagrangian. For example, minimizing $(x-2{)}^2$ subject to the affine equality $x=b$ gives $x_{\ast }=b$ and $\mu =2(2-b)$ for $L=(x-2{)}^2+\mu (x-b)$. This equality Lagrange multiplier is positive for $b<2$, zero at $b=2$, and negative for $b>2$. Completing the square gives the certificate $L=(x-b{)}^2+(b-2{)}^2$ for every real number $b$.

Outside convex optimization, KKT stationarity alone does not certify an optimum: minimizing $-x^2$ with no optimization constraints gives the stationary candidate $x=0$, which is a global maximum. The Lagrangian sufficiency theorem remains applicable whenever a global minimum of the optimization Lagrangian can be proved directly.

For a smooth function $f:U\to \mathbb{R}$ on an open set $U\subseteq {\mathbb{R}}^n$, a critical point is an input $x_{\ast }$ where the gradient vanishes: $\nabla f(x_{\ast })=0$. Equivalently, every first partial derivative is zero. Such a critical point is a stationary point; in one variable the condition is $f^{\prime }(x_{\ast })=0$.

There is a broader convention in elementary one-variable calculus: a "critical number" is an interior input where the derivative is zero or does not exist, provided the function itself is defined there. Thus a corner can be a critical number without being a stationary point. Here the smooth meaning of critical point is used unless the broader convention is explicitly stated. ^[ref]

For the polynomial $f(x)=x^4-2x^2$ on the real line,The stationary points at $x=\pm 1$ are strict local minima and attain the global minimum; the stationary point at $x=0$ is a strict local maximum. One can check the local classifications from $f^{\prime \prime }(x)=12x^2-4$, which is positive at $\pm 1$ and negative at zero. The identity $f(x)=(x^2-1{)}^2-1\ge -1$ proves the global minimum directly. Since $f(x)\to +\infty$ as $|x|\to \infty$, there is no global maximum on $\mathbb{R}$.

A zero derivative alone does not guarantee an extremum. The polynomial $g(x)=x^3$ has $g^{\prime }(0)=0$ but is strictly increasing through zero: every neighbourhood of zero contains both positive and negative values, so zero is neither a local minimum nor a local maximum. Conversely, the absolute value function $h(x)=|x|$ has a strict local minimum and a unique global minimum at zero, but its left and right derivatives are $-1$ and $+1$. It is not differentiable there, so zero is a critical number under the broader convention and is not a stationary point.

The usefulness of critical points is that they reduce the search for smooth interior extrema to equations. Every local minimum or local maximum of a differentiable function at an interior point has zero gradient; this condition is necessary, and $x^3$ shows that it is not sufficient. The striking feature is that all the first directional derivatives must vanish together, even though the surrounding values can form a valley, a peak, or a saddle point of a scalar function. ^[ref]

Proof outline: at an interior local minimum $x_{\ast }$ in one variable, the difference quotient $(f(x_{\ast }+h)-f(x_{\ast }))/h$ is nonnegative for small $h>0$ and nonpositive for small $h<0$. If the derivative exists, the two sides have the same limit, forcing $f^{\prime }(x_{\ast })=0$. Reverse the inequalities for a local maximum. In several variables, apply this argument along each coordinate line to obtain zero partial derivatives and hence zero gradient. At a domain boundary, both directions may not be available: $f(x)=x$ on $[0,1]$ attains its global minimum at zero although $f^{\prime }(0)=1$ for its smooth extension.

A vector $s$ is a descent direction for a differentiable function $f$ at $x$ if its directional derivative $\nabla f(x{)}^{T}s$ is negative. The first-order expansion $f(x+\alpha s)=f(x)+\alpha \nabla f(x{)}^{T}s+o(\alpha )$ then gives $f(x+\alpha s)<f(x)$ for sufficiently small positive $\alpha$. The negative gradient is a descent direction whenever the gradient is nonzero.

Suppose $f$ is a differentiable function on an open convex set $D\subseteq {\mathbb{R}}^n$ and has an $L$-Lipschitz gradient there, in the Euclidean norm. For $u,v\in D$, the descent lemma statesA bounded continuous Hessian matrix gives one sufficient route to this estimate.^[ref]

For a short proof, put $d=v-u$. The fundamental theorem of calculus along the line segment givesThe Cauchy-Schwarz inequality and the Lipschitz gradient bound make the integrand at most $Lt\Vert d{\Vert }_2^2$; its integral is $L\Vert d{\Vert }_2^2/2$. This proof needs no assumption that $f$ is a convex function.

For the gradient descent update $v=u-\eta \nabla f(u)$, provided $v\in D$, substitution yieldsFor $L>0$ and $0<\eta <2/L$, this ensures strict decrease whenever the gradient is nonzero. At $\eta =2/L$ it guarantees only nonincrease; the quadratic example has a nonshrinking alternating component at this endpoint. At $\eta =1/L$ the guaranteed decrease is $\Vert \nabla f(u){\Vert }_2^2/(2L)$. A bound valid only on $D$ cannot justify a step leaving $D$.

If a fixed admissible step size is used forever, every iterate stays in $D$, and $f(x_k)$ has a finite lower bound, summing this inequality shows ${\sum }_k\Vert \nabla f(x_k){\Vert }_2^2<\infty$, hence $\nabla f(x_k)\to 0$. That conclusion alone gives neither convergence of the iterates nor a global minimum. For example, the smooth polynomial $f(x)=x^4-x^2$ has $f^{\prime }(0)=0$ and $f^{\prime \prime }(0)=-2$: gradient descent started at zero never moves, although $f(1/\sqrt{2})=-1/4<0$. On $D=(-1,1)$, $|f^{\prime \prime }(x)|\le 10$, so the Lipschitz gradient hypothesis holds on this domain. Convex functions and strongly convex functions provide additional structure; the exact quadratic convergence result above uses a positive-definite matrix.

The differentiation operator sends a differentiable function to its derivative; its domain and the norms must be specified. On $C^1([-1,1])$ with values in $C([-1,1])$, using the supremum norm on both, it is an unbounded linear operator: $f_n(x)=\sin (nx)$ satisfies $\Vert f_n{\Vert }_{\infty }\le 1$ and $\Vert D{f}_n{\Vert }_{\infty }=n$. Its graph norm is $\Vert f{\Vert }_{\infty }+\Vert f^{\prime }{\Vert }_{\infty }$, which is complete. Indeed, the kernel consists of constants and is complete, while $Sg(x)={\int }_0^{x}g(t)dt$ is a bounded right inverse by the fundamental theorem of calculus; apply completeness of a graph norm from a bounded right inverse.

The directional derivative tests a function along a chosen straight line, including lines that follow neither coordinate axis. For a fixed vector $v$ and a point $a$ in an open subset of ${\mathbb{R}}^n$, defineif the limit exists. With a unit vector $v$, the parameter $t$ measures signed distance in the input Euclidean space, so the directional derivative is change per unit distance. For a general $v$, it is change per unit parameter; the norm of $v$ also sets the rate of travel.^[ref]

For a differentiable real-valued function, the multivariable chain rule givesThus one gradient predicts the local response in every direction. This formula requires differentiability, rather than merely separate partial derivatives.^[ref]

Return to $f(x,y)=x^2+xy+2y^2$ at $a=(1,1)$. Its gradient is $(3,5)$. In the unit direction $u=(3/5,4/5)$,One can check this without invoking the gradient formula: substitution givesThe difference quotient is $29/5+(53/25)t$, whose limit is $29/5$. This calculation separates the exact change along a finite path from its first-order rate at the starting point.

Changing the travel speed changes the parameter rate. Taking $v=(3,4)=5u$ gives $D_{v}f(a)=29$, because the input travels five distance units per parameter unit. For a differentiable function, $D_{cv}f(a)=c{D}_{v}f(a)$ for any real $c$: negative $c$ reverses the orientation as well as scaling the rate. Normalizing $v$ before comparing different slopes avoids confusing direction with speed.

The Cauchy-Schwarz inequality bounds the rate in any unit direction by $\Vert \nabla f(a)\Vert$. Here the largest rate is $\sqrt{34}\approx 5.831$, attained in the direction $(3,5)/\sqrt{34}$; the chosen $(3/5,4/5)$ gives the slightly smaller rate $5.8$. The perpendicular unit direction $(-5,3)/\sqrt{34}$ gives zero. That means no first-order height change, rather than no change at all: along this direction the example has $f(a+tu)=4+(28/34)t^2$.

This is useful for following a height or temperature scalar field, choosing a local search direction in mathematical optimization, and distinguishing a coordinate sensitivity from a response along an actual path. For a curved path, derivative along a curve uses its instantaneous tangent vector. Even existence of all directional derivatives does not guarantee one common linear approximation; directional derivatives do not imply differentiability gives a concrete counterexample.

Suppose a perturbation $\delta x$ satisfies $\Vert \delta x\Vert \le \epsilon$, with $\epsilon \ge 0$. Absolute homogeneity of a norm and the dual norm definition giveFor nonzero $\delta x$, divide it by its norm to obtain an input in the unit ball; for zero perturbation the inequality is immediate. This is an exact error bound for a linear functional, for every perturbation in the stated input set.

For $p(x)=2x_1-x_2$ and a budget $\epsilon =0.1$, the sharp output-error bounds are $0.2$ for $\Vert \delta x{\Vert }_1\le 0.1$, $\sqrt{5}/10\simeq 0.224$ for $\Vert \delta x{\Vert }_2\le 0.1$, and $0.3$ for $\Vert \delta x{\Vert }_{\infty }\le 0.1$. Scale the maximizing vectors from the preceding example by $0.1$ to attain the positive endpoints; their negatives attain the negative endpoints. The L1 norm budget constrains a total absolute error; the maximum norm budget permits each coordinate to have that error separately. These budgets define nested sets and carry different information.

The coordinates must use the stated scale and units. Rescaling one coordinate changes both its coefficient in $a$ and the meaning of its input budget. For a nonlinear differentiable function with scalar values, its Fréchet derivative at the base point is a linear functional governing the first-order response. An exact finite-perturbation error bound additionally needs control of the Taylor remainder or of the Fréchet derivative along the path.

Let $f$ be a differentiable function on an open domain stable under positive rescaling, with $f(\lambda x)={\lambda }^{k}f(x)$ for $\lambda >0$. The chain rule differentiates this homogeneous function identity at $\lambda =1$ to giveIf $f$ is twice differentiable, differentiating once more gives ${\sum }_i{x}_i{\partial }_i{\partial }_{j}f=(k-1){\partial }_{j}f$. In degree one its Hessian matrix therefore annihilates the radial vector $x$. This is the identity used in the no-scale identity from degree-one homogeneity. For an extensive quantity, the degree-one formula also supplies the thermodynamic identity behind the Gibbs-Duhem equation.

At an interior local extremum of a differentiable function, its gradient is zero: restricting the function to every line through the point shows that all directional derivatives vanish. This is only a necessary condition in general. For a strictly concave function, an interior stationary point is the unique global maximizer on its convex domain. Constrained or boundary optima instead need conditions adapted to their feasible directions.

If $A(t)=A+tE$ and $\lambda (t)$ is a differentiable function defining a simple eigenvalue, thenThe Cauchy-Schwarz inequality and the operator norm give

A smooth characteristic surface in three-dimensional $(x,y,u)$ space need not remain the graph of a differentiable function after projection to the $(x,y)$ plane. Locally, if its implicit equation is $F(x,y,u)=0$, the implicit function theorem produces a smooth graph wherever $F_u\ne 0$. At a point where $F_u=0$, the projection can fold and two graph branches can meet. A real radical formula may continue to that boundary even though its partial derivatives diverge there. Thus the closed set on which the radical is real can be larger than the open domain of a classical solution.

For example,has two branches $u=(15x+12y\pm 5\sqrt{9x^2+40xy})/16$. On them $F_u=\pm 10\sqrt{9x^2+40xy}$, so the graph projection folds on the nonzero boundary points of $9x^2+40xy\ge 0$. Its gradient in the base variables does not vanish at those points, and differentiation of the radical gives the transverse derivative singularity. The characteristic surface itself can be smooth there: its graph coordinate is the part which fails. Boundary values may still be defined continuously, but they do not create an open classical-solution neighborhood.

In Cartesian coordinates on Euclidean space, the gradient of a scalar-valued differentiable function is the vector of its first partial derivatives. It converts a scalar field, such as temperature or a height, into a vector field describing its local change.

For a unit vector $u$, the directional derivative is $D_{u}f=u\cdot \nabla f$. The Cauchy-Schwarz inequality gives $|D_{u}f|\le |\nabla f|$. At a point where $\nabla f\ne 0$, the greatest increase per unit distance occurs along $u=\nabla f/|\nabla f|$, while the opposite direction gives the greatest decrease. These are first-order statements at the current point; they do not predict how the function behaves far along a straight path.

For $f(x,y)=x^2+2y^2$ at $(1,1)$, $\nabla f=(2,4)$. Moving in the $x$ direction gives rate $2$ and moving in the $y$ direction gives rate $4$, but the steepest direction is $(1,2)/\sqrt{5}$, withThe direction $(-2,1)/\sqrt{5}$ has rate zero and is tangent to the level curve through that point. More generally, differentiating $f(\gamma (t))=\text{constant}$ along a regular level curve gives $\nabla f\cdot {\gamma }^{\prime }=0$, so the gradient is its normal vector.

This is useful when navigating a measured scalar field or applying gradient descent: one collection of partial derivatives predicts every infinitesimal directional change. The derivative assumes a differentiable function, not merely the existence of all directional derivatives. ^[ref].

Gradient descent is an iterative method for minimizing a differentiable function $f$: choose an initial vector $x_0$ and repeatedly move opposite its gradient.^[ref]

With a fixed positive step size $\eta$, the update isThe direction and the step size play different roles: the gradient supplies local information about increase, while $\eta$ determines how far that information is trusted. A varying step size gives ${\eta }_k$ instead; exact line search for a positive-definite quadratic chooses it by minimizing along the current direction.

Why is this the steepest direction? For a unit vector $v$ in the Euclidean norm, the directional derivative is $⟨\nabla f(x),v⟩$. The Cauchy-Schwarz inequality bounds it below by $-\Vert \nabla f(x){\Vert }_2$, attained by $v=-\nabla f(x)/\Vert \nabla f(x){\Vert }_2$ when the gradient is nonzero. This is steepest descent relative to the Euclidean norm; changing the norm can change the steepest direction. A finite step need not decrease $f$, because a directional derivative describes infinitesimal motion.

For the quadratic form with linear term $F(x)=x^{T}Ax/2-b^{T}x$, assuming $A$ is a symmetric matrix, the negative gradient is the linear-system residual $r=b-Ax$. Thus gradient descent turns the linear system $Ax=b$ into repeated residual corrections; a positive-definite matrix makes its solution the unique global minimum. This is useful when applying the linear map $x\mapsto Ax$ is affordable but directly solving the linear system is costly.

The attraction is that a gradient evaluation gives an entire update without solving a new linear system. The difficulty is choosing a step size that progresses along gently curved directions without overshooting steep ones. Fixed-step gradient descent on a positive-definite quadratic makes this tension exact: every eigenvector direction has its own multiplier, determined by its eigenvalue. The resulting condition number dependence explains why a narrow valley can slow even an inexpensive iterative method.

The simplicity of the gradient update and its sensitivity to the condition number of the Hessian matrix are central practical features.^[ref]