For a controlled ordinary differential equation , running cost and terminal cost , a smooth optimal value function satisfiesThis is the continuous-time Bellman equation. Split the cost into a short initial interval and the optimal continuation. Expanding the continuation by the chain rule, dividing by the interval length and taking its limit yields the equation. The optimization acts on the control at the current state and time.
For and running cost , the minimized expression is . Completing the square gives the minimizing control and the equation . The quadratic ansatz leads to linear-quadratic optimal control. [ref]
Underactuated Robotics by Russ Tedrake. Underactuated Robotics, chapter8, basic HJB derivation and finite-horizon formulation
For a controlled diffusion process , the corresponding equation adds inside the infimum. A value function need not be differentiable; the viscosity solution formulation handles many such cases. A classical verification argument additionally requires the candidate feedback to be admissible and to attain the indicated infimum.
For a nonnegative classical solution of a finite-horizon Hamilton-Jacobi-Bellman equation with nonnegative running reward , the process is a local supermartingale under every admissible control. The Itô formula gives nonpositive drift because the chosen control cannot exceed the supremum in the equation. Stop where the stochastic integral becomes a true martingale, take expectations and apply the Fatou lemma. Nonnegativity prevents loss of a lower bound when removing the stops, giving the verification upper bound. A maximizing control attains the bound when its stopped reward processes are uniformly integrable.
A standard finite-horizon linear-quadratic problem uses linear dynamics and minimizeswhere are symmetric positive semidefinite matrices and is positive-definite. The control input is unrestricted in Euclidean space. These hypotheses, rather than the word quadratic alone, make the control minimization well posed.
The value function has the quadratic form . Substitution into the Hamilton-Jacobi-Bellman equation and completion of the control square giveThus a backward matrix Riccati equation gives a linear feedback control. [ref]
Underactuated Robotics by Russ Tedrake. Underactuated Robotics by Russ Tedrake, chapter8, Finite-horizon LQR
For the scalar system , terminal cost zero and running cost , the solution is and . Differentiation verifies . Infinite-horizon formulas need additional stabilizability and detectability conditions; they do not follow from an arbitrary quadratic cost.
A backward scalar or matrix recurrence for the quadratic coefficients in a value function for a linear-quadratic optimal control problem. It results from completing the control square in the Bellman equation. Multiplicative noise modifies the quadratic coefficients through its second moments.
Articles by others on the same topic
The Hamilton–Jacobi–Bellman (HJB) equation is a fundamental partial differential equation in optimal control theory and dynamic programming. It provides a necessary condition for an optimal control policy for a given dynamic optimization problem. ### Context In many control problems, we aim to find a control strategy that minimizes (or maximizes) a cost function over time.