= Hamilton-Jacobi-Bellman equation
{c}
{wiki}
= HJB equation
{c}
{synonym}
For a controlled <ordinary differential equation> $\dot x=f(x,u,t)$, running cost $L(x,u,t)$ and terminal cost $\Phi(x)$, a smooth optimal <value function> satisfies
$$
V_t+\inf_u\{L+\nabla V\cdot f\}=0,\qquad V(T,x)=\Phi(x).
$$
This is the continuous-time <Bellman equation>. Split the cost into a short initial interval and the optimal continuation. Expanding the continuation by the <chain rule>, dividing by the interval length and taking its limit yields the equation. The optimization acts on the control at the current state and time.
For $\dot x=u$ and running cost $x^2+u^2$, the minimized expression is $x^2+u^2+V_xu$. Completing the square gives the minimizing control $u=-V_x/2$ and the equation $V_t+x^2-V_x^2/4=0$. The quadratic ansatz leads to <linear-quadratic optimal control>. <Underactuated Robotics by Russ Tedrake>[Underactuated Robotics, chapter8, basic HJB derivation and finite-horizon formulation]{ref}
For a controlled <diffusion process> $dX=fdt+\sigma dW$, the corresponding equation adds $\tfrac12\operatorname{tr}(\sigma\sigma^TD_x^2V)$ inside the infimum. A value function need not be differentiable; the <viscosity solution> formulation handles many such cases. A classical verification argument additionally requires the candidate feedback to be admissible and to attain the indicated infimum.
Back to article page