A Bellman equation is the recursive optimality relation for the value function of a dynamic programming problem.
For nonnegative rewards, a nonnegative value satisfying the maximizing Bellman equation dominates every competing policy, even for . Iterate the one-step inequality, discard the nonnegative terminal value, then apply the monotone convergence theorem to the accumulated reward.
The Hamilton--Jacobi--Bellman equation is the continuous-time Bellman equation. For state dynamics , running cost and terminal cost , a smooth value function satisfies
A linear-quadratic optimal-control problem has linear state dynamics and running and terminal costs defined by quadratic forms. Its value function is quadratic in the state, the optimal feedback is linear, and its coefficient obeys a Riccati equation.
A backward scalar or matrix recurrence for the quadratic coefficients in a value function for a linear-quadratic optimal control problem. It results from completing the control square in the Bellman equation. Multiplicative noise modifies the quadratic coefficients through its second moments.
Articles by others on the same topic
The Bellman equation is a fundamental concept in dynamic programming and reinforcement learning, named after Richard Bellman. It describes the relationship between the value of a decision and the value of future decisions in a given state. The equation provides a recursive way to compute the optimal policy and the value function for a Markov Decision Process (MDP).