A controlled Markov process has transition probabilities or rates depending on its current state and the chosen control action. A Markov policy selects an action from the current state and possibly the time. Once such a policy is fixed, the induced state evolution is a Markov process. For general history-dependent controls, the state alone need not remain Markov. A dynamic programming operator optimizes over the available actions at each state.
A Markov decision process specifies states, admissible actions, action-dependent transition probabilities and rewards. A policy chooses actions from observed information; a time-homogeneous problem uses transition and reward rules independent of the date. A Bellman equation optimizes current reward plus the conditional value of subsequent decisions. For nonnegative rewards, finite-horizon values increase as the horizon grows, while identification of their limit with the infinite-horizon value requires an approximation argument for admissible policies.
Estimating the expected return under a specified control policy. It is distinct from improving or optimizing that policy. A Bellman expectation equation describes the fixed-policy value, whereas a Bellman optimality equation compares admissible actions.
Policy iteration alternates evaluation of a fixed decision policy with improvement of its actions using the resulting value function. A stable policy satisfies the Bellman equation when each chosen action attains the required optimum.
Articles by others on the same topic
There are currently no matching articles.