A Markov decision process specifies states, admissible actions, action-dependent transition probabilities and rewards. A policy chooses actions from observed information; a time-homogeneous problem uses transition and reward rules independent of the date. A Bellman equation optimizes current reward plus the conditional value of subsequent decisions. For nonnegative rewards, finite-horizon values increase as the horizon grows, while identification of their limit with the infinite-horizon value requires an approximation argument for admissible policies.
Estimating the expected return under a specified control policy. It is distinct from improving or optimizing that policy. A Bellman expectation equation describes the fixed-policy value, whereas a Bellman optimality equation compares admissible actions.
Policy iteration alternates evaluation of a fixed decision policy with improvement of its actions using the resulting value function. A stable policy satisfies the Bellman equation when each chosen action attains the required optimum.
Articles by others on the same topic
A Markov Decision Process (MDP) is a mathematical framework used to model decision-making in situations where the outcomes are partly random and partly under the control of a decision maker. MDPs are widely used in fields like operations research, economics, robotics, and artificial intelligence, especially for reinforcement learning problems. An MDP is defined by the following components: 1. **States (S)**: A finite set of states that represent the possible situations in which an agent can find itself.