Past exam of the mathematics course of the University of Cambridge / 2009 / ii / Paper 3 / 28I (Optimization and Control)

Put $\beta =2/3$. The Bellman equation is $F(x)={\inf }_u\{x^2+u^2+\beta \mathbb{E}F(x+u+ϵ)\}$. For the quadratic candidate $G(x)=P{x}^2+d$, the quantity minimized isFor $P\ge 0$, completing the square gives $u_{\ast }=-\beta Px/(1+\beta P)$. Matching the $x^2$ and constant terms requiresThe first equation is $2P^2-P-3=0$. Its admissible nonnegative root is $P=3/2$, and $d=3$. Hence $\boxed{G(x)=\frac{3}{2}{x}^2+3,u_t=-x_t/2}$.

A suitable discounted verification theorem is: if a nonnegative candidate satisfies the Bellman equation, a stationary minimizing selector has finite cost and its discounted expected terminal candidate tends to zero, and the same terminal limit holds for every finite-cost competing policy, then that selector is optimal and the candidate is the value function. Iterating the Bellman inequality proves the lower bound for each competing cost; iterating equality along the selector proves attainment. Here finite cost implies ${\sum }_t{\beta }^t\mathbb{E}{x}_t^2<\infty$, so ${\beta }^t\mathbb{E}G(x_t)\to 0$. Under the selector $x_{t+1}=x_t/2+{ϵ}_t$, the second moments are bounded by a geometric recursion, giving finite cost and the same terminal limit. Thus $F=G$. Infinite-cost policies cannot improve the value.

For $\lambda =0$, the two coordinates decouple and have the same discount, so $\boxed{u_t=-x_t/2,w_t=-y_t/2}$. The total value is $(3/2)(x^2+y^2)+6$.

For $\lambda =1$, the discount is $\beta =3/4$ and the state cost is $(x+y{)}^2$. Introduce the orthogonal transformation $s=(x+y)/\sqrt{2}$, $q=(x-y)/\sqrt{2}$, with controls $v=(u+w)/\sqrt{2}$, $z=(u-w)/\sqrt{2}$. The cost becomes $2s^2+v^2+z^2$. Each transformed noise has mean zero and variance one; they need not be independent unless the original noises are Gaussian, and no such independence is needed here. The unpenalized difference coordinate has optimal control $z=0$. For the sum coordinate, the quadratic coefficient solvesso $P=8/3$, $v=-2s/3$, and the constant is $d=\beta P/(1-\beta )=8$. ThereforeThe sum state is mean-square stable under this feedback; the uncontrolled difference state may wander, but its value function and state cost are both zero. The same verification argument applies to the penalized sum coordinate.