Past exam of the mathematics course of the University of Cambridge / 2011 / ii / Paper 3 / 28K / Solution

The stage cost is $(x+u{)}^2+6u^2\ge 0$. The finite-horizon Bellman equation isInductively write $C_{s-1}(x)={\Pi }_{s-1}{x}^2$. Completing the square in $u$ gives the unique optimal first controlandFor $0\le q<2$, the recurrence map $F(q)$ satisfies $F(q)<2$ andThus ${\Pi }_s$ increases and is bounded above by two. Its limit solves $L=6(1+L)/(7+L)$, or $(L-2)(L+3)=0$; nonnegativity forces $L=2$. In particular $C_{\infty }(x_0)\ge 2x_0^2$, and indeed this recurrence already identifies the limit.

For the proposed policy, induction on the state equation gives $x_t=(2/3{)}^t{x}_0$ and $u_t=-x_t/3$. The stage cost is $(10/9)x_t^2$, so the infinite geometric series givesThis supplies an admissible policy attaining the lower bound:The last formula is the requested closed-loop control: it uses the observed current state rather than an initial-state schedule.