Past exam of the mathematics course of the University of Cambridge / 2015 / ii / Paper 4 / 25K (Optimization and Control)

Treat the white noise as innovations with conditional mean zero and conditional second moment $v$, independent of the chosen past control. This is the convention needed for the dynamic programming calculation. The printed stage cost is $\frac{1}{2}{x}^2+u^2$, not one-half of both terms.

With terminal value $V_0(x)=x^2$, induction in the Bellman equation givesIf $V_{h-1}(z)=z^2+(h-1)v$, completing the square yieldsThus $\boxed{V_h(x)=x^2+hv,u^{\ast }(x)=-x/2}$ for every horizon, in particular $V_6(x_0)=x_0^2+6v$. The same completion gives the average-cost Bellman equation solution $\boxed{\varphi (x)=x^2,\lambda =v}$, up to an additive constant in $\varphi$.

For every policy in $P$, conditional second moments satisfy $\mathbb{E}[x_{t+1}^2\mid {\mathcal{F}}_t]=(x_t+u_t{)}^2+v\le 0.9x_t^2+v$. Iterating givesThe Bellman inequality implies ${\mathbb{E}}_{\pi }{\sum }_{t<h}(\frac{1}{2}{x}_t^2+u_t^2)\ge hv+x_0^2-{\mathbb{E}}_{\pi }{x}_h^2$. Dividing by $h$ and using the bound proves that every admissible policy has lower limiting average cost at least $v$, even if its ordinary limit does not exist. The feedback $u=-x/2$ belongs to $P$ because $(x+u{)}^2=x^2/4\le 0.9x^2$. It makes the Bellman inequality an equality, and its bounded terminal second moment proves its average cost tends to $\boxed{v}$. Therefore it is optimal over $P$.

Merely uncorrelated noise with the displayed unconditional moments would not ensure the conditional identities for arbitrary adapted controls. The innovation interpretation specifies the hypothesis used in the calculation.