Past exam of the mathematics course of the University of Cambridge / 2006 / ii / Paper 3 / 28I (Optimization and Control)

The standard positive-cost interpretation requires $a>0$ and $a_0\ge 0$. These signs are not printed. Without them the claim of a unique positive fixed point is false: $a=a_0=\lambda =0$ gives $f(z)=0$, with no positive fixed point. The following derivation gives the intended result under those cost assumptions; $b_0$ may be any finite constant.

Let $V_n(x)$ be the value with $n$ stages remaining. Starting from $V_0(x)=\frac{1}{2}{a}_0{x}^2+b_0$, the Bellman equation isIf $V_{n-1}(x)=\frac{1}{2}{a}_{n-1}{x}^2+b_{n-1}$, the minimized expression is a quadratic in $u$ with positive leading coefficient $1+\beta a_{n-1}$. Its derivative vanishes at $u=k_{n}x$, and completing the square givesInduction establishes the value form and the control $u_j=k_{T-j}{X}_j$. Only zero mean and variance ${\sigma }^2$ of the independent noise were needed.

The Riccati recurrence fixed-point equation becomesThe product of its roots is $-a/\beta <0$, so exactly one is positive:For $z\ge 0$, $f^{\prime }(z)=\beta {\lambda }^2/(1+\beta z{)}^2\ge 0$, while $f(z)-z$ is positive below $a_{\ast }$ and negative above it. Monotonicity of $f$ prevents overshooting the fixed point. Thus $a_n$ increases to $a_{\ast }$ if $a_0<a_{\ast }$, decreases to it if $a_0>a_{\ast }$, and is constant if $a_0=a_{\ast }$. Continuity identifies the limit uniquely.

Iteration of the second recurrence gives $b_n={\beta }^n{b}_0+\frac{1}{2}\beta {\sigma }^2{\sum }_{j=0}^{n-1}{\beta }^{n-1-j}{a}_j$. Subtract the same expression with every $a_j$ replaced by $a_{\ast }$. The early finite part vanishes geometrically, and the later part is bounded by ${\sup }_{j\ge J}|a_j-a_{\ast }|/(1-\beta )$ times the fixed factor. Taking $n\to \infty$, then $J\to \infty$, proves