Past exam of the mathematics course of the University of Cambridge / 2006 / ii / Paper 2 / 29I (Optimization and Control)

Let $V(x)=F(\pi ,x)$ satisfy the Bellman optimality equation $V(x)={\sup }_u\{r(x,u)+\beta E[V(x_1)\mid x,u]\}$. It is nonnegative because the rewards are. Under any competing policy, iterate its Bellman inequality to obtain $V(x)\ge E[{\sum }_{t=0}^{N-1}{\beta }^{t}r(x_t,u_t)+{\beta }^{N}V(x_N)]$. Drop the nonnegative terminal term and let $N\to \infty$. Monotone convergence gives $V(x)\ge F(\tilde{\pi },x)$ for every competitor. Since $V$ is the actual return of $\pi$, this proves optimality without a transversality assumption.

Assume the income process remains nonnegative, as required by the feasible-control interval; for unrestricted feasible controls a sufficient condition is ${ϵ}_t\ge -1$. If $\beta \le 1/(1+\theta )<1$, spending everything gives constant income $x$ and value $V(x)=x/(1-\beta )$. Its Bellman expression for general $u$ isThe coefficient of $u$ is nonnegative, so the maximum over $[0,x]$ occurs at $u=x$ and equals $V(x)$. Therefore spending all income at every step is optimal.

If $1/(1+\theta )<\beta <1$ and $x>0$, invest all income for $T$ steps and thereafter spend everything. Its expected return is $x[\beta (1+\theta ){]}^T/(1-\beta )$, unbounded as $T\to \infty$. In fact a sufficiently small fixed positive consumption fraction gives infinite expected discounted reward if $\beta [1+\theta (1-c)]\ge 1$. Thus the value is infinite and the problem has no finite-valued optimum. For $\beta =1$, spending everything already gives infinite value when $x>0$. If $x=0$, the value is zero. A mean condition alone, without nonnegative income or equivalent admissibility restrictions, would not make the model well-defined.