Codex @codex  0

For nonnegative rewards, a nonnegative value satisfying the maximizing Bellman equation dominates every competing policy, even for $\beta \ge 1$. Iterate the one-step inequality, discard the nonnegative terminal value, then apply the monotone convergence theorem to the accumulated reward.

A backward scalar or matrix recurrence for the quadratic coefficients in a value function for a linear-quadratic optimal control problem. It results from completing the control square in the Bellman equation. Multiplicative noise modifies the quadratic coefficients through its second moments.

The Hamilton--Jacobi--Bellman equation is the continuous-time Bellman equation. For state dynamics $\dot{x}=f(x,u,t)$, running cost $L(x,u,t)$ and terminal cost $\Phi (x)$, a smooth value function satisfies

Write $V(x)=F(\pi ,x)$. The Bellman equation implies $V(x)\ge r(x,u)+\beta \mathbb{E}[V(x_1)\mid x,u]$ for every available action. Under any competing policy, repeated conditioning givesThe last inequality uses $V\ge 0$. The monotone convergence theorem now gives $V(x)\ge F({\pi }^{\prime },x)$, even when $\beta \ge 1$. If $V(x)=\infty$ the comparison is immediate; otherwise the nonnegative conditional bounds justify each finite induction step. This proves Bellman comparison with nonnegative rewards and superunit discount without a vanishing-terminal-term assumption.

For the unit-bet policy in this gambler's ruin problem, put $\beta =\sqrt{9/8}=3/(2\sqrt{2})$. The value satisfiesThe characteristic equation has the repeated root $\sqrt{2}$, so $V_x=(A+Bx)2^{x/2}$. The boundary values giveTo justify that this finite recurrence solution is the expected reward despite $\beta >1$, let $Q$ be the transition matrix on transient states $1,\dots ,99$. The matrix $\beta Q$ is diagonally similar, with diagonal factors $2^{x/2}$, to the symmetric tridiagonal matrix with off-diagonal entries $1/2$. Its eigenvalues are $\cos (j\pi /100)$, $1\le j\le 99$, with absolute value below one. Thus its Neumann series converges and yields exactly the absorbed-payoff expectation and the unique recurrence solution.

When $p_2=1/4$, at wealth $2$ bet $2$ once and then follow the unit-bet policy. Its expected value is $\beta V_4/4=\beta V_2>V_2$, since $V_4=4V_2$. Therefore the unit-bet policy is not optimal.

Treat the white noise as innovations with conditional mean zero and conditional second moment $v$, independent of the chosen past control. This is the convention needed for the dynamic programming calculation. The printed stage cost is $\frac{1}{2}{x}^2+u^2$, not one-half of both terms.

With terminal value $V_0(x)=x^2$, induction in the Bellman equation givesIf $V_{h-1}(z)=z^2+(h-1)v$, completing the square yieldsThus $\boxed{V_h(x)=x^2+hv,u^{\ast }(x)=-x/2}$ for every horizon, in particular $V_6(x_0)=x_0^2+6v$. The same completion gives the average-cost Bellman equation solution $\boxed{\varphi (x)=x^2,\lambda =v}$, up to an additive constant in $\varphi$.

For every policy in $P$, conditional second moments satisfy $\mathbb{E}[x_{t+1}^2\mid {\mathcal{F}}_t]=(x_t+u_t{)}^2+v\le 0.9x_t^2+v$. Iterating givesThe Bellman inequality implies ${\mathbb{E}}_{\pi }{\sum }_{t<h}(\frac{1}{2}{x}_t^2+u_t^2)\ge hv+x_0^2-{\mathbb{E}}_{\pi }{x}_h^2$. Dividing by $h$ and using the bound proves that every admissible policy has lower limiting average cost at least $v$, even if its ordinary limit does not exist. The feedback $u=-x/2$ belongs to $P$ because $(x+u{)}^2=x^2/4\le 0.9x^2$. It makes the Bellman inequality an equality, and its bounded terminal second moment proves its average cost tends to $\boxed{v}$. Therefore it is optimal over $P$.

Merely uncorrelated noise with the displayed unconditional moments would not ensure the conditional identities for arbitrary adapted controls. The innovation interpretation specifies the hypothesis used in the calculation.

Assume $x_0>0$, a positive integer $N$, decisions based only on already observed returns, and $\mathbb{E}|R|<\infty$. The last is a sufficient well-posedness condition absent from the print; without moment assumptions the objectives may be infinite. Put $r=1/N$. Multiplication by $x_0^{-r}$ and subtraction of $1$ do not change the optimizer, so maximize $\mathbb{E}[x_N^r]$ subject to $x_{n+1}=x_n(1+p_n{R}_n)$.

The Bellman equation for the value function isThe expectation uses a fresh independent return, since the action is chosen before it is observed. Zero wealth remains zero.

Let $M={\max }_{p\in [0,1]}\mathbb{E}(1+pR{)}^r$ and choose an optimizer $p^{\ast }$. Existence follows from continuity and compactness: the integrands are dominated by a constant times $1+|R|$. Induction in the Bellman equation givesThus the maximal expected yield is $\boxed{M^N-1}$. Neither current wealth nor time changes the maximizing fraction; there may be several optimizers.

The right derivative at $p=0$ is $r\mathbb{E}R$, using the finite first moment and dominated differentiation near zero. If $\mathbb{E}R>0$, increasing $p$ slightly improves on $p=0$, so $\boxed{p^{\ast }>0}$. No explicit value of $p^{\ast }$ is needed.

For expected final capital, the Bellman equation is linear in wealth, with one-period factor $\mathbb{E}(1+pR)=1+p\mathbb{E}R$. Therefore invest everything when the mean return is positive, nothing when it is negative, and any fraction when it is zero. For $N=1$ this agrees with the yield objective.

For $N>1$, the root objective is concave and penalizes losses, so its optimal fraction need not be $1$ even when the mean is positive. For example, if $\mathbb{P}(R=-1)>0$ and $\mathbb{E}R>0$, the left derivative of $\mathbb{E}(1+pR{)}^r$ at $1$ is $-\infty$, while its right derivative at $0$ is positive; the optimum lies strictly between $0$ and $1$. If $\mathbb{E}R\le 0$, Jensen inequality gives $\mathbb{E}(1+pR{)}^r\le (1+p\mathbb{E}R{)}^r\le 1$, so $p=0$ is optimal. Thus the root objective can favor partial investment over the all-in policy.

The stochastic objective is interpreted as its expectation; a literal pathwise minimization is generally impossible without knowing future noises. Let the value function from time $t$ be $V_t(x)=P_t{x}^2+d_t$, with $P_T=1,d_T=0$. Independence from past observations and zero contemporaneous covariance giveThe Bellman equation is $V_t(x)=x^2+{\min }_u\{c{u}^2+P_{t+1}{V}_{\xi }(Ax+u{)}^2+P_{t+1}{V}_{ϵ}+d_{t+1}\}$. Completing the square yieldsThis feedback is optimal among all controls measurable from the specified past, not merely linear controls. It depends on $V_{\xi }$ and does not depend on $V_{ϵ}$, though the minimal cost does. There is no cost-relevant $u_T$ in the displayed objective.

The backward Riccati recurrence iterates $F(P)=1+A^{2}c{V}_{\xi }P/(c+V_{\xi }P)$ from $1$. It is increasing, maps into $[1,1+A^{2}c]$, and has exactly one positive fixed point. Consequently it converges toAs $T\to \infty$, the finite-horizon feedback at any fixed time tends to this feedback. The closed-loop multiplier is $Ac/(c+V_{\xi }P)$, and its squared-mean factor $\alpha =V_{\xi }{A}^2{c}^2/(c+V_{\xi }P{)}^2$ is less than one, since $(P-1)/P=A^{2}c{V}_{\xi }/(c+V_{\xi }P)$ and $\alpha =(P-1)c/[P(c+V_{\xi }P)]$. Thus the second moment tends to $V_{ϵ}/(1-\alpha )$. The finite-horizon value has $d_1=V_{ϵ}{\sum }_{t=1}^{T-1}{P}_{t+1}$; its average converges by Cesaro summation. For a fixed initial state of finite second moment,The same average is attained by the stationary feedback, by the stationary Bellman equation and bounded second moments.