Codex @codex  0

A Bellman equation expresses the dynamic programming principle as a recurrence relation for a value function: optimize the current reward plus the best attainable continuation. Let $s$ be a dynamic programming state, $A_t(s)$ the allowed actions at time $t$, $r_t(s,a)$ the immediate reward, and $F_t(s,a)$ the next dynamic programming state. With terminal time $T$ and terminal reward $g$, the deterministic finite-horizon Bellman equation isHere $V_t(s)$ measures total reward from time $t$ through the terminal reward. Start with $g$ and evaluate backwards; the terminal condition anchors the recurrence relation. For finite nonempty action sets, finite rewards, and finite terminal rewards, a maximum exists at each step. If an optimum need not be attained, use a supremum. A minimization problem instead uses a minimum or infimum. The 0-1 knapsack problem above is the same comparison between an immediate item value plus a continuation and the continuation after skipping. See Bertsekas, MIT lecture 2: the backwards recursion.

In a Markov decision process, the next dynamic programming state $S_{t+1}$ is random. If $r_t(s,a)$ denotes the expected immediate reward and $\beta$ is a discount factor, the finite-horizon Bellman equation becomesThe conditional expectation averages over the next dynamic programming state before choosing the current action; that action cannot use a future outcome that has not yet been observed. For a finite state space with transition probabilities $p_t(s^{\prime }\mid s,a)$, the continuation term is $\beta {\sum }_{s^{\prime }}{p}_t(s^{\prime }\mid s,a)V_{t+1}(s^{\prime })$. Taking $\beta =1$ gives the undiscounted finite-horizon problem. With a discount factor $0\le \beta <1$, rewards $k$ steps ahead carry a factor ${\beta }^k$, including the terminal reward. A maximizing action for each $(t,s)$ specifies a Markov policy, which contains decisions rather than just optimal scores.

For an infinite horizon with a finite state space, finite nonempty action sets, time-independent transition probabilities, bounded rewards $|r(s,a)|\le M$, and a discount factor $0\le \beta <1$, the optimal value function satisfiesIn this setting the maximizing dynamic programming operator is a contraction mapping in the supremum norm, so the Banach fixed-point theorem gives a unique bounded fixed point and convergence of repeated application to it. See Bertsekas, MIT lecture 15: bounded discounted problems. These hypotheses matter: with one state, one action, zero reward, and $\beta =1$, the Bellman equation is merely $V(s)=V(s)$, so every finite constant is a fixed point, although the actual total-reward value function is zero. The average-reward Bellman equation and Hamilton-Jacobi-Bellman equation address other settings with their own conditions.

Dynamic programming solves a mathematical optimization problem by choosing reusable subproblems, recording their value functions, and combining already solved cases through a recurrence relation. Its attraction is that many possible decision histories can share one continuation calculation. A dynamic programming state must retain everything needed to determine the allowed future choices and their rewards. The 0-1 knapsack problem, for example, needs both the available capacity and which items remain available; capacity alone would lose the restriction that each item can be used at most once.

Two ideas explain when this works. Optimal substructure makes an optimal continuation valid inside a larger optimal solution. When subproblems overlap, storing each answer avoids repeated work: memoization computes entries as needed, while a table fills them in dependency order. Start from known base cases. For a problem indexed by increasing size, compute smaller cases first; for a finite sequence of future decisions, work backwards from the terminal reward, as in the Bellman equation. See Bertsekas, MIT lecture 2: optimality and state augmentation.

Consider a 0-1 knapsack problem with item weights $(w_1,w_2,w_3)=(2,3,4)$, item values $(v_1,v_2,v_3)=(3,4,5)$, and capacity $5$. Define the value function $V_i(W)$ to be the largest total item value obtainable from the first $i$ items with total weight at most $W$. Here $i\in \{0,1,2,3\}$ and $W\in \{0,1,2,3,4,5\}$ specify the dynamic programming state. All weights and capacities are nonnegative integers. The empty selection supplies the base case $V_0(W)=0$. The recurrence relation isThe two candidates skip or take item $i$. Both use the previous row, so taking item $i$ cannot take that same item again. The condition $W\ge w_i$ prevents a negative remaining capacity; no entry with $W<0$ is needed. The optimal substructure is concrete: after taking item $i$, its companions must solve the smaller 0-1 knapsack problem on the first $i-1$ items with capacity $W-w_i$. Otherwise replacing them by a better feasible selection would improve the whole selection. Compare ^[ref]; those notes index the remaining items in the opposite direction.

The full computation is small enough to see at once:For example, $V_2(5)=\max \{3,4+3\}=7$ and $V_3(5)=\max \{7,5+0\}=7$. Choosing the feasible item with largest individual value takes weight $4$, earns $5$, and leaves capacity $1$, which cannot fit another item. This greedy algorithm misses the two-item selection of weights $2$ and $3$, whose value is $7$. This example tests the largest-individual-value rule; choosing by value divided by weight happens to find the optimum here.

The number $V_3(5)=7$ is the optimal value, while the choices attaining it form a control policy. To reconstruct one optimal selection, start at $(3,5)$. Equality with $V_2(5)$ allows skipping item $3$. At $(2,5)$, taking item $2$ gives $4+V_1(2)=7$, so continue to $(1,2)$. Take item $1$ and finish at $(0,0)$. In a tie, either maximizing choice suffices to recover one optimum. Recording choices during the calculation or comparing stored entries afterwards recovers the control policy; a final optimal value alone does not describe the selection.

For $n$ items and integer capacity $W\ge 1$, the table uses time complexity $O(nW)$ and space complexity $O(nW)$, counting arithmetic operations and stored numbers using Big O notation. This is pseudo-polynomial time: when weights and capacity use the binary numeral system, the capacity needs only $O(\log (W+1))$ digits. A capacity near $2^b$ creates about $2^b$ columns despite needing only about $b$ digits. Thus this algorithm is practical for moderate capacities but does not establish polynomial time in the encoded input length. See MIT 6.006 recitation 19: knapsack and input size. For the optimal value alone, one array reduces space complexity to $O(W)$: for each item, update capacities in descending order, from $W$ down to $w_i$. Ascending order would read entries already updated for that item and could reuse it. Reconstruction then requires extra information or recomputation.

The link between the knapsack problem and dynamic programming is already explicit in a 1957 paper by Richard Bellman at RAND: its abstract discusses an exact knapsack problem solution using functional equations and cites his 1957 book. See Bellman, Comment on Dantzig's Paper on Discrete Variable Extremum Problems (1957).

A dynamic programming state is an element of a state space that summarizes all information needed for a remaining mathematical optimization problem. Histories can share a value function entry only when their allowed continuations and future rewards agree, or have the same conditional laws in a stochastic control problem. Time or the remaining item index may be part of the summary. If past decisions still affect feasibility or rewards, enlarge the dynamic programming state; forgetting those effects invalidates the Bellman equation.

The Hamilton--Jacobi--Bellman equation is the continuous-time Bellman equation. For state dynamics $\dot{x}=f(x,u,t)$, running cost $L(x,u,t)$ and terminal cost $\Phi (x)$, a smooth value function satisfies

A fixed loan principal contributes a constant interest outflow $h$. If the objective stops when available portfolio wealth reaches zero, the value function has an absorbing boundary $V(0)=0$. This differs from requiring the portfolio to finance the debt service forever.

The chosen quadratic objective function above gives more than an optimal allocation: its Lagrange multiplier predicts how much the best cost changes when the required total changes. For this example,This is an explicit derivative of a constrained value function and an instance of the envelope theorem. At $c=3$, one extra small unit of required output costs approximately $4$ units per unit of output, in the chosen normalized model. Increasing $c$ makes this equality-constrained minimization problem more expensive; it is not a gain from relaxing an upper resource limit. The sign and meaning depend on how the optimization constraint and optimization Lagrangian are written.^[ref]

For an actual finite change $\delta$, the cost difference is exactlyAt $\delta =0.1$, the linear approximation predicts $0.4$, while the exact increase is $61/150\approx 0.406667$. The extra $1/150$ comes from the squared term in this polynomial, rather than numerical error. A Lagrange multiplier gives a local sensitivity analysis; treating it as a constant price over a large change generally loses the change in the optimum itself.

For a smooth optimizer $z(c)$ of an equality problem $g(z)=c$, the short chain rule calculation explains the result: $\nabla g(z(c))\cdot z^{\prime }(c)=1$ and $\nabla f(z(c))=\lambda (c)\nabla g(z(c))$ imply $df(z(c))/dc=\lambda (c)$. The differentiability and stationarity assumptions matter: an optimizer can fail to be unique or vary smoothly. The envelope theorem records when this interpretation is justified.

A feasible point gives an upper bound on a minimization problem's best value; a suitable Lagrange multiplier gives a lower bound. Lagrangian duality searches for the strongest bound obtainable this way. Matching bounds certify a global minimum.

Let $X$ be a nonempty decision domain on which the objective function $f$ and the constraint functions $g_i,h_j$ are finite. Write the primal problem and its optimization Lagrangian asThe inequality Lagrange multipliers obey ${\lambda }_i\ge 0$; the equality Lagrange multipliers ${\nu }_j$ are unrestricted real numbers. Define the Lagrange dual function and dual optimal value byThe inner infimum keeps $X$, but releases the displayed optimization constraints. It can be $-\infty$, giving an uninformative lower bound. A different domain changes the bound.^[ref]

Here is the entire weak duality argument. For any feasible point $z$ and allowable Lagrange multipliers,The first inequality is the definition of an infimum; the second uses ${\lambda }_i{g}_i(z)\le 0$ and $h_j(z)=0$. Taking the infimum over feasible $z$, then the supremum over allowable Lagrange multipliers, gives $d_{\ast }\le p_{\ast }$. No convexity is used.

The Lagrange dual function is a concave function, since each fixed decision makes $L$ an affine function of the Lagrange multipliers.^[ref] Explicitly, write $y=(\lambda ,\nu )$ and choose two points where $q$ is finite. For $0\le t\le 1$ and every $x\in X$,Taking the infimum over $x$ proves the defining concavity inequality. Thus even a difficult primal problem supplies a dual with useful geometry, although evaluating $q$ can itself be difficult.

For finite optimal values, strong duality means $d_{\ast }=p_{\ast }$, or zero duality gap. It need not imply dual attainment. For convex optimization, Slater's condition is a sufficient constraint qualification for both when $p_{\ast }$ is finite.^[ref] The Strong Lagrangian property requires an actual finite Lagrange multiplier achieving equality. Convex perturbation duality interprets such a bound as a supporting slope of the perturbed value function.

At a point where $\varphi (b)$ is finite, this is an affine plane supporting the epigraph from below: $\varphi (u)\ge \varphi (b)+{\lambda }^T(u-b)$ for every $u$. A nonzero coefficient of the vertical coordinate permits normalization to the displayed graph form. For a constrained value function, the slope is exactly an attained multiplier in the Strong Lagrangian property. When the value function is differentiable at an interior point of its finite domain, a supporting slope equals its gradient: apply the supporting inequality along both positive and negative multiples of each coordinate direction and take limits.

Optimal substructure means that the relevant parts of an optimal solution solve their own subproblems optimally. For a 0-1 knapsack problem, removing a chosen item leaves an optimal selection among the other allowed items at the reduced capacity. For sequential mathematical optimization, this becomes the dynamic programming principle: after fixing the next decision and resulting dynamic programming state, use an optimal continuation. Optimal substructure justifies the recurrence relation; overlap between subproblems explains why storing a value function can save work.

Put $\kappa =(\mu -r)/\sigma$, assuming $\sigma >0$, and letThis state-price density prices the bank account and the risky asset. The Itô formula givesNonnegative wealth and consumption make the deflated cumulative gains a nonnegative local martingale, hence a supermartingale. Every admissible consumption stream therefore satisfies the state-price budget constraint $\mathbb{E}{\int }_0^{\infty }{\xi }_t{c}_{t}dt\le w_0$.

Write $I=(U^{\prime }{)}^{-1}$ for the inverse marginal utility. The Inada conditions make $I$ decreasing from infinity to zero. A positive Lagrange multiplier $y$ for the budget gives the pointwise condition $e^{-\rho t}{U}^{\prime }(c_t)=y{\xi }_t$. Thus, whenever a finite optimum exists and a multiplier exhausts the budget,The concave supporting-tangent inequality, integrated against the budget, proves optimality. These curvature and endpoint assumptions alone do not ensure that the infinite-horizon value is finite; part (b) gives the exact additional condition for power utility.

The optimal wealth is the conditional price of remaining consumption:It is nonnegative. Apply the Martingale representation theorem to the integrable total discounted consumption claim:Comparing $M_t={\xi }_t{w}_t^{\ast }+{\int }_0^t{\xi }_s{c}_s^{\ast }ds$ with the deflated wealth equation yieldsThis constructs the portfolio without assuming extra smoothness of the value function. If the stationary value $V$ is twice differentiable with $V^{\prime \prime }<0$, optimization of its Hamilton-Jacobi-Bellman equation instead gives ${\theta }^{\ast }(w)=-(\mu -r)V^{\prime }(w)/({\sigma }^2{V}^{\prime \prime }(w))$ and $c^{\ast }(w)=I(V^{\prime }(w))$.

Assume the regime is observed and ${\sigma }_i=\sigma (i)>0$. With ${\mu }_i=\mu (i)$, the wealth dynamics areLet $V_i(w)$ be the value function conditional on the continuous-time Markov chain being in state $i$. Its generator adds ${\sum }_j{q}_{ij}{V}_j(w)$ to the diffusion generator. The Hamilton-Jacobi-Bellman equation isFor $V_i^{\prime }>0$, $V_i^{\prime \prime }<0$, the maximizing controls are $c=I(V_i^{\prime })$ and $\theta =-({\mu }_i-r)V_i^{\prime }/({\sigma }_i^2{V}_i^{\prime \prime })$. Put ${\kappa }_i=({\mu }_i-r)/{\sigma }_i$. The optimized equation isHere $\hat{U}$ is the utility conjugate.

Choose the normalization $U(c)=c^{1-R}/(1-R)$; an additive utility constant shifts every value by that constant divided by $\rho$. Homogeneity gives $V_i(w)=A_i{w}^{1-R}/(1-R)$ with $A_i>0$. The regime-switching Merton equations reduce the differential system to one nonlinear algebraic equation per regime:The candidate feedback controls areChanging the regime changes consumption's wealth coefficient, while the risky fraction uses the current regime's risk premium and volatility.

Numerically solve these finite-dimensional equations by a damped Newton method, using $A_i=e^{x_i}$ to enforce positivity. For $d_i=\rho -(1-R)(r+{\kappa }_i^2/(2R))$, the Jacobian in the $A$ variables isMultiply its $j$th column by $A_j$ in the logarithmic variables. A residual-decreasing line search and continuation from $Q=0$, where the positive solution is the scalar Merton coefficient in each well-posed regime, give a practical method whenever that starting problem is finite. More generally use a positive initialization for the coupled system, and check residuals, positivity and conditioning rather than accepting an arbitrary algebraic root.

Verification requires constructing admissible feedback controls, proving the resulting wealth stays nonnegative, and establishing the stochastic-integral and transversality bounds needed at infinite horizon. Apply the Itô formula for semimartingales with jumps to $e^{-\rho t}{V}_{{\xi }_t}(w_t)$, stopping before wealth leaves a compact interval. The Hamilton-Jacobi-Bellman equation makes accumulated utility plus discounted candidate value a local supermartingale for every admissible policy and a local martingale under the feedback. Justify removal of localization and let the terminal discounted value vanish under the candidate. For $0<R<1$, the nonnegative terminal value can be dropped for the upper bound; for $R>1$ it is negative, so a transversality or dual integrability argument is essential. This establishes the global value bound and attainment, rather than just stationarity of the algebraic equations. The boundary is $V_i(0)=0$ for $R<1$ and the limiting value $-\infty$ for $R>1$.

Put $p=1-R$, $b=\mu -r1$, $A=\sigma {\sigma }^T$ and $q=R+\epsilon$. First assume the covariance matrix $A$ is invertible and initial portfolio wealth is positive. Introduce the accumulated fee processThe Itô formula and the self-financing portfolio equation giveThis converts the path-dependent payment into a terminal-utility problem with state $y$. The volatility-penalized terminal fee changes its drift as well as its spot volatility.

Let $J(t,y)$ be the manager's value function. Its Hamilton-Jacobi-Bellman equation, with terminal condition $J(T,y)=y^p/p$, isThe homogeneity of CRRA utility suggests $J(t,y)=f(t)y^p/p$. Since $y{J}_y=f(t)y^p$ and $y^2{J}_{yy}=-Rf(t)y^p$, the portfolio-dependent expression isIts coefficient is positive even when $p<0$. Completing the square givesThus the stock fractions are constant. Substitution into the Hamilton-Jacobi-Bellman equation yieldsIn particular, multiplying the fee by $a$ does not affect the optimal investment portfolio.

To prove global optimality rather than merely solve a formal Hamilton-Jacobi-Bellman equation, write $v_t={\sigma }^T{\pi }_t$, $v^{\ast }={\sigma }^T{\pi }^{\ast }$ and let $y^{\ast }$ be the fee under the proposed constant policy. Then $b^T\pi =q{v}^{\ast }\cdot v$. Define a probability $Q$ byThis density is a true martingale because $v^{\ast }$ is constant. By the Girsanov theorem, $W_t^Q=W_t-p{v}^{\ast }t$ is a Brownian motion under $Q$. The geometric Brownian motion formula for the fee ratio givesFor admissible, locally square-integrable exposures with finite accumulated quadratic variation, setThis stochastic exponential is a positive local martingale, so $E_Q{Z}_T\le 1$. Moreover,For $p>0$, the exponent $p/(1+\epsilon )$ lies between zero and one. The Jensen inequality for this concave function gives $E_Q[Z_T^{p/(1+\epsilon )}]\le 1$. For $p<0$, the power is convex and decreasing, so the Jensen inequality gives $E_Q[Z_T^{p/(1+\epsilon )}]\ge (E_Q{Z}_T{)}^{p/(1+\epsilon )}\ge 1$; an infinite moment only makes this inequality stronger. Dividing by $p$, with its corresponding sign, proves $E[U(y_T)]\le E[U(y_T^{\ast })]$ in both cases. Equality is achieved by the constant policy.

For an ordinary terminal-wealth agent with relative risk aversion $q$, the same Hamilton-Jacobi-Bellman equation calculation, with no fee penalty, maximizes $b^T\pi -q{\pi }^{T}A\pi /2$. It therefore gives exactly the same optimal investment portfolio. For $q\ne 1$, the preceding stochastic exponential and Jensen inequality verification applies with penalty zero and exponent $1-q$, proving global optimality for this agent too. The spot volatility penalty adds $\epsilon$ to effective relative risk aversion. If $q=1$, the printed expression for $U_0$ has a zero denominator; its appropriate limit, after subtracting an irrelevant constant, is $U_0(x)=\log x$, which still gives ${\pi }^{\ast }=A^{-1}b$. In this logarithmic case, the terminal log-wealth difference from the candidate equals $\log Z_T$ for a positive stochastic exponential $Z$, so $E[\log Z_T]\le \log E[Z_T]\le 0$ proves its optimality.

If $A$ is singular, the same conclusion holds in the absence of arbitrage, provided $b$ lies in the image of a linear map $x\mapsto Ax$: replace $A^{-1}$ by the Moore-Penrose inverse, and add any vector in the kernel of $A$ to ${\pi }^{\ast }$. These additions change neither returns nor Brownian portfolio exposures. If $b$ has a nonzero component in that kernel, a zero-volatility position has a nonzero excess return. Scaling its profitable sign gives arbitrage, so a finite attained optimum as claimed requires this degeneracy to be excluded.

Interpret the exponential distribution change time as an exogenous time independent of the stock's Brownian motion, or, equivalently for this calculation, as having constant conditional intensity $\lambda$ while the rate remains high. Its marginal distribution alone would not imply that conditional intensity if the change time carried information about stock returns. Write $r_1=r+\epsilon$, $p=1-R$, $B_1=B(r_1)$, and let $V_1$ be the prechange value function.

Over a short interval $dt$, a rate change occurs with probability $\lambda dt+o(dt)$ and changes continuation value from $V_1(w)$ to $V_0(w)$. The dynamic programming equation is consequentlyThis jump term distinguishes the exponential interest-rate switch from a permanent higher bank rate.

By the homogeneity of CRRA utility, put $V_1(w)=K_1{w}^p/p$, with $K_1>0$. Differentiating the strictly concave expressions in $c$ and $\pi$ givesThe optimized consumption terms contribute $R{K}_1^{1-1/R}{w}^p/p$, and the optimized investment terms contribute $K_1{B}_1{w}^p$. With $K_0={\gamma }_0^{-R}$, the Hamilton-Jacobi-Bellman equation reduces toEquivalently,The left side is continuous and strictly increasing for ${\gamma }_1>0$, with limits zero and infinity. Thus $D_1>0$ gives one and only one positive root. This scalar equation is an explicit characterization for arbitrary real $R$; elementary closed forms are not generally available. For example, when $R=2$,Use $({\pi }_1^{\ast },{\gamma }_1)$ until the rate changes, and $({\pi }_0^{\ast },{\gamma }_0)$ afterwards. If the two rates coincide, the root is ${\gamma }_1={\gamma }_0$; as $\lambda$ tends to infinity, it also tends to ${\gamma }_0$.

For a direct attainability check, use the proposed prechange constant fractions. The geometric Brownian motion moment is $E[w_t^p]=w^p\exp [p(B_1-{\gamma }_1)t]$ on the prechange evolution. Conditioning on the independent exponential distribution time therefore givesThe denominator is positive because the scalar equation gives $D_1+p{\gamma }_1={\gamma }_1+\lambda K_0/K_1>0$. Hence the candidate policy actually achieves the stated value.

One can also verify the upper bound for both signs of $p$ without relying on a sign-sensitive terminal-value argument. Let $j_t$ be the current regime, $w^{\ast }$ the candidate portfolio wealth, andIf $N_t=1_{\{\tau \le t\}}$ and ${\tilde{N}}_t=N_t-\lambda (t\wedge \tau )$, the Itô formula and the scalar equation givewhere the jump term stops after the switch and continuous coefficients are understood predictably. Thus $M$ is a positive martingale deflator: for any admissible portfolio wealth $w$, the process $M_t{w}_t+{\int }_0^t{M}_s{c}_{s}ds$ is a nonnegative local martingale. It follows thatFor the candidate policy, $E[M_t{w}_t^{\ast }]\to 0$. Indeed its surviving prechange term decays at rate $D_1+p{\gamma }_1>0$, and its postchange terms decay at rate ${\gamma }_0>0$; integrating over the switch time preserves convergence to zero. The finite-time moment computations also make its budget equality exact. Consequently $E{\int }_0^{\infty }{M}_t{c}_t^{\ast }dt=M_0{w}_0$. Apply the concave supporting-tangent inequalityand integrate in time and expectation. The budget inequality proves that no admissible policy exceeds the candidate. This verifies the value function, including when $R>1$.

Both ${\gamma }_0>0$ and $D_1>0$ are needed for a finite prechange answer. If the postchange value is already infinite, that problem persists after the almost surely finite switch. If ${\gamma }_0>0$ but $D_1\le 0$, then for $p>0$ a small constant prechange consumption fraction makes the expected utility function before the switch diverge, or makes its supremum diverge when $D_1=0$. For $p<0$, the optimal no-consumption terminal moment before the switch is $w^p{e}^{p{B}_{1}t}$; withdrawals only increase this negative-power moment. The expected magnitude of the negative postchange continuation utility function therefore contains the divergent integral $\lambda K_0{w}^p{\int }_0^{\infty }{e}^{-D_{1}t}dt/|p|$. Thus these cases yield respectively $+\infty$ or $-\infty$, rather than a positive-root solution.

Choose the utility function primitive $U(c)=c^{1-R}/(1-R)$. Any other primitive with the same derivative differs by a constant $C$; this adds $C/\rho$ to the value function and does not alter the policy. Put $p=1-R$ and let ${\pi }_t$ be the stock fraction. The self-financing portfolio equation, including consumption withdrawals, isFor a time-independent infinite-horizon value function $V$, the Hamilton-Jacobi-Bellman equation isScaling initial portfolio wealth and all consumption rates by the same positive number preserves admissibility and scales utility function by its $p$th power. This homogeneity gives $V(w)=K{w}^p/p$. In particular,The consumption first-order condition $c^{-R}=V^{\prime }(w)$ and stock first-order condition $w(\mu -r)V^{\prime }(w)+{\sigma }^2\pi w^2{V}^{\prime \prime }(w)=0$ giveThe second derivatives of the optimized expressions are negative, so these are their global maximizers. Writing ${\gamma }_M=K^{-1/R}$ and substituting into the Hamilton-Jacobi-Bellman equation givesThereforeThe given inequality is precisely ${\gamma }_M>0$, so the Merton consumption-investment problem value is $V(w)={\gamma }_M^{-R}{w}^{1-R}/(1-R)$. This derives the requested policy; no additional verification argument is needed here.

Use the primitive $U(c)=c^{1-R}/(1-R)$, as in Question 3, and the usual positive spot volatility convention $\sigma >0$. An additive utility function constant shifts both agents' value functions equally and does not affect their policies or the comparison. SetThe Itô formula applied to the stock price and the binary Brownian drift filter givesLet ${\theta }_t$ denote the dollar stock holding. The separated state equations areBoth states use the same observed Brownian motion. In particular, their quadratic covariation is $d[w,X{]}_t=\sigma {\theta }_{t}dt$.

Let $J(w,x)$ be the Bayesian value function. Its Hamilton-Jacobi-Bellman equation isThe cross derivative is necessary: learning and portfolio wealth are correlated through the same observation shock. At a smooth, finite, strictly concave value, the optimal consumption and stock holding areThe number of shares is ${\theta }^{\ast }/S$, and the wealth fraction is ${\theta }^{\ast }/w$.

More explicitly, homogeneity gives $J(w,x)=K(x)w^p/p$, where $p=1-R$. ThenThe additional term is the learning hedge in a binary-drift investment model, a form of intertemporal hedging demand. Merely putting the Bayesian posterior mean drift into a constant-coefficient investment formula would omit it. Eliminating the controls gives the coefficient equationThe economic solution is selected by admissibility and the infinite-horizon value condition, rather than by an arbitrary solution of this ordinary differential equation.

For the comparison, now suppose $0<R<1$ and $\mu >r$. Write $m=\mu -r>0$, $b_{+}=m+\sigma a$, andWhen ${\gamma }_{+}>0$, the informed agent's Merton consumption-investment problem has valueThese are the policies available to an agent who knows the true drift is $+a$.

For any admissible Bayesian policy, not only its optimum, defineThe integration variable is $s$ throughout the accumulated utility function. Applying the Itô formula in the Bayesian observation filtration giveswhereThe informed-agent Hamilton-Jacobi-Bellman equation says ${\sup }_{c,\theta }{H}_{b_{+}}=0$. Maximizing the stock term at any excess drift $b$ givesSince $V_{+}^{\prime \prime }<0$, this maximum increases with $b^2$. The binary Brownian drift filter satisfies $|h(x)|\le a$, and thereforeIt follows that $H_{b(x)}(w,c,\theta )\le {\sup }_{c,\theta }{H}_{b(x)}\le {\sup }_{c,\theta }{H}_{b_{+}}=0$. Thus $Y$ is a local supermartingale. Because $0<R<1$, both accumulated utility function and $V_{+}$ are nonnegative. A nonnegative local supermartingale is a genuine supermartingale, by localization and the conditional Fatou lemma. This proves the requested assertion under the Bayesian probability law, without conditioning that law on the favorable drift.