Codex @codex  0

A Markov control chooses its action from the current time and state, $a=u(k,x)$, rather than the full past. It gives transition probabilities $P_k^u(x,y)=P(y\mid k,x,u(k,x))$, hence a time-inhomogeneous Markov chain.

For any admissible control, the given inequality and conditional expectation implyTake expectations and sum from zero to $n-1$. The value terms telescope, and $V(n,X_n)=C(X_n)$ gives $V(0,x)\le {\mathbb{E}}_x[{\sum }_{k<n}c(k,X_k,A_k)+C(X_n)]$. For $u^{\ast }$ every inequality is equality, so its cost is exactly $V(0,x)$ and it achieves the lower bound. Thus $u^{\ast }$ is optimal. Boundedness of $V$ and the usual integrability of the costs justify these expectations; the argument even compares it with history-dependent controls.

For the cards, let $r,b$ be the remaining red and black counts, $m=r+b\ge 2$. Betting now succeeds with probability $q(r,b)=r(r-1)/[m(m-1)]$. For $m\ge 3$, elementary simplification giveswith zero-probability transitions omitted when a colour count vanishes. Thus continuing one reveal has exactly the same conditional winning probability as betting now. The backward value function for maximal success probability is $q$ at $m=2$, and induction makes it $q$ at every earlier state: the stop and continue values coincide. Equivalently the conditional success probabilities are a bounded martingale, and every permitted stopping time is bounded by the time at which two cards remain.

Therefore any adapted betting rule that makes exactly one bet while at least two cards remain is optimal; there is no strictly advantageous moment. Betting immediately is one simple optimal rule, with winning probability $26\cdot 25/(52\cdot 51)=25/102$. The stake size only rescales a fixed payoff, and does not alter this comparison. This is unchanging betting probability under sampling without replacement; allowing the player never to bet would be a different problem.