Optimal control

Choice of a control input to minimize an objective subject to differential equations for a state and constraints on the control, state or terminal time. Necessary conditions such as the Pontryagin maximum principle require regularity hypotheses and do not by themselves prove global optimality.

For bank dynamics $\dot{x}=A+rx-u$ and utility ${\int }_0^T{e}^{-\beta t}\log udt+\mu e^{-\beta T}x(T)$, substituting the terminal balance separates the concave optimization pointwise and gives the displayed control for $\mu >0$. Put $W=e^{rT}x(0)+A(e^{rT}-1)/r$ and $B={\int }_0^T{e}^{\beta s}ds$. Its terminal balance is $W-B/\mu$, so the unique positive multiplier achieving zero terminal wealth is $B/W$. The same control solves the problem with no terminal reward and constraint $x(T)\ge 0$, by the supporting-line inequality for logarithms.

For $X_{n+1}=X_n+U_n+{\epsilon }_{n+1}$ and $Y_{n+1}=X_n+{\eta }_{n+1}$ with independent unit-variance normal noises, the equilibrium prediction-error variance is $P=(1+\sqrt{5})/2$. Orthogonality to the next innovation gives the filtering gain $H=P/(1+P)$. Minimizing the quadratic average cost gives a control Riccati coefficient $R$ satisfying $R^2=R+1$, so $R=P$ and $K=R/(1+R)=H$. Completing the control square yields minimal cost $R+(1+R)K^{2}P=1+\sqrt{5}$. The one-step observation delay is essential to the value of $P$.

Quantum optimal control minimizes a state-transfer, observable, or gate error together with resource costs, subject to a controlled Schrödinger equation or Lindblad equation. Forward propagation of the state and backward propagation of a costate compute the gradient efficiently. The optimum depends on the admissible time-dependent fields and laboratory constraints.

For a real quantum optimal control objective $J=⟨⟨A|{\rho }_v(T)⟩⟩-\mathrm{Re}\int ⟨⟨A_v|{\dot{\rho }}_v-K[f]{\rho }_v⟩⟩dt-{\sum }_m{c}_m\int f_m^{2}dt/2$, first variation gives the state equation, the adjoint equation, $A_v(T)=A$, and $c_m{f}_m=\mathrm{Re}⟨⟨A_v|{\partial }_{f_m}K|{\rho }_v⟩⟩$. The adjoint is taken for the Hilbert-Schmidt inner product. With dissipation it is essential to use $K^{†}$, not silently the same generator as the forward state. These are necessary stationarity equations; they do not establish global optimality.

For $J=⟨\psi (T)|A|\psi (T)⟩-(\lambda /2){\int }_0^T{f}^{2}dt$ and $i\hslash \dot{\psi }=(H_0+f{H}_1)\psi$, enforce the state equation with $-2\mathrm{Re}\int ⟨\chi |\dot{\psi }+iH\psi /\hslash ⟩dt$. Variation gives the same Schrödinger equation for the costate, propagated backward from $\chi (T)=A\psi (T)$. Control variation gives the displayed real gradient. Forward state propagation, backward costate propagation, and a constrained gradient update form direct-adjoint looping. These are necessary stationarity equations, not a guarantee of a global optimum.

The Cauchy-Schwarz inequality for the Hilbert-Schmidt inner product gives $0\le J\le 1$ for $n$-dimensional unitary operators. Equality $J=0$ holds exactly when $U=e^{i\varphi }{U}_T$. When absolute gate phase matters, use the phase-sensitive Hilbert-Schmidt distance $\Vert U-U_T{\Vert }^2/(2n)=1-\mathrm{Re}\mathrm{Tr}(U_T^{†}U)/n$ instead.

Parametrize a reproducible pulse by a finite vector $a$ and evaluate its objective experimentally on freshly prepared systems. Repeatedly update the pulse from measured objective values or finite-difference derivatives, including penalties for resources. This is closed-loop control across experimental runs, even when each pulse is open-loop control within a run. It reduces reliance on an inaccurate dynamical model but still requires preparation reproducibility, sufficient measurement statistics and stability of the experimental conditions.

Gradient ascent pulse engineering parametrizes each control by time-slice amplitudes and optimizes a differentiable quantum-control objective. Forward products of slice unitary operators and backward adjoint propagation give derivatives with respect to every amplitude. For $U_j=e^{-i{H}_j\Delta t}$, the exact derivative is ${\partial }_{f_m}{U}_j=-i{\int }_0^{\Delta t}{e}^{-i{H}_j(\Delta t-\tau )}{H}_m{e}^{-i{H}_j\tau }d\tau$. Updating all slice amplitudes along these derivatives is gradient ascent for a fidelity, or gradient descent for its negative.

For wealth dynamics $\dot{x}=rx+\ell -c$ and utility $c^{\alpha }(1-\ell {)}^{\beta }$, with $0<\alpha ,\beta <1$ and $\alpha +\beta >1$, the unconstrained interior control equations do not give the maximum. Put $s=1-\ell$. At positive shadow price $p$, maximization over $c$ leaves a function of $s$ proportional to $s^{\beta /(1-\alpha )}-ps$, up to constants. It is convex and hence maximal at $s=0$ or $s=1$. The optimal controls work with zero consumption first, then cease work and consume at $c(t)=C{e}^{rt/(1-\alpha )}$. The switch occurs where this consumption reaches $\alpha /(1-\alpha )$; the full discounted budget fixes $C$ and the switch time. The decreasing-consumption interior stationary candidate violates the Hamiltonian maximization condition.

A stochastic control problem chooses an admissible control input using information available in a filtration to optimize an expected objective for a random dynamical system. Dynamic programming often characterizes its value function through a Hamilton-Jacobi-Bellman equation.

A controlled Markov process has transition probabilities or rates depending on its current state and the chosen control action. A Markov policy selects an action from the current state and possibly the time. Once such a policy is fixed, the induced state evolution is a Markov process. For general history-dependent controls, the state alone need not remain Markov. A dynamic programming operator optimizes over the available actions at each state.

A Markov decision process specifies states, admissible actions, action-dependent transition probabilities and rewards. A policy chooses actions from observed information; a time-homogeneous problem uses transition and reward rules independent of the date. A Bellman equation optimizes current reward plus the conditional value of subsequent decisions. For nonnegative rewards, finite-horizon values increase as the horizon grows, while identification of their limit with the infinite-horizon value requires an approximation argument for admissible policies.

Estimating the expected return under a specified control policy. It is distinct from improving or optimizing that policy. A Bellman expectation equation describes the fixed-policy value, whereas a Bellman optimality equation compares admissible actions.

Policy iteration alternates evaluation of a fixed decision policy with improvement of its actions using the resulting value function. A stable policy satisfies the Bellman equation when each chosen action attains the required optimum.

In singular stochastic control, a control is a finite-variation process rather than only a rate integrated against time. Its increments can have atoms or a part singular with respect to time. The Hamilton-Jacobi-Bellman equation becomes a variational inequality with a constraint on directional derivatives of the value function.

Direct-adjoint looping repeatedly integrates a constrained state equation forward, integrates its costate or formal adjoint backward using terminal data, extracts the control gradient, and updates the control input within its constraints. It gives a practical route to local optimization but does not by itself prove a global minimum.

For a differentiable objective on an admissible real Hilbert space $H$, with $\Vert {\mathbf{u}}_0{\Vert }^2/2=E_0>0$, stationarity on the sphere in a normed vector space of fixed kinetic energy means that the projected gradient is normal to the sphere in a normed vector space. Thus $P_H\nabla J=\lambda {\mathbf{u}}_0$ for a real Lagrange multiplier. It follows by testing all tangent variations orthogonal to ${\mathbf{u}}_0$. The multiplier need not be positive; second-order conditions are still needed for a local minimum.

A control input taking only extreme admissible values except possibly on a switching or singular set. If the Hamiltonian of an optimal-control problem is affine in a scalar control $u\in [0,1]$, a nonzero coefficient selects $u=0$ or $u=1$; a zero coefficient does not determine the control.

An adjoint variable enforcing the state dynamics in optimal control. In the normal minimization convention it satisfies $\dot{p}=-H_x$ for the Hamiltonian of an optimal-control problem, with terminal conditions determined by the endpoint constraints.

In a normal minimization convention for optimal control, the running cost $L$ and dynamics $\dot{x}=f$ define $H=L+p\cdot f$. The multiplier $p$ is a costate; this Hamiltonian is a device for variational necessary conditions rather than necessarily a physical energy.

For a regular finite-horizon optimal control problem, an optimal trajectory admits nontrivial multipliers and a costate obeying adjoint equations, together with pointwise optimization of the Hamiltonian of an optimal-control problem. In the normal minimization convention $H=L+p\cdot f$, the conditions are $\dot{x}=H_p$, $\dot{p}=-H_x$ and minimization of $H$ over admissible controls; endpoint and state constraints change the terminal conditions. Abnormal extremals use a zero cost multiplier and must not be silently excluded.

For a closed planar path with $\dot{x}=u$, $\dot{y}=v$ and $\dot{z}=uy-vx$, the signed area satisfies $z(1)=-2$ times the usual oriented area. Periodic Wirtinger and Cauchy-Schwarz inequalities give $|z(1)|\le \frac{1}{2\pi }{\int }_0^1(u^2+v^2)dt$. Thus terminal $z(1)=1$ requires cost at least $\pi$. A clockwise circular path traversed once at constant speed attains equality, proving the global minimum rather than merely a stationary Pontryagin extremal.

For energy remaining $x$, $\dot{x}=-u$, $0\le u\le 1$, running cost $up(t)$ and terminal cost $-R(\tau )$, the normal maximizing Hamiltonian is $u(p_{\ast }-p(t))$, where the constant costate defines $p_{\ast }$. Hence full power is chosen below the threshold and zero power above it. Free completion time on an active terminal arc gives $p_{\ast }+R^{\prime }(\tau )=p(\tau )$. Global optimality still requires a comparison of feasible completion times; the maximum principle is only a necessary condition.

For a smooth normal optimal control problem with freely variable terminal time, fixed terminal state independent of that time and terminal cost $\Phi$, the optimized Hamiltonian of an optimal-control problem at completion satisfies $H(T)+{\Phi }_T=0$. It follows by varying the terminal time in the augmented objective; restrictions on terminal time or a moving endpoint introduce additional terms or inequalities.