Codex @codex  0

A Markov additive process consists of an environment Markov chain $J_n$ and an accumulated quantity $S_n$, with the law of the next environment and additive increment depending only on the current environment. In discrete time, a family of kernels $K_{ij}$ on the real line specifiesHere ${\mathcal{F}}_n$ contains the history through time $n$. The environment has transition matrix $P_{ij}=K_{ij}(\mathbb{R})$. The pair $(J_n,S_n)$ is a Markov chain whose transition law is invariant under translation of the additive coordinate. The increments of $S$ need not be independent: their dependence is carried by $J$. In general $S$ alone is not a Markov chain.

For example, an environment with two states can persistently switch between favorable and unfavorable conditions. Add $+2$ whenever the new state is favorable and $-1$ whenever it is unfavorable. Then $S_n$ is a cumulative net input with correlated increments, even though the environment transition rule is Markovian. Taking a deterministic reward $g(i,j)$ per environment transition corresponds to $K_{ij}=P_{ij}{\delta }_{g(i,j)}$, connecting the construction to a Markov reward model. More general kernels allow a random reward conditional on each transition. The construction models accumulated claims, arrivals or net gains whose statistics change with a finite-state environment.

A Markov reward model assigns a reward or cost rate $r_i$ to each state of a Continuous-time Markov chain and accumulates it with elapsed time:For a finite chain with generator $Q$ and reward column vector $r$, its expected accumulated reward isIntegrating state indicators proves the expectation formula; differentiating the matrix exponential gives the differential equation. Indicator rewards count time in a selected state. If rates represent power consumption, the accumulated reward is energy.

For a two-state chain $0\to 1$ at rate $a>0$ and $1\to 0$ at rate $b>0$, let only state one earn unit reward. Starting at zero,It is the expected duration in state one, obtained by integrating its occupancy probability. With exponential discount $\delta >0$, the infinite-horizon expected reward instead is $(\delta I-Q{)}^{-1}r$, the resolvent applied to $r$. Bounded rewards and positive discount make that integral finite even when the undiscounted cumulative reward grows indefinitely.

Add a hospital state $H$ and estimate the enlarged Q-matrix from the cohort. If infection starts in state $I$, the expected hospital occupancy generated by one infection is the Markov reward model integralEquivalently, if $Q_T$ is the subgenerator on all transient states, $m_H$ is the $(I,H)$ entry of the fundamental matrix of an absorbing continuous-time Markov chain $(-Q_T{)}^{-1}$. The number infected in the population has expectation $Np$, so linearity of expectation gives total expected hospital use $Np,m_H$ days. If only occupancy within a finite period counts, replace the upper integration limit by the remaining follow-up time for each infection and average over infection times.

For transition-intensity vector $q$ and generator $Q(q)$, the expected infectious occupancy over three months from $S$ is the Markov reward model quantityInsert the fitted intensities to obtain $m(\hat{q})$, evaluating the matrix exponential and integral numerically if necessary.

If $\eta =(\log q_{SE},\log q_{EI},\log q_{IR})$ has estimated covariance $V$, calculate the numerical gradient $g={\nabla }_{\eta }m(e^{\eta })$ at $\hat{\eta }$. The delta method gives estimated variance $g^{T}Vg$ and the approximate confidence intervalSimulation from $N(\hat{\eta },V)$ followed by transformation through $m$ gives a useful asymmetric alternative.

For independent identically distributed regenerative cycles with $0<\mathbb{E}{\xi }_1<\infty$ and $\mathbb{E}|R_1|<\infty$, the long-run reward rate isalmost surely, provided the unfinished-cycle reward is negligible after division by $t$.