 Solution

Let $N(u)$ count all jumps of the continuous-time Markov chain. Its Markov-chain entry count into state $s$ isThusThis is a Lebesgue-Stieltjes integral against a counting process, so each jump contributes one. The initial occupation of state $s$ is not counted as an entry. An ordinary time integral of an indicator random variable asserting a jump at exactly time $u$ would be zero, since individual jump times have zero Lebesgue measure. The requested integral must therefore use this counting-measure interpretation. Assume non-explosion and finite expected jump counts on bounded intervals, as holds for a finite-state continuous-time Markov chain.