Past exam of the options II course of the University of Oxford / 2011 / M / M5 / b / i / Solution

Use the state space $\{1,\dots ,N\}$, since an empty box is refilled immediately. The probability that formerly led from $i$ to $0$ now leads to $N$. Thus the transition matrix isFor $i=N$, the second line includes the self-loop obtained when all stickers are taken. When $N=1$, this is simply the one-state Markov chain.