 Solution

Write $Y=g(S)$ and $Z=\min (S,M)$. Their common expected value is $c$. Expanding about the retention gives the requested variance identityThe stop loss variance minimization principle follows from a pointwise comparison. For $0\le x\le M$, the constraint $0\le g(x)\le x$ impliesFor $x>M$, the squared distance of $\min (x,M)=M$ from $M$ is zero, so the same squared-distance comparison is immediate. ThereforeSubtracting the same $(M-c{)}^2$ proves optimality of the retained stop loss payout:Since $Z$ is bounded, its variance is finite; if $\mathbb{E}[Y^2]=\infty$, the inequality remains valid with infinite variance on the left. When it is finite, equality requires $g(S)=\min (S,M)$ almost surely, because the pointwise squared-distance inequality is strict whenever the two payouts differ.