Solution (source code)

= Solution

Let $K(t)=e^{tR}Se^{-tR}$. The <product rule> and differentiation of the <matrix exponential> give
$$
K'(t)=RK(t)-K(t)R=[R,K(t)],\qquad K''(t)=[R,[R,K(t)]],
$$
where brackets are the <operator commutator>. Evaluating at zero gives $K(0)=S$, $K'(0)=[R,S]$ and $K''(0)=[R,[R,S]]$. The <Taylor series> at zero is consequently
$$
\boxed{e^{tR}Se^{-tR}=S+t[R,S]+\frac{t^2}{2}[R,[R,S]]+O(t^3).}
$$
In particular $[R,[R,S]]=R^2S-2RSR+SR^2$, which also follows by multiplying the two <matrix exponential> series through quadratic order. For fixed finite <matrices> the remainder has the stated <Big O notation> in any <matrix norm>.