Solution (source code)

= Solution

The rotation <unitary operator> is \b[$U(\mathbf n,\theta)=\exp(-i\theta\,\mathbf n\cdot\mathbf J/\hbar)$.] A state with zero <orbital angular momentum> has a rotationally invariant orbital factor; a spinless $j=0$ state is wholly unchanged. Zero orbital angular momentum alone does not rule out a nontrivial rotation of intrinsic <spin angular momentum>.

For <spin one-half>, $\mathbf J=\hbar\boldsymbol\sigma/2$. Direct multiplication of the <Pauli matrices> gives $\sigma_i\sigma_j=\delta_{ij}I+i\epsilon_{ijk}\sigma_k$. The antisymmetric part vanishes when contracted with $a_ia_j$, hence
$$
\boxed{(\mathbf J\cdot\mathbf a)^2=\frac{\hbar^2|\mathbf a|^2}{4}I}.
$$
For $\mathbf a\ne0$, the operator has trace zero, so its <eigenvalues> are $\pm\hbar|\mathbf a|/2$; for $\mathbf a=0$ both are zero. With unit $\mathbf n$, the even and odd powers in the exponential give
$$
U(\mathbf n,\theta)=I\cos(\theta/2)-i\,\mathbf n\cdot\boldsymbol\sigma\sin(\theta/2).
$$

Take $\mathbf n'=(\sin\theta,0,\cos\theta)$. The rotated state $|m'\rangle_\theta=U(\mathbf y,\theta)|m'\rangle$ satisfies \b[$(\mathbf n'\cdot\mathbf J)|m'\rangle_\theta=m'\hbar|m'\rangle_\theta$], because $U J_3U^\dagger=\mathbf n'\cdot\mathbf J$. By the <Born rule>, the probability of this outcome in the initial state $|m\rangle$ is $|{}_\theta\langle m'|m\rangle|^2=|\langle m|U(\mathbf y,\theta)|m'\rangle|^2$. In the $J_3$ eigenbasis,
$$
U(\mathbf y,\theta)=\begin{pmatrix}\cos(\theta/2)&-\sin(\theta/2)\\\sin(\theta/2)&\cos(\theta/2)\end{pmatrix}.
$$
The transition from $m=1/2$ to $m'=-1/2$ thus has probability $\sin^2(\theta/2)=1-\cos^2(\theta/2)$. \b[$\boxed{A=1,\quad B=-1}$.]