Solution (source code)

= Solution

Use orthonormal cylindrical bases in the reference and deformed configurations. The radial coordinate remains $\rho$, the angular coordinate becomes $\theta=\Theta+\alpha\xi_3$, and the axial coordinate is unchanged. Removing the rotation of the cylindrical basis, the <deformation gradient> is
$$
A=\begin{pmatrix}1&0&0\\0&1&s\\0&0&1\end{pmatrix},\qquad s=\alpha\rho.
$$
It is a local <simple shear> with unit <determinant>. The squared <principal stretches> are the <eigenvalues> of
$$
A^TA=\begin{pmatrix}1&0&0\\0&1&s\\0&s&1+s^2\end{pmatrix}.
$$
One <principal stretch> is one. The remaining squared <principal stretches> have product one and sum $2+s^2$. Consequently they can be written $\lambda^2$ and $\lambda^{-2}$, where
$$
\boxed{\lambda=\sqrt{1+s^2/4}+s/2,\qquad\lambda^{-1}=\sqrt{1+s^2/4}-s/2.}
$$
For negative $s$ their ordering reverses, which is immaterial to the <strain energy density> of an isotropic material.

Let the reference length be $L$. By <material isotropy>, the total <elastic energy> and end rotation are
$$
E(\alpha)=2\pi L\int_0^a W(\lambda,\lambda^{-1},1)\rho\,d\rho,\qquad \Theta_{\mathrm{end}}=\alpha L.
$$
For a quasistatic change, <incompressibility> makes the constraint reaction do no work. The velocity on an end section is its rigid rotational velocity; its traction power is the resultant <torque> times the angular velocity. Thus <conservation of energy>, or equivalently the <torque and axial force of a twisted incompressible cylinder> work identity, gives $M\,d\Theta_{\mathrm{end}}=dE$ and
$$
\boxed{M(\alpha)=2\pi\frac{d}{d\alpha}\int_0^a W(\lambda,\lambda^{-1},1)\rho\,d\rho.}
$$
The radius $a$ is fixed in this derivative. This argument computes the required end <torque> along the prescribed deformation family; proving that a suitable end-only traction distribution sustains that family is unnecessary here.

For a <neo-Hookean solid>, the identity $\lambda-\lambda^{-1}=s$ gives $\lambda^2+\lambda^{-2}-2=s^2$. Therefore
$$
W=\frac\mu2\alpha^2\rho^2,\qquad E/L=\frac{\pi\mu a^4\alpha^2}{4},\qquad \boxed{M=\frac{\pi\mu a^4}{2}\alpha.}
$$
This is the signed <torque> in the twist direction; its magnitude is $\pi\mu a^4|\alpha|/2$. For the positive twist convention the signed value equals the magnitude.