Solution (source code)

= Solution

Let $s=(x_0,y_0,z_0)$ be the sphere centre. The circle centre is its perpendicular projection onto the plane:
$$
\boxed{h=s-(n\cdot s-c_0)n,\qquad d=|s-h|=|n\cdot s-c_0|.}
$$
For the specific plane this is
$$
d=\frac{|5x_0+2y_0+3z_0-11|}{\sqrt{38}},\qquad
h=s-\frac{5x_0+2y_0+3z_0-11}{38}(5,2,3).
$$
For a point on the section, its displacement from $h$ lies in the plane and is perpendicular to $s-h$. The <Pythagorean theorem> therefore gives
$$
\boxed{r=\sqrt{R^2-d^2}.}
$$
A genuine circle requires $d<R$. At $d=R$ the section is a single tangent point, and at $d>R$ the plane misses the sphere. A central cut $d=0$ gives a great circle of radius $R$.