Solution (source code)

= Solution

The upward orientation of $\mathbf X_r\times\mathbf X_\theta$ corresponds to traversing the boundary from the apex to the starting rim, around the rim in increasing azimuth, and back to the apex. This is the indicated $ABCA$ orientation and is outward from the solid. Integrate that <vector area element> directly:
$$
\mathbf S=\int_0^\alpha\!\int_0^R
\left(\frac{2ar^2}{b^2}\cos\theta,
\frac{2ar^2}{b^2}\sin\theta,r\right)\,dr\,d\theta.
$$
The radial integrals are $2aR/3$ in the horizontal components and $b^2/2$ in the vertical component. Hence the <vector area> is
$$
\boxed{\mathbf S_{ABCA}
=\frac{2a|b|}{3}\sin\alpha\,\mathbf i
+\frac{2a|b|}{3}(1-\cos\alpha)\,\mathbf j
+\frac{\alpha b^2}{2}\,\mathbf k}.
$$
The absolute value is needed only because the original statement permits negative $b$. Reversing the boundary orientation reverses the entire vector. For a full rotation the horizontal components cancel, leaving $\pi b^2\mathbf k$, the signed projected <area> of the base.