 Solution

Let $P\mathbf{x}=(\mathbf{n}\cdot \mathbf{x})\mathbf{n}$ be the axial orthogonal projection and $N\mathbf{x}=\mathbf{n}\times \mathbf{x}$. Then $NP=PN=0$, while the vector triple-product identity gives $N^2=P-I$. The map is $aP+a(I-P)+bN$. Its inverse is $P/a$ on the axis. On the perpendicular plane,Thus the inverse of an axis-angle linear map isor, in the requested vector form,The denominators are nonzero because $a,b>0$. Multiplication on each invariant component verifies the matrix inverse without choosing a coordinate frame.