 Solution

The Roche lobe of star 1 is the volume around it bounded by the critical closed equipotential passing through the inner Lagrange point $L_1$. Material on a lower potential surface remains confined to star 1; at the critical surface a path opens toward star 2.

Let $\mathbf{g}=-\nabla \Psi$ and take $d\mathbf{S}$ outward. For a nearly spherical interior surface, the divergence theorem givesThe self-gravity term contributes $-4\pi G{M}_1$. The companion lies outside $S$, so its potential is harmonic inside and contributes zero net flux. For the centrifugal term, ${\nabla }^2(-{\Omega }^2{s}^2/2)=-2{\Omega }^2$, so its acceleration has divergence $2{\Omega }^2$ and contributes $2{\Omega }^{2}V$. Using $V=4\pi r^3/3$ and $S=4\pi r^2$,The sign is the outward-normal component; the dominant self-gravity is inward and therefore negative.