 Solution

Right multiplication gives a right action $(Hx)\cdot g=Hxg$. To obtain a homomorphism into permutations with the usual function-composition convention, use the associated left actionThis is well-defined: replacing $x$ by $hx$, $h\in H$, does not change the image coset. Moreover $\rho (g)\rho (k)(Hx)=Hx{k}^{-1}{g}^{-1}=\rho (gk)(Hx)$, and each map is bijective with inverse $\rho (g^{-1})$.

An element $g$ is in the kernel precisely when $Hx{g}^{-1}=Hx$ for every $x\in G$. This says $x{g}^{-1}{x}^{-1}\in H$, equivalently $g\in x^{-1}Hx$. HenceThis is the largest normal subgroup of $G$ contained in $H$.

Now the index of $H$ is $q$, so $G/\ker \rho$ embeds into $S_q$. Its order therefore divides $q!$. Since $p>q$ is prime, $p$ does not divide $q!$. From $|G|=pq$ it follows that $p$ divides $|\ker \rho |$. But $\ker \rho \subseteq H$ and $|H|=p$, forcing $\ker \rho =H$. Therefore $H$ is normal.

For the last case let $H=⟨b⟩$. Normality makes conjugation by $a$ an automorphism, so $a^{-1}ba=b^r$ with $r\in \{1,2,3,4\}$. Applying it twice and using $a^2=1$ gives $b=b^{r^2}$, so $r^2\equiv 1(\mathrm{mod}5)$. Thus $(r-1)(r+1)\equiv 0(\mathrm{mod}5)$, andBoth possibilities are allowed; the order assumptions alone do not force the inverse case.