Solution (source code)

= Solution

Use the unsquared convention for <quantum fidelity>. For <density operators> on a common finite-dimensional <Hilbert space>,
$$
\boxed{F(\rho,\sigma)=\|\sqrt\rho\sqrt\sigma\|_1=\operatorname{Tr}\sqrt{\sqrt\rho\,\sigma\sqrt\rho}.}
$$
Here $\|A\|_1=\operatorname{Tr}\sqrt{A^\dagger A}$ is the <trace norm>, and all square roots are the positive operator square roots. The two displayed expressions agree because $\sqrt\rho\sqrt\sigma$ and its adjoint have the same singular values. This convention has $0\leq F\leq1$; some literature squares this quantity, but that convention is not used here.

For normalized <pure states>, the rank-one operator $\sqrt\rho\sqrt\sigma=\langle\varphi|\psi\rangle|\varphi\rangle\langle\psi|$ has a single nonzero singular value. Therefore
$$
\boxed{F(|\varphi\rangle\langle\varphi|,|\psi\rangle\langle\psi|)=|\langle\varphi|\psi\rangle|.}
$$