= Solution
By orthogonal <diagonalization> of the positive-definite <symmetric matrix> $A$, form its positive symmetric inverse square root $S=A^{-1/2}$. Then $S^TAS=I$, and $C=SBS$ is real symmetric. Let $Q$ have an orthonormal eigenbasis of $C$ as its columns, so $Q^TCQ=\operatorname{diag}(\lambda_1,\ldots,\lambda_n)$. Taking $P=SQ$ gives
$$
\boxed{P^TAP=I,\qquad P^TBP=\operatorname{diag}(\lambda_1,\ldots,\lambda_n)}.
$$
Both factors of $P$ are invertible. This proves the <simultaneous congruence diagonalization with a positive form> by an explicit construction; it is congruence rather than simultaneous similarity.
To find the entries without constructing $P$, take determinants in
$$
P^T(B-\lambda A)P=\operatorname{diag}(\lambda_j-\lambda).
$$
Since $\det P\ne0$, the entries are precisely the roots, with multiplicity, of $\det(B-\lambda A)=0$. Equivalently they are the <eigenvalues> of $A^{-1}B$, although that matrix need not itself be symmetric.
For the given $A$, its leading principal minors are $2,3,4$, so it is positive definite. Expanding the <determinant> gives
$$
\det(B-\lambda A)=\det\begin{pmatrix}4-2\lambda&-\lambda&0\\-\lambda&-2-2\lambda&-\lambda\\0&-\lambda&-2\lambda\end{pmatrix}=16\lambda+4\lambda^2-4\lambda^3.
$$
Thus the diagonal entries, in any order, are
$$
\boxed{0,\qquad\frac{1+\sqrt{17}}2,\qquad\frac{1-\sqrt{17}}2}.
$$
Back to article page