 Solution

For any $q$ in the polynomial trial space, $f-q=(f-p)+(p-q)$ and $p-q$ is in that same space. The assumed orthogonality therefore gives the Pythagorean identityThus $p$ is the best approximation; equality is possible only when $q=p$ because the inner product is positive-definite.

Write the linear polynomial as $p=A+Bx$. Orthogonality of $x^2-p$ to $1$ and $x$ in the weighted inner product givesSolving yieldsThe weighted squared error is ${\int }_0^{1}x(x^2-6x/5+3/10{)}^{2}dx=1/600$, providing an additional exact check. The weight's zero value at one endpoint does not destroy positive definiteness on continuous functions: a nonzero continuous function is nonzero on an interval of positive weight.