Solution (source code)

= Solution

Complete the square:
$$
N(M-N)=\frac{M^2}{4}
-\left(N-\frac M2\right)^2.
$$
If $M$ is even, an integer $N$ can equal $M/2$, giving the maximum $M^2/4$. If $M$ is odd, every integer $N$ is at least $1/2$ from $M/2$, giving the maximum $(M^2-1)/4$. Thus
$$
\boxed{
N(M-N)\le
\begin{cases}
M^2/4,&M\text{ even},\\
(M^2-1)/4,&M\text{ odd}.
\end{cases}}
$$