 Solution

For a right prism with equilateral triangular ends, the triangle area is $\sqrt{3}{\alpha }^2/4$. Thus its volume and external area areApply the Lagrange multiplier method to $V-\lambda (A-A_0)$ with positive lengths. The stationarity equations areThe second gives $\lambda =\sqrt{3}\alpha /12$. Substitution in the first then gives $6\beta =\sqrt{3}\alpha +3\beta$, or $\beta =\alpha /\sqrt{3}$. The area constraint becomesConsequentlyThe resulting volume is $V_{\max }=3^{3/4}/4{\mathrm{m}}^3$. As a check on the maximum, eliminating $\beta$ gives $V(\alpha )=\sqrt{3}{A}_0\alpha /12-{\alpha }^3/8$, whose second derivative is negative for every admissible positive $\alpha$; volume also tends to zero at both degenerate boundary shapes.