Past exam of the mathematics course of the year 1 of the natural sciences course of the University of Cambridge / 2015 / Paper 2 / 19

At a regular constrained stationary point, the change in $f$ along every allowed tangent direction must vanish. Thus its gradient is normal to the constraint curve, giving the Lagrange multiplier equationsprovided $\nabla g\ne 0$. Solve these equations for $x,y,\lambda$, then classify the candidates using the constraint or a constrained second-derivative test. Points where the constraint gradient vanishes require separate analysis.

For $n$ variables and $r$ regular independent constraints $g_j=c_j$, use one multiplier per constraint:There are $n+r$ scalar equations in the $n+r$ unknowns. Independence of the constraint gradients is the regularity condition; boundary points must be treated separately when present.

Use $\mathcal{L}=-{\sum }_i{x}_i\log x_i-\lambda ({\sum }_i{x}_i-1)$. Differentiating with respect to each positive $x_i$ givesEvery $x_i$ is therefore equal; normalization forcesThe Hessian matrix of $f$ is diagonal with entries $-1/x_i$, so its quadratic form is strictly negative for every nonzero perturbation. In particular it is negative along the tangent space ${\sum }_i\delta x_i=0$, making this the unique global maximum on the positive probability simplex. This is the maximum entropy distribution on a finite set, using natural logarithms.

Introduce a second multiplier $b$ and useThe stationary equations become $-\log x_i-1-\lambda -b{y}_i=0$. Writing $a=e^{-1-\lambda }>0$ givesThe two constraints determine the constants throughEquivalently, $a=1/Z(b)$ with $Z(b)={\sum }_i{e}^{-b{y}_i}$, and $-d\log Z/db=Y$. This is the maximum entropy distribution with a prescribed finite mean.

For nonconstant $y_i$, a strictly positive solution exists exactly when ${\min }_i{y}_i<Y<{\max }_i{y}_i$. The mean's derivative with respect to $b$ is $-{\mathrm{Var}}_b(y)<0$, so $b$ is unique in that case. At either extreme mean no strictly positive solution exists. If all $y_i=y_0$, feasibility requires $Y=y_0$ and the extra constraint is redundant: $x_i=1/n$, while $a,b$ are not separately identifiable. The negative entropy Hessian ensures that any feasible interior stationary point is the unique constrained maximum.