= Jensen's inequality
{c}
{title2=$f(\mathbb E X)\leq\mathbb E[f(X)]$}
{wiki}
= Jensen inequality
{c}
{synonym}
<Jensen's inequality> says that applying a <convex function> to a <weighted mean> cannot exceed the same <weighted mean> of its outputs. For nonnegative weights $w_i$ with $\sum_{i=1}^n w_i=1$ and inputs $x_i$ in a <convex set>,
$$
f\left(\sum_{i=1}^n w_i x_i\right)\leq\sum_{i=1}^n w_i f(x_i).
$$
Nonnegative weights are essential: this is a <convex combination>, rather than an arbitrary <affine combination>. For two inputs it is precisely the defining <inequality> for a <convex function>; repeatedly grouping inputs gives the finite weighted version.
Take $f(x)=x^2$, weights $1/4$ and $3/4$, and inputs $1$ and $3$. Their <weighted mean> is $2.5$, so the two sides are
$$
f(2.5)=6.25\leq\tfrac14 f(1)+\tfrac34 f(3)=7.
$$
\b[The Jensen gap is $0.75$.] The <inequality> measures the effect of replacing distinct inputs by one representative input before applying a nonlinear <function>.
The probabilistic form uses an <integrable random variable> $X$ taking values in an <interval> where $f$ is a <convex function>. If $f(X)$ is also an <integrable random variable> and $\mathbb E X$ belongs to that <interval>, then
$$
f(\mathbb E X)\leq\mathbb E[f(X)].
$$
The direction reverses for a <concave function>. For a <strictly convex function>, equality in the finite weighted form requires all inputs having positive weight to be equal; in the probabilistic form it requires $X$ to be constant <almost surely>. An <affine function> gives equality for any admissible distribution. https://stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf[Boyd and Vandenberghe's slides, section 3.14, state the expectation form]{ref}.
For the <quadratic function> $f(x)=x^2$ and a <random variable> with a finite <second moment>, the difference between the two sides is exactly its <variance>:
$$
\mathbb E[X^2]-(\mathbb E X)^2=\operatorname{Var}(X)\geq0.
$$
Consider two equally likely values $2-d$ and $2+d$. Their <expected value> stays at $2$ as $d$ increases, but their <variance> is $d^2$. Thus the <expected value> of the square grows from $4$ to $4+d^2$. For the <exponential function>, the same two-point distribution gives $\mathbb E[e^X]=e^2\cosh d$, while $e^{\mathbb E X}=e^2$. Both are examples of <Jensen's inequality>, but their <Jensen gaps> depend differently on the spread of the inputs.
\Image[/jensen-mean-and-spread.png]
{title=Keeping the mean fixed while spreading the inputs increases the average of these convex functions}
{description=The <random variable> takes $2-d$ and $2+d$ with equal <probability>. The <quadratic function> produces a <Jensen gap> of $d^2$, exactly the <variance>. The <exponential function> produces a <Jensen gap> of $e^2(\cosh d-1)$, involving the <hyperbolic cosine>. The horizontal reference in each panel applies the <function> directly to the unchanged <expected value> $2$.}
Another useful choice is the <concave function> $\log x$ on positive inputs. Its version of <Jensen's inequality> gives $\sum_iw_i\log x_i\leq\log(\sum_iw_ix_i)$. Applying the increasing <exponential function> shows that the weighted <geometric mean> cannot exceed the weighted <arithmetic mean>. For the equally weighted inputs $1$ and $9$, these are $3$ and $5$. This recovers the <arithmetic-geometric mean inequality> and illustrates how choosing one <function> can generate an entire family of bounds.
A short proof idea explains the probabilistic form. If $f$ has a <subgradient> $g$ at $\mu=\mathbb E X$, the supporting <affine function> gives $f(X)\geq f(\mu)+g(X-\mu)$. Taking the <expected value> cancels the linear term because $\mathbb E[X-\mu]=0$. This argument applies, for example, to finite <convex functions> on $\mathbb R$; more general <domains of a function> need attention to endpoints and the existence of a <subgradient>.
<Johan Jensen>'s 1906 paper “Sur les fonctions convexes et les inégalités entre les valeurs moyennes” is an early source for the connection between <convex functions> and inequalities for <means>. https://zenodo.org/records/2371297[Jensen's original article and publication record, DOI 10.1007/BF02418571]{ref}.
Back to article page
