Jensen's inequality ( $f (E X) \leq E [f (X)]$ )

Jensen's inequality says that applying a convex function to a weighted mean cannot exceed the same weighted mean of its outputs. For nonnegative weights

w_{i}

with

\sum_{i = 1}^{n} w_{i} = 1

and inputs

x_{i}

in a convex set,

f (\sum_{i = 1}^{n} w_{i} x_{i}) \leq \sum_{i = 1}^{n} w_{i} f (x_{i}) .

(1)

Nonnegative weights are essential: this is a convex combination, rather than an arbitrary affine combination. For two inputs it is precisely the defining inequality for a convex function; repeatedly grouping inputs gives the finite weighted version.

Take

f (x) = x^{2}

, weights

1/4

and

3/4

, and inputs

1

and

3

. Their weighted mean is

2.5

, so the two sides are

f (2.5) = 6.25 \leq \frac{1}{4} f (1) + \frac{3}{4} f (3) = 7.

(2)

The Jensen gap is $0.75$ . The inequality measures the effect of replacing distinct inputs by one representative input before applying a nonlinear function.

The probabilistic form uses an integrable random variable

X

taking values in an interval where

f

is a convex function. If

f (X)

is also an integrable random variable and

E X

belongs to that interval, then

f (E X) \leq E [f (X)] .

(3)

The direction reverses for a concave function. For a strictly convex function, equality in the finite weighted form requires all inputs having positive weight to be equal; in the probabilistic form it requires

X

to be constant almost surely. An affine function gives equality for any admissible distribution. ^[ref].

For the quadratic function

f (x) = x^{2}

and a random variable with a finite second moment, the difference between the two sides is exactly its variance:

E [X^{2}] - (E X)^{2} = Var (X) \geq 0.

(4)

Consider two equally likely values

2 - d

and

2 + d

. Their expected value stays at

2

d

increases, but their variance is

d^{2}

. Thus the expected value of the square grows from

4

4 + d^{2}

. For the exponential function, the same two-point distribution gives

E [e^{X}] = e^{2} cosh d

, while

e^{E X} = e^{2}

. Both are examples of Jensen's inequality, but their Jensen gaps depend differently on the spread of the inputs.

Figure 1.
Keeping the mean fixed while spreading the inputs increases the average of these convex functions
. The random variable takes $2 - d$ and $2 + d$ with equal probability. The quadratic function produces a Jensen gap of $d^{2}$ , exactly the variance. The exponential function produces a Jensen gap of $e^{2} (cosh d - 1)$ , involving the hyperbolic cosine. The horizontal reference in each panel applies the function directly to the unchanged expected value $2$ . **Details**.

Another useful choice is the concave function

lo g x

on positive inputs. Its version of Jensen's inequality gives

\sum_{i} w_{i} lo g x_{i} \leq lo g (\sum_{i} w_{i} x_{i})

. Applying the increasing exponential function shows that the weighted geometric mean cannot exceed the weighted arithmetic mean. For the equally weighted inputs

1

and

9

, these are

3

and

5

. This recovers the arithmetic-geometric mean inequality and illustrates how choosing one function can generate an entire family of bounds.

A short proof idea explains the probabilistic form. If

f

has a subgradient

g

μ = E X

, the supporting affine function gives

f (X) \geq f (μ) + g (X - μ)

. Taking the expected value cancels the linear term because

E [X - μ] = 0

. This argument applies, for example, to finite convex functions on

R

; more general domains of a function need attention to endpoints and the existence of a subgradient.

Johan Jensen's 1906 paper “Sur les fonctions convexes et les inégalités entre les valeurs moyennes” is an early source for the connection between convex functions and inequalities for means. ^[ref].