Jensen’s Inequality

A convex function $f(x)$ satisfies

$$f\left(\sum _{i=1}^M{\lambda }_i{x}_i\right)\le \sum _{i=1}^M{\lambda }_{i}f(x_i)$$

where ${\lambda }_i\ge 0$ and ${\sum }_i{\lambda }_i=1$, for any set of points $\{x_i\}$. This is known as Jensen’s Inequality.

If we interpret ${\lambda }_i$ as the probability distribution over a discrete variable $x$ taking the values $\{x_i\}$, then we can rewrite the above as:

$$f(E[x])\le E[f(x)]$$

where $E[\cdot ]$ denotes the expectation.

Continuous Form

For continuous variables, Jensen’s Inequality takes the form

$$f\left(\int \mathbf{x}p(\mathbf{x})dx\right)\le \int f(\mathbf{x})p(\mathbf{x})dx$$