Maximum Entropy

Discrete

The maximum entropy configuration can be found by maximizing $H$ using a Lagrange multiplier to enforce the normalization constraint on the probabilities, such that ${\sum }_{i=1}^n{p}_i=1$.

Thus, we maximize

$$\tilde{H}=-\sum _{i}p(x_i)\ln p(x_i)+\lambda \left(\sum _{i}p(x_i)-1\right)$$

from which we find that all of the $p(x_i)$ are equal and are given by $p(x_i)=1/M$ where $M$ is the total number of states $x_i$. The corresponding value of the entropy is then $H=\ln M$. This result can also be derived from Jensen’s Inequality. To verify that the stationary point is indeed a maximum, we can evaluate the second derivative of the entropy, which gives

$$\frac{\partial \tilde{H}}{\partial p(x_i)\partial p(x_j)}=-I_{ij}\frac{1}{p_i}$$

where $I_{ij}$ are the elements of the identity matrix. We see that these values are all negative and, hence, the stationary point is indeed a maximum.

Continuous

We saw that the maximum entropy configuration for discrete distributions corresponds to a uniform distribution of probabilities across the possible states of the variable. For continuous distributions, if we want the maximum to be well-defined, we need to constrain the first and second moments $p(x)$ and to preserve the normalization constraint.

Thus, we maximize differential entropy with three constraints:

$$\begin{array}{rl}{\int }_{-\infty }^{\infty }p(x)dx& =1\\ {\int }_{-\infty }^{\infty }xp(x)dx& =\mu \\ {\int }_{-\infty }^{\infty }(x-\mu {)}^{2}p(x)dx& ={\sigma }^2\end{array}$$

We can then do constrained optimization with Lagrange Multipliers so that we maximize the following functional with respect to $p(x)$:

$$\begin{array}{r}-{\int }_{-\infty }^{\infty }p(x)\ln p(x)dx+{\lambda }_1\left({\int }_{-\infty }^{\infty }p(x)dx-1\right)\\ +{\lambda }_2\left({\int }_{-\infty }^{\infty }xp(x)dx-\mu \right)+{\lambda }_3\left({\int }_{-\infty }^{\infty }(x-\mu {)}^{2}p(x)dx-{\sigma }^2\right)\end{array}$$

We set the derivative of this function to zero giving:

$$p(x)=\exp \{-1+{\lambda }_1+{\lambda }_{2}x+{\lambda }_3(x-\mu {)}^2\}$$

The Lagrange multipliers can be found by back-substitution of this result into the three constraint equations, leading to the result:

$$p(x)=\frac{1}{(2\pi {\sigma }^2{)}^{1}2}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$$

and so the distribution that maximizes the differential entropy is the Gaussian.

If we evaluate the differential entropy of the Gaussian, we obtain

$$H[x]=\frac{1}{2}\{1+\ln (2\pi {\sigma }^2)\}$$

Thus, we see again that the entropy increases as the distribution becomes broader (as ${\sigma }^2$ increases).

This result also shows that the differential entropy, unlike the discrete entropy, can be negative, because $H(x)<0$ for ${\sigma }^2<1/(2\pi e)$.