Differential Entropy

We can extend the definition of entropy to include distributions $p(x)$ over continuous variables $x$.

We can do this by dividing $x$ into “bins” of width $\Delta$. Then, assuming $p(x)$ is continuous, the mean value theorem tells us that, for each such bin, there must exist a value $x_i$ in the range $i\Delta \le x_i\le (i+1)\Delta$ such that

$${\int }_{i\Delta }^{(i+1)\Delta }p(x)dx=p(x_i)\Delta$$

We can now quantize the continuous variable $x$ by saying any $x$ within the $i$th bin is approximated/quantized to the value $x_i$. Since $x_i$ represents all values in its bin, the probability of observing the value $x_i$ is then the integral of $p(x)$ over the bin (above), which is $p(x_i)\Delta$.

The entropy of this discrete distribution gives:

$$\begin{array}{rl}{H}_{\Delta }& =-\sum _{i}p(x_i)\Delta \ln (p(x_i)\Delta )\\ & =-\sum _{i}p(x_i)\Delta \ln p(x_i)-\ln \Delta \end{array}$$

We omit the second term $-\ln \Delta$ on the right hand side, since it’s independent of $p(x)$.

Now we consider the limit $\Delta \to 0$, which gives:

$$\underset{\Delta \to 0}{\lim }\left\{-\sum _{i}p(x_i)\Delta \ln p(x_i)\right\}=\boxed{-\int p(x)\ln p(x)dx}$$

The boxed quantity on the right-hand side is called the differential entropy. The discrete and continuous forms of the entropy differ by a quantity $\ln \Delta$, which diverges in the limit $\Delta \to 0$.

For a density defined over multiple continuous variables denoted by the vector $\mathbf{x}$, the differential entropy is given by

$$H[\mathbf{x}]=-\int p(\mathbf{x})\ln p(\mathbf{x})dx$$