Transformation of Densities

How does a probability density transform under a non-linear change of variable? Probability densities have different behavior than simple functions under such transforms.

Consider a single variable $x$, and we make a change of variables $x=g(y)$, such that $f(x)$ becomes a new function $\tilde{f}(y)$ such that

$$\tilde{f}(y)=f(g(y))$$

For a probability density $p_x(x)$, if we want to find a density for a new variable $y$, such that $x=g(y)$. This density is expressed as $p_y(y)$. To make this transformation, we consider the probabilities of $x$ and $y$ falling into infinitesimally small ranges.

• The probability that $x$ in the range $(x,x+\delta x)$ is $p_x(x)\delta x$ (see probability density).
• Similarly, the probability that $x$ in the range $(y,y+\delta y)$ is $p_y(y)\delta y$.

Now, since $x$ and $y$ are related by $x=g(y)$, we can say that a small change in $y$ will cause a corresponding small change in $x$. This can be expressed mathematically by considering that probability is conserved when we change variables, such that:

$$p_x(x)\delta x\approx p_y(y)\delta y$$

This becomes exactly equal when we take the limit of $\delta x\to 0$ and $\delta y\to 0$:

$$\underset{\delta x\to 0}{\lim }{p}_x(x)\delta x=\underset{\delta y\to 0}{\lim }{p}_y(y)\delta y$$

We can then turn this into:

$$\begin{array}{rl}{p}_y(y)& =p_x(x)\left|\frac{dx}{dy}\right|\\ & =p_x(g(y))\left|\frac{dg}{dy}\right|\end{array}$$

Here we’re using the modulus $|\cdot |$ because the derivative could be negative, but we want to scale the density by the proportion of lengths, which is a positive value.

This sort of procedure is very powerful, as any density $p(y)$ can be obtained from a fixed density $q(x)$ by making a non-linear change of variable $y=f(x)$ in which $f(x)$ is monotonic so that $0\le f^{\prime }(x)<\infty$, and $q(x)$ is non-zero everywhere. However, it can also make things more complicated, such as wen trying to find Maximum of Transformed Density.

This property is important to Normalizing Flows.