Importance Sampling

Unlike normalizing flows, it’s not possible to exactly evaluate the probability of a sample with the VAE, which describes this probability as:

$$\begin{array}{rl}Pr(x)& =\int Pr(x|z)Pr(z)dz\\ & ={\mathbb{E}}_z[Pr(x|z)]\\ & ={\mathbb{E}}_z[{\text{Norm}}_x[f[z,\varphi ],{\sigma }^{2}I]]\end{array}$$

In principle, we could approximate this probability using the Monte Carlo estimate by drawing samples from $Pr(z)={\text{Norm}}_z[0,I]$ and computing:

$$Pr(x)\approx \frac{1}{N}\sum _{n=1}^{N}Pr(x|z_n)$$

However, the curse of dimensionality means that almost all values of $z_n$ that we draw would have a very low probability $Pr(x|z_n)$. We would have to draw an enormous number of samples to get a reliable estimate. A better approach is to use importance sampling.

Here, we sample $z$ from an auxiliary distribution $q(z)$, evaluate $Pr(x|z_n)$, and rescale the resulting values by probability under the new distribution:

$$\begin{array}{rl}Pr(x)& =\int Pr(x|z)Pr(z)dz\\ & =\int \frac{Pr(x|z)Pr(z)}{q(z)}q(z)dz\\ & ={\mathbb{E}}_{q(z)}\left[\frac{Pr(x|z)Pr(z)}{q(z)}\right]\\ & \approx \frac{1}{N}\sum _{n=1}^N\frac{Pr(x|z_n)Pr(z_n)}{q(z_n)}\end{array}$$

where now we draw the samples from $q(z)$. If $q(z)$ is close to the region of $z$ where the $Pr(x|z)$ has high likelihood, then we will focus the sampling on the relevant area of space and estimate $Pr(x)$ much more efficiently.

The product $Pr(x|z)Pr(z)$ that we are trying to integrate is proportional to the posterior distribution $Pr(z|x)$ by Bayes’ Rule. Hence, a sensible choice of auxiliary distribution $q(z)$ is the variational posterior $q(z|x)$ computed by the encoder.

In this way, we can approximate the probability of new samples. With sufficient samples, this will provide a better estimate than the lower bound and could be used to evaluate the quality of the model by evaluating the log-likelihood of test data. Alternatively, it could be used as a criterion for determining whether new examples belong to the distribution or are anomalous.