Normalizing flows is capable of computing the exact log-likelihood of a new sample, unlike GANs, VAEs, and Diffusion Model. GANs are not probabilistic, and both VAEs and diffusion models can only return a lower bound on the likelihood.
One application of density estimation is anomaly detection; the data distribution of a clean dataset is described using a normalizing flow model. New examples with low probability are flagged as outliers. However, caution must be used as there may exist outliers with high probability that don’t fall in the typical set.
Approximating other density models
Normalizing flows can also learn to generate samples that approximate an existing density which is easy to evaluate but difficult to sample from. In this context, we denote the normalizing flow as the student and the target density as the teacher.
To make progress, we generate samples from the student. Since we generated these samples ourselves, we know their corresponding latent variables , and we can calculate their likelihood in the student model without inverting. Thus, we can use a model like a masked autoregressive flow where inversion is slow.
We define a loss function based on the reverse KL divergence that encourages the student and teacher likelihood to be identical and use this to train the student model:
This approach contrasts with the typical use of normalizing flows to build a probability model of data that came from an unknown distribution with samples using maximum likelihood, which relies on the cross-entropy term from the forward KL divergence:
Normalizing flows can model the posterior in VAEs using this trick.