Nonlinear Latent Variable Model

In the nonlinear latent variable model, both the data $x$ and the latent variable $z$ are continuous and multivariate.

The prior $Pr(z)$ is a standard multivariate normal:

$$Pr(z)={\text{Norm}}_z[0,I]$$

The likelihood $Pr(x|z,\varphi )$ is also normally distributed; its mean is a nonlinear function $f[z,\varphi ]$ of the latent variable, and its covariance ${\sigma }^{2}I$ is spherical:

$$Pr(x|z,\varphi )={\text{Norm}}_x[f[z,\varphi ],{\sigma }^{2}I]$$

• The function $f[z,\varphi ]$ is described by a deep network with parameters $\varphi$.
• The latent variable $z$ is lower dimensional than the data $x$.
• The model $f[z,\varphi ]$ describes the important aspects of the data, and the remaining unmodeled aspects are ascribed to the noise ${\sigma }^{2}I$.

The data probability $Pr(x|\varphi )$ is found by marginalizing over the latent variable $z$:

$$\begin{array}{rl}Pr(x|\varphi )& =\int Pr(x,z|\varphi )dz\\ & =\int Pr(x|z,\varphi )\cdot Pr(z)dz\\ & =\int {\text{Norm}}_x[f[z,\varphi ],{\sigma }^{2}I]\cdot {\text{Norm}}_z[0,I]dz\end{array}$$

This can be viewed as an infinite weighted sum (infinite mixture) of spherical Gaussians with different means, where the weights are $Pr(z)$ and the means are the network outputs $f[z,\varphi ]$.

Generation

A new example $x^{\ast }$ can be generated using ancestral sampling. We draw $z^{\ast }$ from the prior $Pr(z)$ and pass this through the network $f[z^{\ast },\varphi ]$ to compute the mean of the likelihood $Pr(x|z^{\ast },\varphi )$, from which we draw $x^{\ast }$. Both the prior and likelihood are normal distributions, so this is straightforward.