 Variational autoencoder

A variational autoencoder is a generative model with a latent variable $z$ and a learned approximation to its posterior distribution given an observation $x$. Write the model as $p_{\theta }(x,z)=p(z)p_{\theta }(x\mid z)$. A recognition network, implemented as a neural network, produces the parameters of $q_{\varphi }(z\mid x)$; a generative decoder specifies $p_{\theta }(x\mid z)$. Using one recognition rule across observations is amortized inference, rather than solving a fresh independent optimization problem for every input.^[ref]

Training maximizes an evidence lower bound, often writtenThe first term scores reconstruction under the declared likelihood; the second compares the inferred latent law with the prior. The identityproves that it is a lower bound, since Kullback-Leibler divergence is nonnegative. It becomes tight when the recognition law equals the model posterior, under the usual density/support conditions.^[ref]

For a one-dimensional Gaussian example, let the prior be $N(0,1)$ and the recognition law be $N(\mu ,{\sigma }^2)$. Substituting their log densities and using $\mathbb{E}[z^2]={\mu }^2+{\sigma }^2$ givesAt $\mu =1$, $\sigma =2$, this is $2-\log 2\simeq 1.30685$. The value is a penalty for this particular latent law, not the whole training loss. Gaussian reparameterization writes $z=\mu +\sigma \epsilon$ with fixed-law standard-normal noise so that gradients can pass through the sampled latent value. After training, generation instead draws from the prior and then the generative decoder; it need not start from an observed input.

A Variational Autoencoder (VAE) is a type of generative model that is used in unsupervised machine learning tasks to learn the underlying structure of data. It combines principles from probabilistic graphical models and neural networks. Here are the key components and ideas behind VAEs: ### Structure A VAE typically consists of two main components: 1. **Encoder (Recognition Model)**: This part of the VAE takes input data and encodes it into a lower-dimensional latent space.