Variational autoencoder

A variational autoencoder is a generative model with a latent variable $z$ and a learned approximation to its posterior distribution given an observation $x$. Write the model as $p_{\theta }(x,z)=p(z)p_{\theta }(x\mid z)$. A recognition network, implemented as a neural network, produces the parameters of $q_{\varphi }(z\mid x)$; a generative decoder specifies $p_{\theta }(x\mid z)$. Using one recognition rule across observations is amortized inference, rather than solving a fresh independent optimization problem for every input.^[ref]

Training maximizes an evidence lower bound, often writtenThe first term scores reconstruction under the declared likelihood; the second compares the inferred latent law with the prior. The identityproves that it is a lower bound, since Kullback-Leibler divergence is nonnegative. It becomes tight when the recognition law equals the model posterior, under the usual density/support conditions.^[ref]

For a one-dimensional Gaussian example, let the prior be $N(0,1)$ and the recognition law be $N(\mu ,{\sigma }^2)$. Substituting their log densities and using $\mathbb{E}[z^2]={\mu }^2+{\sigma }^2$ givesAt $\mu =1$, $\sigma =2$, this is $2-\log 2\simeq 1.30685$. The value is a penalty for this particular latent law, not the whole training loss. Gaussian reparameterization writes $z=\mu +\sigma \epsilon$ with fixed-law standard-normal noise so that gradients can pass through the sampled latent value. After training, generation instead draws from the prior and then the generative decoder; it need not start from an observed input.

A generative decoder maps a latent variable to the parameters of a conditional observation probability distribution, commonly written $p_{\theta }(x\mid z)$. In a variational autoencoder, a neural network can supply those parameters. The conditional law, including its observation noise and support, is part of the model; the decoder need not return one deterministic reconstructed observation.^[ref]

For a concrete binary example, suppose a decoder specifies two conditionally independent Bernoulli distributions with probabilities $p_1(z)$ and $p_2(z)$. ThenIf a chosen latent value gives $(p_1,p_2)=(0.8,0.3)$, the conditional probability of $x=(1,0)$ is $0.8(1-0.3)=0.56$, and its negative log likelihood is approximately $0.5798$. The vector $(0.8,0.3)$ is a vector of probabilities, not itself a binary sample. A model for continuous observations would need a different conditional distribution.^[ref]

Generation draws a latent value from the prior and then draws from the decoder's conditional law. Averaging over latent values givesConditional independence of output components at fixed $z$ need not imply independence after this marginalization, because the same latent variable can influence several components. The recognition network runs in the other direction for inference; it is not an algebraic inverse of the generative decoder.

A recognition network is a learned mapping from observed data to parameters of an approximate posterior distribution over latent variables. In a variational autoencoder, it specifies $q_{\varphi }(z\mid x)$, while the generative decoder specifies the opposite conditional direction $p_{\theta }(x\mid z)$. The recognition network supports amortized inference; it need not coincide with the exact posterior of the learned generative model.^[ref]

For a scalar Gaussian recognition law, the neural network can output a mean ${\mu }_{\varphi }(x)$ and log variance ${\ell }_{\varphi }(x)$. Thenwhere $\epsilon \sim N(0,1)$ is parameter-independent noise. Exponentiating the log variance ensures a positive variance, and the factor $1/2$ converts it to the standard deviation used in Gaussian reparameterization.^[ref]

For an original calculation, suppose one input gives $\mu =1$ and $\ell =\log 4$. The recognition variance is4 and its standard deviation is2. A noise draw $\epsilon =1/2$ gives $z=2$, while the posterior mean supplied by the same network remains1. Confusing log variance with log standard deviation would instead produce the wrong scale.

A deterministic network evaluation and a random latent sample are distinct steps. Re-evaluating the same fixed network on the same input returns the same distribution parameters; new noise draws can still yield different $z$ values. Returning only the mean omits the uncertainty represented by that distribution. Its adequacy must be assessed against the intended inference problem, not inferred merely from the fact that it outputs a positive variance.