What does the approximate posterior on latent variables, $q_\phi(z|x)$, tend to when optimising VAE's
Practically, when optimizing VAE, you assume that prior $p(z)\sim N(0,1)$; i.e. the unit Gaussian distribution. However, in testime you sample z from $p(z|x)$; the encoder model. Why is that?
Let's go back to the start. We have a model $p_{\theta}(x)$ and the data $\{x_1, ..., x_N\}$. Solving the maximum log-likelihood problem, we have
\begin{equation}
\begin{split}
\theta &amp;= argmax_{\theta}\frac{1}{N}\sum_i log p_{\theta}(x_i) \\
&amp;= argmax_{\theta}\frac{1}{N}\sum_i \left( \int p_{\theta}(x_i|z)p(z)dz \right)
\end{split}
\end{equation}
which is intractable to calculate. So what to do now?
Here it comes the Variational Inference: &quot;Use the expected log-likelihood instead.&quot;
\begin{equation}
\begin{split}
\theta &amp;= argmax_{\theta}\sum_i E_{z \sim p_{\theta}(z|x)}\left[ logp_{\theta}(x,z) \right]
\end{split}
\end{equation}
We approximate $q(z) \approx p_{\theta}(z|x)$. Thus, we unfold $logp(x)$:
\begin{equation}
\begin{split}
logp(x) &amp;= log \int p_{\theta}(x_i|z)p(z)dz \\
&amp;= log \int p_{\theta}(x_i|z)p(z) \frac{q(z)}{q(z)}dz \\
&amp;= log E_{z \sim q(z)}\left[ \frac{p_{\theta}(x|z)p(z)}{q(z)} \right] \\
&amp;\geq E_{z \sim q(z)}\left[ log \frac{p_{\theta}(x|z)p(z)}{q(z)} \right]  \\
&amp;= E_{z \sim q(z)}\left[ logp_{\theta}(x|z) + logp(z) \right] - E_{z \sim q(z)}\left[ logq(z)\right] \\
&amp;= E_{z \sim q(z)}\left[ logp_{\theta}(x|z) + logp(z) \right] + H(q) \\
&amp;= ELBO(p,q)
\end{split}
\end{equation}
where $H(q)$ is the entropy of q. As you highlighted we can also write
$$ ELBO(p,q) = logp(x) - D_{KL}\left( q(z) || p(z|x) \right) $$
This means that (a) maximizing $ELBO(p,q)$ w.r.t. to $q$ then KL-Divergence is minimized and (b) maximizing $ELBO(p,q)$ w.r.t to $p$ then the model is improved as the log-likelihood is improved. So this point of yours is true.
However, how can we actually train this model?
The answer is by using Amortized Variational Inference! Practically, we use 2 Neural Networks, $\phi$ and $\theta$, so that we have 2 models: $q_{\phi}(z|x)$ (encoder) and $p_{\theta}(x|z)$ (decoder). Thus, we replace $q(z)$ with $q_{\phi}(z|x)$ and $ELBO(p,q)$ with $ELBO(\theta,\phi)$ .
To fully answer your question, I should reach your first ELBO formula. I will unfold my first equation about ELBO:
\begin{equation}
\begin{split}
ELBO(\theta,\phi) &amp;= E_{z \sim q_{\phi}(z)}\left[ logp_{\theta}(x|z)\right] + E_{z \sim q_{\phi}(z)}\left[ logp(z) \right] + H\left(q_{\phi}(z|x)\right) \\
&amp;= E_{z \sim q_{\phi}(z)}\left[ logp_{\theta}(x|z)\right] - D_{KL}\left( q_{\phi}(z|x) || p_{\theta}(z) \right) 
\end{split}
\end{equation}
Therefore, using the Amortized Variational Inference, we maximize the above objective.