What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?
Ans 1.
The motive of Variational Inference(on which VAE is based), is to decrease $KL(q(z|x)||p(z))$, where p(z) is our chosen distribution of the hidden variable z. After doing some math, we can write this expression as-
$
KL(q||x) = log(p(x)) - \Sigma_z q(z)log(\frac{p(x,z)}{q(z)})
$
For a given x, the first term of RHS is constant. So we maximise the second term so that the KL divergence goes to zero.
We can write the second term as
$E_{q(z)}[log(p(x|z)] - KL(q(z|x)||p(z))$
(try writing p(x,z) as $\frac{p(x|z)}{p(z)}$ and then expand. Here, p(z) is the distribution of our choice, i.e. a Gaussian distribution). We argue that the process from z to x is deterministic and z is gaussian. So, the first term could be replaced by $exp(log(-||x-\hat{x}||^2))$(we replaced z by $\hat{x}$ because it's deterministic- this is now the exact proof). So we have-
$
maximize(-||x-\hat(x)||-KL(q(z|x)||p(z)))
$
and we get our loss function.
We also know that variational autoencoders almost never find the optimal solution, so I am not sure how would playing around with the weights affect it(Nor do I know if it makes sense mathematically).
Ans 2.
We can say that the KL divergence has a regularising effect.
This page has some nice experiments which will help you understand what happens to the latent space when you decrease the KL divergence part.
Ans 3.
Yes, you can say that. You are fixing the dimensions, but are lenient on the distribution. In fact, you are approaching autoencoders by doing this.
Separate-
I want to point you towards this article. It explains why we choose to minimise $KL(q(z|x)||p(z))$ instead of $KL(p(z)||q(z|x))$ (the latter is intractable) and what would happen if we choose less independent variables for our estimator q(z).
Also, have you tried increasing the dimensions of the latent space? It can also have a 'de-regularizing' effect. It seems that the model is underfitting the data- the reconstruction loss is high with the normal loss, compared to when you decrease the regularizing term.
Hope it helps.