Fully Convolutional Variational Autoencoder
In my experience, when people say "fully" convolutional, even when doing something typical like ImageNet classification, they are still typically referring to a network with at least one final dense layer.

If I understand your question correctly, you're looking to create a VAE with some convolutional layers which has the same sized output as input, but you're confused how to upsample in the decoder such that you go from fewer latent dimensions to an output of the same size as your input. There are a few ways people typically handle this, two of which are deconvolution and sub-pixel (description of the difference, good deconv tutorial). Both allow you to use something like a convolution to take the output from $ d &lt;&lt; n $ latent dimensions and upsample it into an $ n $ dimensional output.

Here's an example written in Keras that does this using deconvolutional layers.