A good way to understand the mathematical details of variational autoencoders through implementation?
I even don't know what exactly back-propagation is used for, if this would help.


Programming backprop manually is one of the best exercises in machine learning. In fact, to do this you don't need a special package, like Tensorflor or Keras, just a general computing library like NumPy (and a whiteboard).

Though I dare to say that VAE is not the best thing to implement, if you are not 100% familiar with backprop. It's pretty advanced technique and you risk to mix up fundamental (e.g. backprop) and specific ideas (e.g. deliberately crafted loss function) and slow down your understanding.

I really think that writing a simple feed-forward neural net (including forward and backward passes) is a very good starting point, after which you can move on to VAE details, if you'd like.


  Any suggestions on where/how to start?


If you feel confident with algebra and calculus, I highly recommend this Stanford course, especially its assignments. The course itself is about computer vision and convolutional neural networks, but the first assignments are very general and will give a lot of insight of what and how neural networks are doing.