How to fix these vanishing gradients?
In order to fix the problem of vanishing gradients, you can use Xavier Initilization. Also, the implementation of Xavier Initialization in tensorflow can be done by following this thread.