Applying bayesian methods to a simple neural network
Bayesian models specify priors to inform and constrain the models and get uncertainty estimation in form of a posterior distribution. Non-Bayesian Deep Learning computes a scalar value for weights and biases at each layer. Bayesian Deep Learning calculates a posterior distribution of weights and biases at each layer which better estimates uncertainty but increases computational cost. 

Edward is a Python package for Bayesian inference, including Deep Learning.