Hidden neuron representation of weights
It has learnt to recognize the digits, but it might have put too much weight on single pixels. Try to add different amounts of L2 regularization or dropout and compare the visualizations of the weights. Adding some kind of regularization should make the net rely less on single / independent pixels and more on the inherent structure of the digits, giving you smoother weights / visualization.