spikes in the cost vs iteration graph using ReLU activation function The cost vs iteration graph while using the ReLU activation function has number of spikes 



The accuracy on MNIST dataset (both train and test) is around 95%, also on using sigmoid as an activation function I get a smooth downward sloping curve so I think the implementation is correct. Are these spikes expected for ReLU? How would you explain this property?
When I run my implementation of MNIST digit recognition I even get those spikes for a sigmoid transfer function. You have to question yourself: how bad are those spikes? I presume you use stochastic gradient descent right. For every iteration you consider a batch (a subspace of the samples) and train on that. Some of those batches contain very difficult pictures and your network will fail at them, resulting in a high cost. This in turn will lead to a temporary high adaptations of your weights and biases. You see that it drops again quite quickly because the other samples will again send it in the right direction. 

I think you have more spikes because your ReLU is just better at generalization and thus after a while you will get more peaks (but lower) because the network gets better and better. And in the end the most important criteria is your accuracy and not the cost ;). This has less problems with the spikes because it takes into account all the samples instead of a batch.

Concluding: stochastic gradient descent will go to a low point, but because of it random walk it might bump into some peaks, but in the end it will find a nice low cost.

If you still have some questions, feel free to ask, I am still quite new here and still have to learn how to give good answers.