How to interpret plot of cost function in gradient descend algorithm?
I don't know what to make out of the rising and falling in the cost in the beginning. Is that something that we can expect? If so, why? Shouldn't the gradient descend algorithm decrease the cost after every step?


Gradient descent only minimizes the cost if the learning rate is low enough. Also, in practice most of the time you don't use gradient descent on the whole dataset, but on a mini-batch. This means you don't get an guarantee that the global cost is falling for a single step.

An analogy which is often used is that of a landscape with hills. The landscape is your cost. Each point in the landscape is a specific configuration of weights. The gradient shows in the direction of steepest descent. That might be a rabbit hole on a hill, but it might also be downwards to the valley. Now, imagine it shows to the rabbit hole. Instead of going down the rabbit hole, you make a bigger step and so you go up the hill. But then you are on a different position. For your next step, you come away from the rabbit hole and get towards a better (local) minimum.

As always when it comes to training, I would like to point out the awesome gifs by Alec Redford: Gifs 1, and Gifs 2