Neural Network accuracy and loss guarantees?
For my answers, I assume you are talking about batch (not mini-batch or stochastic) gradient descent.


No. Assume you initialize all weights with the same value. Then all gradients (in the same layer) will be the same. Always. Hence the network effectively only learns one parameter per layer. It is possible (and likely) that this is neither a gloabl nor a local minimum of the network (which has more parameters).
Yes, as the learning rate is "small enough for all practical purposes". (No, if you use SGD or mini-batch gradient descent)
Usure. I think the correct answer is "No, the network can make more mistakes in between with cross entropy.". It is certainly sure to improve CE loss while at the same time getting worse at accuracy (see proof below). However, I'm not sure if the gradient would ever lead to such a result.


Example for 3

#!/usr/bin/env python

from math import log

def ce(vec):
    """index 0 is the true class."""
    return -(log(vec[0]) + sum([log(1-el) for el in vec[1:]]))

a = [0.1001, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0999]
print(ce(a))
b = [0.49, 0.51, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
print(ce(b))


gives:

ce(a) = 3.24971912864
ce(b) = 1.42669977575


Hence the cross entropy loss dropped (as expected), the probability for the correct class increased (as expecte) but it makes a mistake (if you simply take the argmax).