Tensorflow oscillating Test and Train Accuracy?
From your results I see that the network is working good: the loss function is reducing, that means that the calculated results are approaching the expected ones.
But in facts the accuracy is not, so there should be some error in the way you calculate it.

The fact that the loss function start to oscillate at some point (~after 20000 iterations) is expected, and that is the reason why usually one lower the learning rate after some epochs, or use some other technique of annealing.