What if multiple final prediction values for multi-class Neural Network are equal
I'd say it's not very common to have absolutely equal probabilities with a model that has been trained for sufficient epochs and that does not overfit hard. 

That said, if you happen to have equal probabilities, the choice of the class to predict should depend on the domain you're approaching, e.g:


A possible implementation is like in numpy's argmax, by choosing the first index.
You could also create a new class "unknown" specifically for this issue, in your test set.
Is it worse for your business problem if you make a mistake and predict class "2" instead of class "1"? You should then choose the class whose probability of mistake is less costly (Think about the medical domain and predicting several stages of cancer) 
Is the class balance the same? You could go for the most probable class in the training set