How would one debug, understand or fix the outcome of a neural network?
If the observation that the neural network saw was recorded, then yes the prediction can be explained. 

There was a paper written fairly recently on this topic called "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016). In this paper, the author described an algorithm called LIME which is able to explain any machine learning models predictions. It can be used to establish why a machine learning model made a prediction, help a data scientist debug a model, and help a data scientist improve the accuracy of a specific model. LIME can be used to explain the predictions of any neural network including CNNs, RNNs, and DNNs.