Model that predicts probability of correctness of another model Problem:
Given a neural network for image classification with $1000$ classes, the objective is to create another model which will output the probability of the neural network giving the correct prediction for a specific input image.
Thoughts:
My ideas so far have been:

Creating a convolutional network and training it with raw images together with a label which indicates if the first NN predicted it correct or not.
Creating a fully connected network and training it with outputs of hidden layers/features of the first NN instead of the raw image, together with the labels as before.
Creating a fully connected network and training it with the top-k outputs of the softmax layer of the first NN together with the labels.

The first method yielded an accuracy of $0.51$, the second $0.58$ and the last one $0.79$.
Can you suggest another method (or a modification of one from the above) which can achieve greater accuracy?
This is a problem known as quality estimation (QE) in the application of Machine Translation (there's a regular Shared Task about it). The goal is to train a model to predict the quality of the output of a ML system.
There is an important theoretical obstacle to this kind of task: if such a QE model was able to predict perfectly, then in theory it would be possible to create a perfect model for the original task by trying many different models/parameters until the QE system predicts a high level of quality. What this means practically is that a QE model can only be as good as the system it's trying to estimate since it relies mostly on the same information, otherwise the system is clearly sub-optimal. Essentially the best the QE system can do is to determine how hard it is to correctly predict a particular input image, but it cannot predict with any certainty whether the actual prediction is correct or not.