What are the ways to calculate the error rate of a deep Convolutional Neural Network, when the network produces different results using the same data? I am new to the object recognition community. Here I am asking about the broadly accepted ways to calculate the error rate of a deep CNN when the network produces different results using the same data.

1. Problem introduction

Recently I was trying to replicate some classic deep CNNs for the object recognition tasks. Inputs are some 2D image data including objects and the output are the identification/classification results of the object. The implementation involves the use of Python and Keras.

The problem I was facing is that, I may get different validation results among multiple runs of the training even using the same training/validation data sets. To me, that made it hard to report the error rate of the model since every time the validation result may be different.

I think this difference is because of the randomness involved in different aspects of deep CNN, such as random initialization, the random ‘dropout’ used in the regulation, the ‘shuffle’ process used in the choosing of epochs, etc. But I do not know yet the “right” ways to deal with this difference when I want to calculate the error rate in object recognition field. 

2. My exploration – online search

I have found some answers online here. The author proposed two ways, and he/she recommended the first one shown below:


  The traditional and practical way to address this problem is to run your network many times (30+) and use statistics to summarize the performance of your model, and compare your model to other models.


The second way he/she introduced is to go to every relevant aspect of the deep CNN, to "freeze" their randomness nature on purpose. This kind of approach has also been introduced from Keras Q&amp;A here. They call this issue the “making reproductive results”.

3. My exploration – in academia community (no result yet, need your help!)

Since I was not sure whether the two ways mentioned above are the “right” ones broadly accepted, I was going further exploring in the object recognition academia community.

Now I just begin to read from imageNet website. But I have not found the answer yet. Maybe you could help me knowing the answer easier. Thanks!

Daqi
Unless the entropy is collected from quantum, thermal, or other true entropy sources, it is considered pseudo-random at best.  This is why you can have a deterministic process that is not perfectly repeatable, exhibiting some variance.

List of Sources of Pseudo-random Behavior

The list of sources in the question is a good start.


Initialization
Ddropout used in regulation
Mini-batch shuffle


If stochastic gradient descent is used in the back-propagation, that could be an additional source, and there may be others.  It is difficult to say without scrutinizing the code.  CDN emerged along different paths from CNN and MLP so there is no unambiguously distinct classical version of it.

The Right Way

Since the variance is a known phenominon and the norm, there may be no particular question for which the academic community needs to be consulted for an answer.

The right ways to deal with the non-repeatability of validation statistics related to object recognition also has a norm: Follow standard reporting conventions that apply to all experimental results reporting.

The only solution that is mathematically rigorous and not changing the experiment to produce a desired result (which is not science) is the first of the two found in your investigation.


The suggestion to perform $N$ runs where $N \ge 30$ and report mean and variance (not standard deviation) is reasonable and responsible.
The suggestion to do this with $O$ ADDITIONAL models where $O \ge 3$ is another such practice.


Freezing is Necessarily Arbitrary

Some would consider freezing pseudo-random sources arbitrarily something other than, "making reproductive results," and consider it fabricating reproducable results.

Unfortunately, the temptation to please the reader or corporate stakeholder occasionally preempts truth in reporting.  The result is counterproductive.  In the worst case, it leads to broad acceptance of the wrong way.  Those in the academic community should know that, whether or not they unanimously do.  Be a leader and do what is backed by solid reasoning, whether that is the first suggestion found or a third choice that does not involve modifying the experiment arbitrarily to force results to appear more exact than the algorithm, methodology, or framework actually is.