Training a neural network by classifying its own output
No, that doesn't seem likely to be a very useful direction.

Neural networks are a form of statistical machine learning.  The premise behind all statistical machine learning schemes -- including neural networks -- is that the instances in the training set need to come from the same distribution as the instances that you will apply the classifer to (i.e., the instances in the test set).

If you artificially generate instances via some arbitrary procedure, they might not come from the right distribution.  That will invalidate the core assumption behind statistical machine learning and may make the classifier perform poorly.

Your approach would only work if we knew that the neural network was able to generate images that come from the right distribution (i.e., the distribution on natural images that you'll want to classify).  But there's no reason to believe that will be the case.  In particular, I suspect it most likely won't be the case.

Or, to put it another way, your approach would only work if we had a way to characterize the probability distribution on natural images and sample from this distribution.  However, this appears to be as hard as building a classifier, or even harder.  So, there is no free lunch.

You might interested in the distinction between discriminative vs generative models.  Standard neural networks are discriminative models, and thus don't provide any (direct) way to generate images.  It is also possible to build generative models, which can generate images.  However, building a generative model is believed to be at least as hard as building a discriminative model, and probably even harder.  So, your proposal amounts to saying: "if I had a generative model, I could use that to build a training set to train a discriminative model".  But if building a generative model is likely even harder than building a discriminative model, so this doesn't seem to help much.  There's no free lunch.



Separately: You don't say how you plan to use a neural network to generate images.  Neural networks are a classifier, so they can classify an existing image, but not generate a new image.  You have to do something special to generate images from a neural network.