Why does applying PCA on targets causes underfitting? The goal:

I am new to machine learning and experimenting with neural networks.
I would like to build a network that takes as an input a series of 5 images and predicts the next image. My data set is completely artificial, just for my experimentation. As an illustration, here are a couple of examples of input and expected output:



The images of the data points and of the targets are from the same source: the target image of a data point appears in other data points, and vice-versa.

What I have done:

For now I have built a perceptron with one hidden layer and the output layer gives the pixels of the prediction. The two layers are dense and made of sigmoid neurons and I used the mean squared error as the objective. As the images are fairly simple and do not vary much, this does the job well: with 200-300 examples and 50 hidden units, I get a good error value (0.06) and good predictions on test data. The network is trained with gradient descent (with learning rate scaling). Here are the kinds of learning curves I get, and the error evolution with number of epochs:

 

What I am trying to do:

This is all good, but now I would like to reduce the dimensionality of the data set so that it would scale to bigger images and more examples. So I applied PCA. However I did not apply it on the list of data points but on the list of images, for two reasons:


On the data set as a whole, the convariance matrix would be 24000x24000, which does not fit in the memory of my laptop;
By doing on the images, I can also compress the targets, since they are made of the same images.


As the images look all similar, I managed to reduce their size from 4800 (40x40x3) to 36 while loosing only 1e-6 of the variance. 

What does not work:

When I feed my reduced data set and its reduced targets to the network, the gradient descent converges very fast to a high error (around 50 !). You can see the equivalent plots as the ones above:

 

I had not imagined that a learning curve could start at a high value, and then go down and back up... And what are the usual causes of gradient descent stopping so fast? Could it be linked to parameter initialization (I use GlorotUniform, the default of the lasagne library).

Then I noticed that if I feed the reduced data but the original (uncompressed) targets, I get back the initial performance. So it seems that applying PCA on the target images was not a good idea. Why is that? After all, I just multiplied the inputs and the targets by the same matrix, so the training input and target are still linked in a manner that the neural network should be able to figure out, no? What am I missing?

Even if I introduce and extra layer of 4800 units so that there is the same total number of sigmoid neurons, I get the same results. To sum up, I have tried:


24000 pixels => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
180 "pixels" => 50 sigmoids => 36 sigmoids (= 36 "pixels")
180 "pixels" => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 sigmoids (= 36 "pixels")
180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 linear (= 36 "pixels")


(1) and (3) work fine; but not (2), (4) and (5), and I don't understand why. In particular, since (3) works, (5) should be able to find the same parameters as (3) and the eigen vectors in the last linear layer. Is that not possible for a neural network?
First, thanks for the edits to your original question since we now know that you are applying the same transformation to all of your data.

Q: Why do perceptrons perform so much better than generalized linear models for some problems? A: Because they are inherently nonlinear models, with a great deal of flexibility.  The drawback is that the additional knobs require more data to correctly tune.

Big picture: 

Less data can cause high-bias.  High-bias can be overcome by more data.  You have reduced your data from a 4800 feature dataset to a 38 feature dataset, so should expect to see increased bias.  Neural networks require more data than models without hidden layers.

Linearity vs. Nonlinearity

Your artificial neural network (perceptron) is an inherently nonlinear model, yet you are deciding to remove features from your dataset with a linear model (PCA).  The presence of a single hidden layer explicitly creates second order terms in your data, and there are two additional nonlinear transformations (input==>hidden and hidden==>prediction), which add sigmoidal nonlinearity at each step.

The fact that the training data and the target data are both multiplied by the same matrix to achieve the reduced dimensionality really just means that your perceptron needs to learn the matrix if it wants to reconstruct nonlinear aspects of the original data.  This requires more data or for you to reduce less of the dimensionality.

An Experiment:

I suggest trying an experiment where you perform second order polynomial feature extraction on the full dataset and then performing PCA on that enhanced dataset.  See how many features you end up with while retaining 99% of the variance of the enhanced dataset.  If it is larger than the dimensionality of your initial dataset then stick with the un-enhanced and un-reduced dataset.  If it is betweeen the dimensionality of the original data and 38 then try training the perceptron with that data.

A better idea:

Rather than using the (linear) variance to determine the feature reduction of your PCA projection, try training and cross validating your model with different amounts of PCA dimensionality reduction.  You will likely find that there is a sweet spot for a given set of images.  For instance, an SVM on the MNIST digit data performs best when the 784 pixel features are reduced down to 50 linearly independent features using PCA.  Though this is not obvious from analyzing the variance of the principal components. 

Other options:

There are nonlinear dimensionality reduction techniques, such as isomap.  You could investigate the use of nonlinear feature reduction since you are clearly loosing information with the linear PCA that you have applied.

You could also look into image specific feature extraction techniques to add some nonlinearity before reducing the dimensionality.

Hope this helps!