Do Le et al. (2012) train all three autoencoder layers at a time, or just one? Le et al. 2012 use a network of 1 billion parameters to learn neurons that respond to faces, cats, pedestrians, etc. without labels (unsupervised).

Their network is built with three autoregressive layers, and six pooling and normalization layers.

In the paper they state,


  Optimization: All parameters in our model were
  trained jointly with the objective being the sum of the
  objectives of the three layers.


Does this mean that all three autoencoder layers were trained simultaneously, or that the first three sub-layers (first autoencoder sub-layer, first L2 pooling sub-layer, and first normalization sub-layer) were trained simultaneously?

I think by asking this question, I have answered it for myself. Please see my answer below.

Follow-on Question

What is the advantage of training all three layers simultaneously? Wouldn't the later layers be learning from poor lower layers to start with, and have to re-learn to adapt?

I do not have a very good answer to the follow-on question. Perhaps the lower layers can actually learn to provide low-level features that support the layers above?

This training of all layers simultaneously is done in all deep neural networks today -- AlexNet, VGG, etc. -- but why?

Reference

Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean, Andrew Y. Ng, Building high-level features using large scale unsupervised learning (2012) arXiv:1112.6209 [cs.LG]
The paper refers to layers and sub-layers, and clearly indicates that one layer includes all three sub-layers, so when they say they train all three layers simultaneously, they are talking about the three autoencoder layers, not the sub-layers.

This also agrees with the fact that only the autoencoder layer has tunable parameters.  The other two layers use uniform weights.

I do not have a good answer for the follow-on question yet... would love to hear what others think!