We present a novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training. An instantiation of SWWAE uses a convolutional net (Convnet) ( (1998)) to encode the input, and employs a deconvolutional net (Deconvnet) ( (2010)) to produce the reconstruction. The objective function includes reconstruction terms that induce the hidden states in the Deconvnet to be similar to those of the Convnet. Each pooling layer produces two sets of variables: the "what" which are fed to the next layer, and its complementary variable "where" that are fed to the corresponding layer in the generative decoder.
@cite_27: We show how a deep denoising autoencoder with lateral connections can be used as an auxiliary unsupervised learning task to support supervised learning. The proposed model is trained to minimize simultaneously the sum of supervised and unsupervised cost functions by back-propagation, avoiding the need for layer-wise pretraining. It improves the state of the art significantly in the permutation-invariant MNIST classification task.
@cite_8: We combine supervised learning with unsupervised learning in deep neural networks. The proposed model is trained to simultaneously minimize the sum of supervised and unsupervised cost functions by backpropagation, avoiding the need for layer-wise pre-training. Our work builds on the Ladder network proposed by Valpola (2015), which we extend by combining the model with supervision. We show that the resulting model reaches state-of-the-art performance in semi-supervised MNIST and CIFAR-10 classification, in addition to permutation-invariant MNIST classification with all labels.
One recent work ( @cite_27 , @cite_8 ) has been proposed to adopt deep auto-encoders to support supervised learning in which completely different strategy is employed to harness the lateral connection between same stage encoder-decoder pairs, however. In that work, decoders receive the entire pre-pooled activation state from the encoder, whereas decoders from SWWAE only receive the where'' state from the corresponding encoder stages. Further, due to a lack of unpooling mechanism incorporated in the Ladder networks, it is restricted to only reconstruct the top layer within generative pathway ( @math model), which looses the "ladder" structure. By contrast, SWWAE doesn't suffer from such necessity.