We explore whether useful temporal neural generative models can be learned from sequential data without back-propagation through time. We investigate the viability of a more neurocognitively-grounded approach in the context of unsupervised generative modeling of sequences. Specifically, we build on the concept of predictive coding, which has gained influence in cognitive science, in a neural framework. To do so we develop a novel architecture, the Temporal Neural Coding Network, and its learning algorithm, Discrepancy Reduction. The underlying directed generative model is fully recurrent, meaning that it employs structural feedback connections and temporal feedback connections, yielding information propagation cycles that create local learning signals. This facilitates a unified bottom-up and top-down approach for information transfer inside the architecture. Our proposed algorithm shows promise on the bouncing balls generative modeling problem. Further experiments could be conducted to explore the strengths and weaknesses of our approach.
@cite_1: A hybrid architecture is presented capable of online learning from both labeled and unlabeled samples. It combines both generative and discriminative objectives to derive a new variant of the Deep Belief Network, i.e., the Stacked Boltzmann Experts Network model. The model's training algorithm is built on principles developed from hybrid discriminative Boltzmann machines and composes deep architectures in a greedy fashion. It makes use of its inherent "layer-wise ensemble" nature to perform useful classification work. We (1) compare this architecture against a hybrid denoising autoencoder version of itself as well as several other models and (2) investigate training in the context of an incremental learning procedure. The best-performing hybrid model, the Stacked Boltzmann Experts Network, consistently outperforms all others.
@cite_6: Two novel deep hybrid architectures, the Deep Hybrid Boltzmann Machine and the Deep Hybrid Denoising Auto-encoder, are proposed for handling semi-supervised learning problems. The models combine experts that model relevant distributions at different levels of abstraction to improve overall predictive performance on discriminative tasks. Theoretical motivations and algorithms for joint learning for each are presented. We apply the new models to the domain of data-streams in work towards life-long learning. The proposed architectures show improved performance compared to a pseudo-labeled, drop-out rectifier network.
The desire for useful local learning, of which target propagation represents a strong modern step towards, is not new, and saw a small resurgence in the early days of training deeper networks in the form of layer-wise training of unsupervised models , supervised models , and semi-supervised models @cite_1 @cite_6 (also known as hybrid training). However, among the many problems with these early approaches to deep learning was the lack of global coordination. Global coordination means that higher-level layers essentially direct lower-level layers in what patterns they should be extracting. A lower-level feature detector might be able to find different aspects of structure in its input since multiple patterns might satisfy its layer-wise objective. However, this detector will only locate the right bit of structure needed for the whole model to make sense, at any time-step, if a higher-level layer signals what pattern it should be looking for. Since greedy layer-wise approaches build the model from the bottom-up, freezing the learned lower-level parameters, this coordination is impossible to achieve.