We explore whether useful temporal neural generative models can be learned from sequential data without back-propagation through time. We investigate the viability of a more neurocognitively-grounded approach in the context of unsupervised generative modeling of sequences. Specifically, we build on the concept of predictive coding, which has gained influence in cognitive science, in a neural framework. To do so we develop a novel architecture, the Temporal Neural Coding Network, and its learning algorithm, Discrepancy Reduction. The underlying directed generative model is fully recurrent, meaning that it employs structural feedback connections and temporal feedback connections, yielding information propagation cycles that create local learning signals. This facilitates a unified bottom-up and top-down approach for information transfer inside the architecture. Our proposed algorithm shows promise on the bouncing balls generative modeling problem. Further experiments could be conducted to explore the strengths and weaknesses of our approach.
@cite_6: Two novel deep hybrid architectures, the Deep Hybrid Boltzmann Machine and the Deep Hybrid Denoising Auto-encoder, are proposed for handling semi-supervised learning problems. The models combine experts that model relevant distributions at different levels of abstraction to improve overall predictive performance on discriminative tasks. Theoretical motivations and algorithms for joint learning for each are presented. We apply the new models to the domain of data-streams in work towards life-long learning. The proposed architectures show improved performance compared to a pseudo-labeled, drop-out rectifier network.
The TNCN's localized learning approach was also motivated by the simple learning algorithm , which showed that a stack of Boltzmann network modules (and other simple, auto-associative variations) could be learned in a pseudo-joint'' layerwise fashion. However, in order to build in some form of global coordination, a variation of back-propagation of errors was used, ultimately creating a global feedback path as part of the overall learning procedure. A more global approach was later presented in @cite_6 , incorporating top-down information much like that in , however, these algorithms were only built for and studied on stateless problems. Furthermore, these approaches would be difficult to scale when extended to sequential modeling problems given their strong dependence on Markov Chain Monte Carlo sampling.