Deep learning is currently the subject of intensive study. However, fundamental concepts such as representations are not formally defined -- researchers "know them when they see them" -- and there is no common language for describing and analyzing algorithms. This essay proposes an abstract framework that identifies the essential features of current practice and may provide a foundation for future developments. The backbone of almost all deep learning algorithms is backpropagation, which is simply a gradient computation distributed over a neural network. The main ingredients of the framework are thus, unsurprisingly: (i) game theory, to formalize distributed optimization; and (ii) communication protocols, to track the flow of zeroth and first-order information. The framework allows natural definitions of semantics (as the meaning encoded in functions), representations (as functions whose semantics is chosen to optimized a criterion) and grammars (as communication protocols equipped with first-order convergence guarantees). Much of the essay is spent discussing examples taken from the literature. The ultimate aim is to develop a graphical language for describing the structure of deep learning algorithms that backgrounds the details of the optimization procedure and foregrounds how the components interact. Inspiration is taken from probabilistic graphical models and factor graphs, which capture the essential structural features of multivariate distributions.
@cite_62: Unlike traditional reinforcement learning (RL), marketbased RL is in principle applicable to worlds described by partially observable Markov Decision Processes (POMDPs), where an agent needs to learn short-term memories of relevant previous events in order to execute optimal actions. Most previous work, however, has focused on reactive settings (MDPs) instead of POMDPs. Here we reimplement a recent approach to market-based RL and for the first time evaluate it in a toy POMDP setting.
@cite_64: We investigate cortical learning from the perspective of mechanism design. First, we show that discretizing standard models of neurons and synaptic plasticity leads to rational agents maximizing simple scoring rules. Second, our main result is that the scoring rules are proper, implying that neurons faithfully encode expected utilities in their synaptic weights and encode high-scoring outcomes in their spikes. Third, with this foundation in hand, we propose a biologically plausible mechanism whereby neurons backpropagate incentives which allows them to optimize their usefulness to the rest of cortex. Finally, experiments show that networks that backpropagate incentives can learn simple tasks.
@cite_91: Abstract It is well-known that chemical synaptic transmission is an unreliable process, but the function of such unreliability remains unclear. Here I consider the hypothesis that the randomness of synaptic transmission is harnessed by the brain for learning, in analogy to the way that genetic mutation is utilized by Darwinian evolution. This is possible if synapses are "hedonistic," responding to a global reward signal by increasing their probabilities of vesicle release or failure, depending on which action immediately preceded reward. Hedonistic synapses learn by computing a stochastic approximation to the gradient of the average reward. They are compatible with synaptic dynamics such as short-term facilitation and depression and with the intricacies of dendritic integration and action potential generation. A network of hedonistic synapses can be trained to perform a desired computation by administering reward appropriately, as illustrated here through numerical simulations of integrate-and-fire model neurons.
@cite_82: We propose a mathematical theory for how networks of neurons in the brain self-organize into functional networks, similarly to the self-organization of supply networks in a free-market economy. The theory is inspired by recent experimental results showing how information about changes to output synapses can travel backward along axons to affect a neuron's inputs. In neuronal development, competition for such retroaxonal'' signals determines which neurons live and which die. We suggest that in adults, an analogous form of competition occurs between neurons, to supply their targets with appropriate information in exchange for a payment'' returned to them backward along the axon. We review experimental evidence suggesting that neurotrophins may constitute such a signaling pathway in the adult brain. We construct a mathematical model, in which a small number of consumer'' neurons receive explicit fast error signals while a larger number of producer'' neurons compete to supply them with information, guided by retroaxonal signals from the consumers and from each other. We define a loss function to measure network performance and define the worth'' of a producer to be the increase in loss that would result if that neuron were to fall silent. We show how slow retroaxonal signals can allow producers to estimate their worth, and how these estimates allow the network to perform a form of parallel search over multiple producer cells. We validate our approximations and demonstrate the proposed learning rule using simulations.
@cite_98: We introduce a framework for training architectures composed of several modules. This framework, which uses a statistical formulation of learning systems, provides a unique formalism for describing many classical connectionist algorithms as well as complex systems where several algorithms interact. It allows to design hybrid systems which combine the advantages of connectionist algorithms as well as other learning algorithms.
@cite_70: Marvin Minsky--one of the fathers of computer science and cofounder of the Artificial Intelligence Laboratory at MIT--gives a revolutionary answer to the age-old question: "how does the mind work?"
@cite_53: Traditional views of neurotrophic factor biology held that trophic factors are released from target cells, retrogradely transported along their axons, and rapidly degraded upon arrival in cell bodies. Increasing evidence indicates that several trophic factors such as brain-derived neurotrophic factor (BDNF), fibroblast growth factor (FGF-2), glial cell-line derived neurotrophic factor (GDNF), insulin-like growth factor (IGF-I), and neurotrophin-3 (NT-3), can move anterogradely along axons. They can escape the degradative pathway upon internalization and are recycled for future uses. Internalized ligands can move through intermediary cells by transcytosis, presumably by endocytosis via endosomes to the Golgi system, by trafficking of the factor to dendrites or by sorting into anterograde axonal transport with subsequent release from axon terminals and uptake by second- or third-order target neurons. Such data suggest the existence of multiple “trophic currencies,” which may be used over several steps in neural networks to enable nurturing relationships between connected neurons or glial cells, not unlike currency exchanges between trading partners in the world economy. Functions of multistep transfer of trophic material through neural networks may include regulation of neuronal survival, differentiation of phenotypes and dendritic morphology, synapse plasticity, as well as excitatory neurotransmission. The molecular mechanisms of sorting, trafficking, and release of trophic factors from distinct neuronal compartments are important for an understanding of neurotrophism, but they present challenging tasks owing to the low levels of the endogeneous factors.
@cite_21: 
@cite_34: This article investigates how neurons can use metabolic cost to facilitate learning at a population level. Although decision-making by individual neurons has been extensively studied, questions regarding how neurons should behave to cooperate effectively remain largely unaddressed. Under assumptions that capture a few basic features of cortical neurons, we show that constraining reward maximization by metabolic cost aligns the information content of actions with their expected reward. Thus, metabolic cost provides a mechanism whereby neurons encode expected reward into their outputs. Further, aside from reducing energy expenditures, imposing a tight metabolic constraint also increases the accuracy of empirical estimates of rewards, increasing the robustness of distributed learning. Finally, we present two implementations of metabolically constrained learning that confirm our theoretical finding. These results suggest that metabolic cost may be an organizing principle underlying the neural code, and may also provide a useful guide to the design and analysis of other cooperating populations.
@cite_39: A plausible definition of "reasoning" could be "algebraically manipulating previously acquired knowledge in order to answer a new question". This definition covers first-order logical inference or probabilistic inference. It also includes much simpler manipulations commonly used to build large learning systems. For instance, we can build an optical character recognition system by first training a character segmenter, an isolated character recognizer, and a language model, using appropriate labelled training sets. Adequately concatenating these modules and fine tuning the resulting system can be viewed as an algebraic operation in a space of models. The resulting model answers a new question, that is, converting the image of a text page into a computer readable text. This observation suggests a conceptual continuity between algebraically rich inference systems, such as logical or probabilistic inference, and simple manipulations, such as the mere concatenation of trainable learning systems. Therefore, instead of trying to bridge the gap between machine learning systems and sophisticated "all-purpose" inference mechanisms, we can instead algebraically enrich the set of manipulations applicable to training systems, and build reasoning capabilities from the ground up.
@cite_43: 
@cite_23: Despite its size and complexity, the human cortex exhibits striking anatomical regularities, suggesting there may simple meta-algorithms underlying cortical learning and computation. We expect such meta-algorithms to be of interest since they need to operate quickly, scalably and effectively with little-to-no specialized assumptions. This note focuses on a specific question: How can neurons use vast quantities of unlabeled data to speed up learning from the comparatively rare labels provided by reward systems? As a partial answer, we propose randomized co-training as a biologically plausible meta-algorithm satisfying the above requirements. As evidence, we describe a biologically-inspired algorithm, Correlated Nystrom Views (XNV) that achieves state-of-the-art performance in semi-supervised learning, and sketch work in progress on a neuronal implementation.
@cite_2: A market-based algorithm is presented which autonomously apportions complex tasks to multiple cooperating agents giving each agent the motivation of improving performance of the whole system. A specific model, called “The Hayek Machine” is proposed and tested on a simulated Blocks World (BW) planning problem. Hayek learns to solve more complex BW problems than any previous learning algorithm. Given intermediate reward and simple features, it has learned to efficiently solve arbitrary BW problems. The Hayek Machine can also be seen as a model of evolutionary economics.
@cite_58: This paper suggests a learning-theoretic perspective on how synaptic plasticity benefits global brain functioning. We introduce a model, the selectron, that (i) arises as the fast time constant limit of leaky integrate-and-fire neurons equipped with spiking timing dependent plasticity (STDP) and (ii) is amenable to theoretical analysis. We show that the selectron encodes reward estimates into spikes and that an error bound on spikes is controlled by a spiking margin and the sum of synaptic weights. Moreover, the efficacy of spikes (their usefulness to other reward maximizing selectrons) also depends on total synaptic strength. Finally, based on our analysis, we propose a regularized version of STDP, and show the regularization improves the robustness of neuronal learning when faced with multiple stimuli.
@cite_20: Neurons deep in cortex interact with the environment extremely indirectly; the spikes they receive and produce are pre- and post-processed by millions of other neurons. This paper proposes two information-theoretic constraints guiding the production of spikes, that help ensure bursting activity deep in cortex relates meaningfully to events in the environment. First, neurons should emphasize selective responses with bursts. Second, neurons should propagate selective inputs by burst-firing in response to them. We show the constraints are necessary for bursts to dominate information-transfer within cortex, thereby providing a substrate allowing neurons to distribute credit amongst themselves. Finally, since synaptic plasticity degrades the ability of neurons to burst selectively, we argue that homeostatic regulation of synaptic weights is necessary, and that it is best performed offline during sleep.
@cite_11: 
Bottou and Gallinari proposed to decompose neural networks into cooperating modules @cite_98 @cite_39 . Decomposing more general algorithms or models into collections of interacting agents dates back to the shrieking demons that comprised Selfridge's Pandemonium @cite_43 and a long line of related work @cite_11 @cite_21 @cite_70 @cite_2 @cite_62 @cite_53 @cite_91 @cite_82 . The focus on components of neural networks as players, or rational agents, in their own right developed here derives from work aimed at modeling biological neurons game-theoretically, see @cite_58 @cite_34 @cite_20 @cite_23 @cite_64 .