Question: Represent the scientific article to find a short TLDR

Question: The information bottleneck method provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label, while minimizing the amount of other, superfluous information in the representation. The original formulation, however, requires labeled data in order to identify which information is superfluous. In this work, we extend this ability to the multi-view unsupervised setting, in which two views of the same underlying entity are provided but the label is unknown. This enables us to identify superfluous information as that which is not shared by both views. A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art on the Sketchy dataset and on label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to traditional unsupervised approaches for representation learning. The goal of deep representation learning is to transform a raw observational input, x, into a representation, z, to extract useful information. Significant progress has been made in deep learning via supervised representation learning, where the labels, y, for the downstream task are known while p(y|x) is learned directly . Due to the cost of acquiring large labeled datasets, a recently renewed focus on unsupervised representation learning seeks to generate representations, z, that allow learning of (a priori unknown) target supervised tasks more efficiently, i.e. with fewer labels . Our work is based on the information bottleneck principle stating that whenever a data representation discards information from the input which is not useful for a given task, it becomes less affected by nuisances, ing in increased robustness for downstream tasks. In the supervised setting, one can directly apply the information bottleneck method by minimizing the mutual information between z and x while simultaneously maximizing the mutual information between z and y . In the unsupervised setting, discarding only superfluous information is more challenging as without labels one cannot directly identify the relevant information. Recent literature (; van den) has instead focused on the InfoMax objective maximizing the mutual information between x and z, I(x, z), instead of minimizing it. In this paper, we extend the information bottleneck method to the unsupervised multi-view setting. To do this, we rely on a basic assumption of the multi-view literature -that each view provides the same task relevant information . Hence, one can improve generalization by discarding from the representation all information which is not shared by both views. We do this through an objective which maximizes the mutual information between the representations of the two views (Multi-View InfoMax) while at the same time reducing the mutual information between each view and its corresponding representation (as with the information bottleneck). The ing representation contains only the information shared by both views, eliminating the effect of independent factors of variations. Our contributions are three-fold: We extend the information bottleneck principle to the unsupervised multi-view setting and provide a rigorous theoretical analysis of its application. We define a new model that empirically leads to state-of-the-art in the low-data setting on two standard multi-view datasets, Sketchy and MIR-Flickr. By exploiting standard data augmentation techniques, we empirically show that the representations obtained with our model in single-view settings are more robust than other popular unsupervised approaches for representation learning, connecting our theory to the choice of augmentation function class. The challenge of representation learning can be formulated as finding a distribution p(z|x) that maps data observations x ∈ X into a code space z ∈ Z. Whenever the end goal involves predicting a label y, we consider only the z that are discriminative enough to identify the label. This requirement can be quantified by considering the amount of target information that remains accessible after encoding the data, and is known in literature as sufficiency of z for y : Definition 1. Sufficiency: A representation z of x is sufficient for y if and only if I(x; y|z) = 0 Any model that has access to a sufficient representation z must be able to predict y at least as accurately as if it has access to the original data x instead. In fact, z is sufficient for y if and only if the amount of information regarding the task is unchanged by the encoding procedure (see Proposition B.1 in the Appendix): I(x; y|z) = 0 ⇐⇒ I(x; y) = I(y; z). Among sufficient representations, the ones that in better generalization for unlabeled data instances are particularly appealing. When x has higher information content than y, some of the information in x must be irrelevant for the prediction task. This can be better understood by subdividing I(x; z) into three components by using the chain rule of mutual information (see Appendix A): I(x; z) = I(x; z|y) Conditional mutual information I(x; z|y) represents the information in z that is not predictive of y, i.e. superfluous information. While I(x; y) is a constant determined by how much label information is accessible from the raw observations; the last term I(x; y|z) represents the amount of information regarding y that is lost by encoding x into z. Note that this last term is zero whenever z is sufficient for x. Since the amount of predictive information I(x; y) is fixed, Proposition 2.1. A sufficient representation z of x for y is minimal whenever I(x; z|y) is minimal. Minimizing the amount of superfluous information can be done directly only in supervised settings. In fact, reducing I(x; z) without violating the sufficiency constraint necessarily requires making some additional assumptions on the predictive task (see Theorem B.1 in the Appendix). In Section 3 we describe a strategy to safely reduce the information content of a representation even when the label y is not observed, by exploiting redundant information in the form of an additional view on the data. Let v 1 and v 2 be two images of the same object from different view-points and y be its label. Assuming that the object is clearly distinguishable from both v 1 and v 2, any representation z with all the information that is accessible from both views would also contain the necessary label information. Furthermore, if z captures only the details that are visible from both pictures, it would reduce the total information content, discarding the view-specific details and reducing the sensitivity of the representation to view-changes. The theory to support this intuition is described in the following where v 1 and v 2 are jointly observed and referred to as data-views. In this section we extend our analysis of sufficiency and robustness to the multi-view setting. Intuitively, we can guarantee that z is sufficient for predicting y even without knowing y by ensuring that z maintains all information which is shared by v 1 and v 2. This intuition relies on a basic assumption of the multi-view environment -that each view provides the same task relevant information. To formalize this we define redundancy. Definition 2. Redundancy: v 1 is redundant with respect to v 2 for y if and only if I(y; v 1 |v 2) = 0 Intuitively, a view v 1 is redundant for a task whenever it is irrelevant for the prediction of y when v 2 is already observed. Whenever v 1 and v 2 are mutually redundant (v 1 is redundant with respect to v 2 for y, and vice-versa), we can show the following: Corollary 1. Let v 1 and v 2 be two mutually redundant views for a target y and let z 1 be a representation of v 1. If z 1 is sufficient for v 2 (I(v 1 ; v 2 |z 1) = 0) then z 1 is as predictive for y as the joint observation of the two views (I(v 1 v 2 ; y) = I(y; z 1)). In other words, whenever it is possible to assume mutual redundancy, any representation which contains all the information shared by both views (the redundant information) is as useful as their joint observation for predicting the label y. By factorizing the mutual information between v 1 and z 1 analogously to Equation 2, we can identify 3 components:. Since I(v 1 ; v 2) is a constant that depends only on the two views and I(v 1 ; v 2 |z 1) must be zero if we want the representation to be sufficient for the label, we conclude that I(v 1 ; z 1) can be reduced by minimizing I(v 1 ; z 1 |v 2). This term intuitively represents the information z 1 contains which is unique to v 1 and not shared by v 2. Since we assumed mutual redundancy between the two views, this information must be irrelevant for the predictive task and, therefore, it can be safely discarded. The proofs and formal assertions for the above statements and Corollary 1 can be found in Appendix B. The less the two views have in common, the more I(z 1 ; v 1) can be reduced without violating sufficiency for the label, the more robust the ing representation. At the extreme, v 1 and v 2 share only label information, in which case we can show that z 1 is minimal for y and our method is identical to the supervised information bottleneck method without needing to access the labels. Conversely, if v 1 and v 2 are identical, then our method degenerates to the InfoMax principle since no information can be safely discarded (see Appendix E). Given v 1 and v 2 that satisfy the mutual redundancy condition for a label y, we would like to define an objective function for the representation z 1 of v 1 that discards as much information as possible without losing any label information. In Section 3.1 we showed that we can maintain sufficiency for y by ensuring that I(v 1 ; v 2 |z 1) = 0, and that decreasing I(z 1 ; v 1 |v 2) will increase the robustness of the representation by discarding irrelevant information. So if we combine these two terms using a relaxed Lagrangian objective, then we obtain: where θ denotes the dependency on the parameters of the encoder p θ (z 1 |v 1), and λ 1 represents the Lagrangian multiplier introduced by the constrained optimization. Symmetrically, we define a Algorithm 1: Figure 1: Visualizing our Multi-View Information Bottleneck model for both multi-view and singleview settings. Whenever p(v 1) and p(v 2) have the same distribution, the two encoders can share their parameters. loss L 2 to optimize the parameters ψ of a conditional distribution p ψ (z 2 |v 2) that defines a robust sufficient representation z 2 of the second view v 2: Although L 1 and L 2 can not be computed directly, by defining z 1 and z 2 on the same domain Z and re-parametrizing the Lagrangian multipliers, their sum can be upper bounded as follows: sufficiency of z1 and z2 for predicting y where D SKL is the symmetrized KL divergence obtained by averaging D KL (p θ (z 1 |v 1)||p ψ (z 2 |v 2)) and D KL (p ψ (z 2 |v 2)||p θ (z 1 |v 1)), while the coefficient β defines the trade-off between sufficiency and robustness of the representation, which is a hyper-parameter in this work. The ing MultiView Infomation Bottleneck (MIB) model (Equation 5) is visualized in Figure 1, while the batchbased computation of the loss function is summarized in Algorithm 1. The symmetrized KL divergence D SKL (p θ (z 1 |v 1)||p ψ (z 2 |v 2)) can be computed directly whenever p θ (z 1 |v 1) and p ψ (z 2 |v 2) have a known density, while the mutual information between the two representations I θψ (z 1 ; z 2) can be maximized by using any sample-based differentiable mutual information lower bound. Both the Jensen-Shannon I JS and the InfoNCE I NCE (van den) estimators used in this work require introducing an auxiliary parameteric model C ξ (z 1, z 2), which is jointly optimized during the training procedure. The full derivation for the MIB loss function can be found in Appendix F. In this section, we introduce a methodology to build mutually redundant views starting from single observations x with domain X by exploiting known symmetries of the task. By picking a class T of functions t: X → W that do not affect label information, it is possible to artificially build views that satisfy mutual redundancy for y with a procedure similar to dataaugmentation. Let t 1 and t 2 be two random variables over T, then v 1:= t 1 (x) and v 2:= t 2 (x) must be mutually redundant for y. Since no function in T affects label information (I(v 1 ; y) = I(v 2 ; y) = I(x; y)), a representation z 1 of v 1 that is sufficient for v 2 must contain same amount of predictive information as x. Formal proofs can be found in Appendix B.4. Whenever the two transformations for the same observations are independent (I(t 1 ; t 2 |x) = 0), they introduce uncorrelated variations in the two views. As an example, if T represents a set of small translations, the two views will differ by a small shift. Since this information is not shared, z 1 that contains only common information between v 1 and v 2 will discard fine-grained details regarding the position. For single-view datasets, we generate the two views v 1 and v 2 by independently sampling two functions from the same function class T with uniform probability. Since the ing t 1 and t 2 have the same distribution, the two generated views will also have the same marginals. For this reason, the two conditional distributions p θ (z 1 |v 1) and p ψ (z 2 |v 2) can share their parameters and only one encoder can be used. Full (or partial) parameter sharing can be also applied in the multiview settings whenever the two views have the same (or similar) marginal distributions. The space of all the possible representations z of x for a predictive task y can be represented as a region in the Information Plane . Each representation is characterised by the amount of information regarding the raw observation I(x; z) and the corresponding measure of accessible predictive information I(y; z) (x and y axis respectively on Figure 2). Ideally, a good representation would be maximally informative about the label while retaining a minimal amount of information from the observations (top left corner of the parallelogram). Further details on the Information Plane and the bounds visualized in Figure 2 are described in Appendix C. Thanks to recent progress in mutual information estimation (; ;), the InfoMax principle has gained attention for unsupervised representation learning (; van den). Since the InfoMax objective involves maximizing I(x; z), the ing representation aims to preserve all the information regarding the raw observations (top right corner in Figure 2). Despite their success, has shown that the effectiveness of the InfoMax models is due to inductive biases introduced by the architecture and estimators rather than the training objective itself, since the InfoMax objective can be trivially maximized by using invertible encoders. On the other hand, Variational Autoencoders (VAEs) define a training objective that balances compression and reconstruction error through an hyper-parameter β. Whenever β is close to 0, the VAE objective aims for a lossless representation, approaching the same region of the Information Plane as the one targeted by InfoMax . When β approaches large values, the representation becomes more compressed, showing increased generalization and disentanglement , and, as β approaches infinity, I(z; x) goes to zero. During this transition from low to high β, however, there are no guarantees that VAEs will retain label information (Theorem B.1 in the Appendix). The path between the two regimes depends on how well the label information aligns with the inductive bias introduced by encoder , prior (and decoder architectures . Concurrent work applies the InfoMax principle in Multi-View settings (; Hénaff et al., 2019; ;), aiming to maximize mutual information between the representation z of a first data-view x and a second one v 2. The target representation for the MultiView InfoMax (MV-InfoMax) models should contain at least the amount of information in x that is predictive for v 2, targeting the region I(z; x) ≥ I(x; v 2) on the Information Plane. Whenever x is redundant with respect to v 2 for y, the representation must be also sufficient for y (Corollary 1). Since z has no incentive in discarding any information regarding x, a representation that is opti- Method mAP@all Prec@200 SaN 0.208 0.292 GN Triplet 0.529 0.716 Siamese CNN 0.481 0.612 Siamese-AlexNet Table 1: Examples of the two views and class label from the Sketchy dataset (on the left) and comparison between MIB and other popular models in literature on the sketch-based image retrieval task (on the right). * denotes models that use a 64-bits binary representation. The for MIB corresponds to β = 1. mal according to the InfoMax principle is also optimal for MV-InfoMax. Our model with β = 0 (Equation 5) belong to this family of objectives since the minimality term is discarded. In contrast to all of the above, our work is the first to explicitly identify and discard superfluous information from the representation in the unsupervised multi-view setting. The idea of discarding irrelevant information was introduced in and identified as one of the possible reasons behind the generalization capabilities of deep neural networks by and. The direct removal of superfluous information has, so far, been done only in supervised settings . Conversely, β-VAE models remove information indiscriminately without identifying which part is superfluous, and the InfoMax and Multi-View InfoMax methods do not explicitly try to remove superfluous information at all. In fact, among the representations that are optimal according to Multi-View InfoMax (purple dotted line in Figure 2), the MIB objective in the representation with the least superfluous information, i.e. the most robust. In this section we demonstrate the effectiveness of our model against state-of-the-art baselines in both the multi-view and single-view setting. In the single-view setting, we also estimate the coordinates on the Information Plane for each of the baseline methods as well as our method to validate the theory in Section 3. The reported in the following sections are obtained using the Jensen-Shannon I JS estimator, which ed in better performance for MIB and the other InfoMax-based models (Table 2 in the supplementary material). In order to facilitate the comparison between the effect of the different loss functions, the same estimator is used across the different models. We compare MIB on the sketch-based image retrieval and Flickr multiclass image classification tasks with domain specific and prior multi-view learning methods. Sketchy The Sketchy dataset consists of 12,500 images and 75,471 handdrawn sketches of objects from 125 classes. As in , we also include another 60,502 images from the ImageNet from the same classes, which in total 73,002 natural object images. As per the experimental protocol of , a total of 6,250 sketches (50 sketches per category) are randomly selected and removed from the training set for testing purpose, which leaves 69,221 sketches for training the model. The sketch-based image retrieval task is a ranking of 73,002 natural images according to the unseen test (query) sketch. Retrieval is done for our model by generating representations for the query sketch as well as all natural images, . The ing flattened 4096-dimensional feature vectors are fed to our image and sketch encoders to produce a 64-dimensional representation. Both encoders consist of neural networks with hidden layers of 2048 and 1024 units respectively. Size of the representation and regularization strength β are tuned on a validation sub-split. We evaluate MIB on five different train/test splits 1 and report mean and standard deviation in Table 5.1. Further details on our training procedure and architecture are in Appendix G. Table 5.1 shows that the our model achieves strong performance for both mean average precision (mAP@all) and precision at 200 (Prec@200), suggesting that the representation is able to capture the common class information between the paired pictures and sketches. The effectiveness of MIB on the retrieval task can be mostly imputed to the regularization introduced with the symmetrized KL divergence between the two encoded views. Other than discarding view-private information, this term actively aligns the representations of v 1 and v 2, making the MIB model especially suitable for retrieval tasks MIR-Flickr The MIR-Flickr dataset consists of 1M images annotated with 800K distinct user tags. Each image is represented by a vector of 3,857 hand-crafted image features (v 1), while the 2,000 most frequent tags are used to produce a 2000-dimensional multihot encoding (v 2) for each picture. The dataset is divided into labeled and unlabeled sets that respectively contain 975K and 25K images, where the labeled set also contains 38 distinct topic classes together with the user tags. Training images with less than two tags are removed, which reduces the total number of training samples to 749,647 pairs . The labeled set contains 5 different splits of train, validation and test sets of size 10K/5K/10K respectively. Following a standard procedure in literature , we train our model on the unlabeled pairs of images and tags. Then a multi-label logistic classifier is trained from the representation of 10K labeled train images to the corresponding macro-categories. The quality of the 1 Processed dataset and splits will be publicly released on paper acceptance 2 These are included only for completeness, as the Multi-View InfoMax objective does not produce consistent representations for the two views so there is no straight-forward way to use it for ranking. representation is assessed based on the performance of the trained logistic classifier on the labeled test set. Each encoder consists of a multi-layer perceptron of 4 hidden layers with ReLU activations learning two 1024-dimensional representations z 1 and z 2 for images v 1 and tags v 2 respectively. Examples of the two views, labels, and further details on the training procedure are in Appendix G. Our MIB model is compared with other popular multi-view learning models in Figure 3 for β = 0 (Multi-View InfoMax), β = 1 and β = 10 −3 (best on validation set). Although the tuned MIB performs similarly to Multi-View InfoMax with a large number of labels, it outperforms it when fewer labels are available. Furthermore, by choosing a larger β the accuracy of our model drastically increases in scarce label regimes, while slightly reducing the accuracy when all the labels are observed (see right side of Figure 3). This effect is likely due to a violation of the mutual redundancy constraint (see Figure 6 in the supplementary material) which can be compensated with smaller values of β for less aggressive compression. A possible reason for the effectiveness of MIB against some of the other baselines may be our ability to use mutual information estimators that do not require reconstruction. Both Multi-View VAE (MVAE) and Deep Variational CCA (VCCA) rely on a reconstruction term to capture crossmodal information, which can introduce bias that decreases performance. In this section, we compare the performance of different unsupervised learning models by measuring their data efficiency and empirically estimating the coordinates of their representation on the Information Plane. Since accurate estimation of mutual information is extremely expensive , we focus on relatively small experiments that aim to uncover the difference between popular approaches for representation learning. The dataset is generated from MNIST by creating the two views, v 1 and v 2, via the application of data augmentation consisting of small affine transformations and independent pixel corruption to each image. These are kept small enough to ensure that label information is not effected. Each pair of views is generated from the same underlying image, so no label information is used in this process (details in Appendix G). To evaluate, we train the encoders using the unlabeled multi-view dataset just described, and then fix the representation model. A logistic regression model is trained using the ing representations along with a subset of labels for the training set, and we report the accuracy of this model on a disjoint test set as is standard for the unsupervised representation learning literature (; ; van den). We estimate I(x; z) and I(y; z) using mutual information estimation networks trained from scratch on the final representations using batches of joint samples {( ∼ p(x, y)p θ (z|x). All models are trained using the same encoder architecture consisting of 2 layers of 1024 hidden units with ReLU activations, ing in 64-dimensional representations. The same data augmentation procedure was also applied for single-view architectures and models were trained for 1 million iterations with batch size B = 64. Figure 4 summarizes the . The empirical measurements of mutual information reported on the Information Plane are consistent with the theoretical analysis reported in Section 4: models that retain less information about the data while maintaining the maximal amount of predictive information, in better classification performance at low-label regimes, confirming the hypothesis that discarding irrelevant information yields robustness and more data-efficient representations. Notably, the MIB model with β = 1 retains almost exclusively label information, hardly decreasing the classification performance when only one label is used for each data point. In this work, we introduce Multi-View Information Bottleneck, a novel method that relies on multiple data-views to produce robust representation for downstream tasks. Most of the multi-view literature operates under the assumption that each view is individually sufficient for determining the label , while our method only requires the weaker mutual redundancy condition outlined in Section 3, enabling it to be applied to any traditional multi-view task. In our experiments, we compared MIB empirically against other approaches in the literature on three such tasks: sketch-based image retrieval, multi-view and unsupervised representation learning. The strong performance obtained in the different areas show that Multi-View Information Bottleneck can be practically applied to various tasks for which the paired observations are either available or are artificially produced. Furthermore, the positive on the MIR-Flickr dataset show that our model can work well in practice even when mutual redundancy holds only approximately. There are multiple extensions that we would like to explore in future work. One interesting direction would be considering more than two views. In Appendix D we discuss why the mutual redundancy condition cannot be trivially extended to more than two views, but we still believe such an extension is possible. Secondly, we believe that exploring the role played by different choices of data augmentation could bridge the gap between the Information Bottleneck principle and with the literature on invariant neural networks , which are able to exploit known symmetries and structure of the data to remove superfluous information. In this section we enumerate some of the properties of mutual information that are used to prove the theorems reported in this work. For any random variables w, x, y and z: (P 1) Positivity: I(x; y) ≥ 0, I(x; y|z) ≥ 0 (P 2) Chain rule: = I(x; y) − I(y; z) Since both I(x; y) and I(y; z) are non-negative (P 1), I(x; y|z) = 0 ⇐⇒ I(y; z) = I(x; y) Theorem B.1. Let x, z and y be random variables with joint distribution p(x, y, z). Let z be a representation of x that satisfies I(x; z) > I(x; z), then it is always possible to find a label y for which z is not predictive for y while z is. (H 1) I(y; z |x) = 0 (T 1) I(x; z) < I(x; z) =⇒ ∃y. I(y; z) > I(y; z) = 0 Proof. By construction. 1. We first factorize x as a function of two independent random variables (Proposition 2.1) by picking y such that: for some deterministic function f. Note that such y always exists. 2. Since x is a function of y and z: (C 4) I(x; z|yz) = 0 Considering I(y; z): = I(y; z|x) + I(x; y; z) Whenever I(x; z) > I(x; z), I(y; z) must be strictly positive, while I(y; z) = 0 by construction. Therefore such y exists. Corollary B.1.1. Let z be a representation of x that discards observational information. There is always a label y for which a z is not predictive, while the original observations are. Hypothesis: Thesis: (T 1) ∃y. I(y; x) > I(y; z) = 0 Proof. By construction using Theorem B.1. 1. Set z = x: Since the hypothesis are met, we conclude that there exist y such that I(y; x) > I(y; z) = 0 Hypothesis: (H 1) I(y; z 1 |v 2 v 1) = 0 Thesis: Proof. Since z 1 is a representation of v 1: Therefore: Proposition B.3. Let v 1 be a redundant view with respect to v 2 for y. Any representation z 1 of v 1 that is sufficient for v 2 is also sufficient for y. (H 1) I(y; z 1 |v 2 v 1) = 0 (H 2) I(y; v 1 |v 2) = 0 Thesis: Proof. Using the from Theorem B.2: (H 1) I(y; z 1 |v 1 v 2) = 0 Thesis: Proof. I(y; z 1) Corollary B.2.1. Let v 1 and v 2 be mutually redundant views for y. Let z 1 be a representation of v 1 that is sufficient for v 2. Then: Hypothesis: Thesis: Proof. Using Theorem B.2 Since I(y; z 1) ≤ I(y; v 1 v 2) is a consequence of the data processing inequality, we conclude that I(y; z 1) = I(y; v 1 v 2) Let x and y be random variables with domain X and Y respectively. Let T be a class of functions t: X → W and let t 1 and t 2 be a random variables over T that depends only on x. For the theorems and corollaries discussed in this section, we are going to consider the independence assumption that can be derived from the graphical model G reported in Figure 5. Figure 5: Visualization of the graphical model G that relates the observations x, label y, functions used for augmentation t 1, t 2 and the representation z 1. Proposition B.4. Whenever I(t 1 (x); y) = I(t 2 (x); y) = I(x; y) the two views t 1 (x) and t 2 (x) must be mutually redundant for y. (H 1) Independence relations determined by G Thesis: (T 1) I(t 1 (x); y) = I(t 2 (x); y) = I(x; y) =⇒ I(t 1 (x); y|t 2 (x)) + I(t 2 (x); y|t 1 (x)) = 0 Proof. (C 1) I(t 1 (x); y|xt 2 (x)) = 0 (C 2) I(y; t 2 (x)|x) = 0 2. Since t 2 (x) is uniquely determined by x and t 2: (C 3) I(t 2 (x); y|xt 2 ) = 0 3. Consider I(t 1 (x); y|t 2 (x)) I(t 1 (x); y|t 2 (x)) (P3) = I(t 1 (x); y|xt 2 (x)) + I(t 1 (x); y; x|t 2 (x)) (C1) = I(t 1 (x); y; x|t 2 (x)) (P3) = I(y; x|t 2 (x)) − I(y; x|t 1 (x)t 2 (x)) (P1) ≤ I(y; x|t 2 (x)) (P3) = I(y; x) − I(y; x; t 2 (x)) (P3) = I(y; x) − I(y; t 2 (x)) + I(y; t 2 (x)|x) (P3) = I(y; x) − I(y; t 2 (x)) + I(y; t 2 (x)|t 2 x) + I(y; t 2 (x); t 2 |x) (C3) = I(y; x) − I(y; t 2 (x)) + I(y; t 2 (x); t 2 |x) (P3) = I(y; x) − I(y; t 2 (x)) + I(y; t 2 (x)|x) − I(y; t 2 (x)|t 2 x) (P1) ≥ I(y; x) − I(y; t 2 (x)) + I(y; t 2 (x)|x) (C2) ≥ I(y; x) − I(y; t 2 (x)) Therefore I(y; x) = I(y; t 2 (x)) =⇒ I(t 1 (x); y|t 2 (x)) = 0 The proof for I(y; x) = I(y; t 1 (x)) =⇒ I(t 2 (x); y|t 1 (x)) = 0 is symmetric, therefore we conclude I(t 1 (x); y) = I(t 2 (x); y) = I(x; y) =⇒ I(t 1 (x); y|t 2 (x)) + I(t 2 (x); y|t 1 (x)) = 0 Theorem B.3. Let I(t 1 (x); y) = I(t 2 (x); y) = I(x; y). Let z 1 be a representation of t 1 (x). If z 1 is sufficient for t 2 (x) then I(x; y) = I(y; z 1). (H 1) Independence relations determined by G (H 2) I(t 1 (x); y) = I(t 2 (x); y) = I(x; y) Thesis: (T 1) I(t 1 (x); t 2 (x)|z 1 ) = 0 =⇒ I(x; y) = I(y; z 1) Proof. Since t 1 (x) is redundant for t 2 (x) (Proposition B.4) any representation z 1 of t 1 (x) that is sufficient for t 2 (x) must also be sufficient for y (Theorem B.2). Using Proposition B.1 we have I(y; z 1) = I(y; t 1 (x)). Since I(y; t 1 (x)) = I(y; x) by hypothesis, we conclude I(x; y) = I(y; z 1) Every representation z of x must satisfy the following constraints: • 0 ≤ I(y; z) ≤ I(x; y): The amount of label information ranges from 0 to the total predictive information accessible from the raw observations I(x; y). • I(y; z) ≤ I(x; z) ≤ I(y; z) + H(x|y): The representation must contain more information about the observations than about the label. When x is discrete, the amount of discarded label information I(x; y) − I(y; z) must be smaller than the amount of discarded observational information H(x) − I(x; z), which implies I(x; z) ≤ I(y; z) + H(x|y). Proof. Since z is a representation of x: Considering the four bounds separately: = H(x|y) + I(y; z) Note that (H 2) is needed only to prove bound 4. For continuous x bounds 1, 2 and 3 still hold. The mutual redundancy condition between two views v 1 and v 2 for a label y can not be trivially extended to an arbitrary number of views, as the relation is not transitive because of some higher order interaction between the different views and the label. This can be shown with a simple example. Given three views v 1, v 2 and v 3 and a task y such that: • v 1 and v 2 are mutually redundant for y • v 2 and v 3 are mutually redundant for y Then, v 1 is not necessarily mutually redundant with respect to v 3 for y. We can show this with a simple example, Let v 1, v 2 and v 3 be fair and independent binary random variables. Defining y as the exclusive or of v 1 and v 3 (y := v 1 XOR v 3), we have that I(v 1 ; y) = I(v 3 ; y) = 0. In this settings, v 1 and v 2 are mutually redundant for y: Analogously, v 2 and v 3 are also mutually redundant for y as the three random variables are not predictive for each other. Nevertheless, v 1 and v 3 and not mutually redundant for y: Where H(v 1 |v 3 y) = H(v 3 |v 1 y) = 0 follows from v 1 = v 3 XOR y and v 3 = v 1 XOR y, while H(v 1) = H(v 3) = 1 holds by construction. This counter-intuitive higher order interaction between multiple views makes our theory non-trivial to generalize to more than two views, requiring an extension of our theory to ensure sufficiency for the label. Different objectives in literature can be seen as a special case of the Multi-View Information Bottleneck principle. In this section we show that the supervised version of Information Bottleneck is equivalent to the corresponding Multi-View version whenever the two redundant views have only label information in common. A second subsection show equivalence between InfoMax and MultiView Information Bottleneck whenever the two views are identical. Whenever the two mutually redundant views v 1 and v 2 have only label information in common (or when one of the two views is the label itself) the Multi-View Information Bottleneck objective is equivalent to the respective supervised version. This can be shown by proving that I(v 1 ; z 1 |v 2) = I(v 1 ; z 1 |y), i.e. a representation z 1 of v 1 that is sufficient and minimal for v 2 is also sufficient and minimal for y. Proposition E.1. Let v 1 and v 2 be mutually redundant views for a label y that share only label information. Then a sufficient representation z 1 of v 1 for v 2 that is minimal for v 2 is also a minimal representation for y. Hypothesis: Thesis: Proof. 1. Consider I(v 1 ; z): = I(v 1 ; z 1 |v 2) + I(v 1 ; y) 2. Using Corollary 1, from (H 2) and (H 3) follows I(v 1 ; y|z 1) = 0 3. I(v 1 ; z) can be alternatively expressed as: = I(v 1 ; z 1 |y) + I(v 1 ; y) Equating 1 and 3, we conclude I(v 1 ; z 1 |v 2) = I(v 1 ; z 1 |y). Whenever v 1 = v 2, a representation z 1 of v 1 that is sufficient for v 1 must contain all the original information. Furthermore since I(v 1 ; z 1 |v 1) = 0 for every representation, no superfluous information can be identified and removed. As a consequence, a minimal sufficient representation z 1 of v 1 for v 1 is any representation for which mutual information is maximal, hence InfoMax. Starting from Equation 3, we consider the sum of the losses L 1 (θ; λ 1) and L 2 (ψ; λ 2) that aim to create the minimal sufficient representations z 1 and z 2 respectively: Considering z 1 and z 2 on the same domain Z, I θ (v 1 ; z 1 |v 2) can be expressed as: Note that the bound is tight whenever p ψ (z 2 |v 2) coincides with p θ (z 1 |v 2). This happens whenever z 1 and z 2 produce a consistent encoding. Analogously I ψ (v 2 ; z 2 |v 1) is upper bounded by D KL (p ψ (z 2 |v 2)||p θ (z 1 |v 1)). I θ (v 1 ; v 2 |z 1) can be rephrased as: * follows from z 2 representation of v 2. The bound reported in this equation is tight whenever z 2 is sufficient for z 1. This happens whenever z 2 contains all the information regarding z 1 (and therefore v 1). Once again, the same bound can symmetrically be used to define I θ (v 1 ; v 2 |z 2) ≤ I(v 1 ; v 2) − I θψ (z 1 ; z 2). Since I(v 1 ; v 2) is constant in θ and ψ, the loss function in Equation 6 can be upper-bounded with; Where: Lastly, multiplying both terms with β:= 2 λ1+λ2 and re-parametrizing the objective, we obtain: G EXPERIMENTAL PROCEDURE AND DETAILS The two stochastic encoders p θ (z 1 |v 1) and p ψ (z 2 |v 2) are modeled by Normal distributions parametrized with neural networks (µ θ, σ 2 θ) and (µ ψ, σ 2 ψ) respectively: Since the density of the two encoders can be evaluated, the symmetrized KL-divergence in equation 4 can be directly computed. On the other hand, I θψ (z 1 ; z 2) requires the use of a mutual information estimator. To facilitate the optimization, the hyper-parameter β is slowly increased during training, starting from a small value ≈ 10 −4 to its final value with an exponential schedule. This is because the mutual information estimator is trained together with the other architectures and, since it starts from a random initialization, it requires an initial warm-up. Starting with bigger β in the encoder collapsing into a fixed representation. The update policy for the hyper-parameter during training has not shown strong influence on the representation, as long as the mutual information estimator network has reached full capacity. All the experiments have been performed using the Adam optimizer with a learning rate of 10 • Input: The two views for the sketch-based classification task consist of 4096 dimensional sketch and image features extracted from two distinct VGG-16 network models which were pre-trained on images and sketches from the TU-Berlin dataset for endto-end classification. The feature extractors are frozen during the training procedure of for the two representations. Each training iteration used batches of size B = 128. • Encoder and Critic architectures: Both sketch and image encoders consist of multi-layer perceptrons of 2 hidden ReLU units of size 2,048 and 1,024 respectively with an output of size 2x64 that parametrizes mean and variance for the two Gaussian posteriors. The critic architecture also consists of a multi layer perceptron of 2 hidden ReLU units of size 512. • β update policy: The initial value of β is set to 10 −4. Starting from the 10,000 th training iteration, the value of β is exponentially increased up to 1.0 during the following 250,000 training iterations. The value of β is then kept fixed to one until the end of the training procedure (500,000 iterations). • Evaluation: All natural images are used as both training sets and retrieval galleries. The 64 dimensional real outputs of sketch and image representation are compared using Euclidean distance. For having a fair comparison other methods that rely on binary hashing , we used Hamming distance on a binarized representation (obtained by applying iterative quantization on our real valued representation). We report the mean average precision (mAP@all) and precision at toprank 200 (Prec@200) on both the real and binary representation to evaluate our method and compare it with prior works. • Input: Whitening is applied to the handcrafted image features. Batches of size B = 128 are used for each update step. • Encoders and Critic architectures: The two encoders consists of a multi layer perceptron of 4 hidden ReLU units of size 1,024, which exactly resemble the architecture used in Figure 6: Examples of pictures v 1, tags v 2 and category labels y for the MIR-Flickr dataset . As visualized is the second row, the tags are not always predictive of the label. For this reason, the mutual redundancy assumption holds only approximately. "watermelon", "hilarious", "chihuahua", "dog" "animals", "dog", "food" "colors", "cores", "centro", "comercial", "building" "clouds", "sky", "structures". Both representations z 1 and z 2 have a size of 1,024, therefore the two architecture output a total of 2x1,024 parameters that define mean and variance of the respective factorized Gaussian posterior. Similarly to the Sketchy experiments, the critic is consists of a multi-layer perceptron of 2 hidden ReLU units of size 512. • β update policy: The initial value of β is set to 10 −8. Starting from 150000 th iteration, β is set to exponentially increase up to 1.0 (and 10 −3) during the following 150,000 iterations. • Evaluation: Once the models are trained on the unlabeled set, the representation of the 25,000 labeled images is computed. The ing vectors are used for training and evaluating a multi-label logistic regression classifier on the respective splits. The optimal parameters (such as β) for our model are chosen based on the performance on the validation set. In Table 3, we report the aggregated mean of the 5 test splits as the final value mean average precision value. • • Encoders, Decoders and Critic architectures: All the encoders used for the MNIST experiments consist of neural networks with two hidden layers of 1,024 units and ReLU activations, producing a 2x64-dimensional parameter vector that is used to parameterize mean and variance for the Gaussian posteriors. The decoders used for the VAE experiments also consist of the networks of the same size. Similarly, the critic architecture used for mutual information estimation consists of two hidden layers of 1,204 units each and ReLU activations. • β update policy: The initial value of β is set to 10 −3, which is increased with an exponential schedule starting from the 50,000 th until 1the 50,000 th iteration. The value of β is then kept constant until the 1,000,000 th iteration. The same annealing policy is used to trained the different β-VAEs reported in this work. • Evaluation: The trained representation are evaluated following the well-known protocol described in Tschannen et al.,000 iterations. The Jensen-Shannon mutual information lower bound is maximized during training, while the numerical estimation are computed using an energy-based bound . The final values for I(x; z) and I(y; z) are computed by averaging the mutual information estimation on the whole dataset. In order to reduce the variance of the estimator, the lowest and highest 5% are removed before averaging. This practical detail makes the estimation more consistent and less susceptible to numerical instabilities. In this section we include additional quantitative and visualizations which refer to the singleview MNIST experiments reported in section 5.2. Table 2: Comparison of the amount of input information I(x; z), label information I(z; y), and accuracy of a linear classifier trained with different amount of labeled Examples (Ex) for the models reported in Figure 4. Both the obtained using the Jensen-Shannon I JSD and the InfoNCE I NCE (van den) estimators are reported. Figure 7 reports the linear projection of the embedding obtained using the MIB model. The latent space appears to roughly consists of ten clusters which corresponds to the different digits. This observation is consistent with the empirical measurement of input and label information I(x; z) ≈ I(z; y) ≈ log 10, and the performance of the linear classifier in scarce label regimes. As the cluster are distinct and concentrated around the respective centroids, 10 labeled examples are sufficient to align the centroid coordinates with the digit labels. H ABLATION STUDIES H.1 DIFFERENT RANGES OF DATA AUGMENTATION Figure 8 visualizes the effect of different ranges of corruption probabily as data augmentation strategy to produce the two views v 1 and v 2. The MV-InfoMax Model does not seem to get any advantage from the use increasing amount of corruption, and it representation remains approximately in the same region of the information plane. On the other hand, the models trained with the MIB objective are able to take advantage of the augmentation to remove irrelevant data information and the representation transitions from the top right corner of the Information Plane (no-augmentation) to the top-left. When the amount of corruption approaches 100%, the mutual redundancy assumption is clearly violated, and the performances of MIB deteriorate. In the initial part of the transitions between the two regimes (which corresponds to extremely low probability of corruption) the MIB models drops some label information that is quickly re-gained when pixel corruption becomes more frequent. We hypothesize that this behavior is due to a problem with the optimization procedure, since the corruption are extremely unlikely, the Monte-Carlo estimation for the symmetrized Kullback-Leibler divergence is more biased. Using more examples of views produced from the same data-point within the same batch could mitigate this issue. The hyper-parameter β (Equation 5) determines the trade-off between sufficiency and minimality of the representation for the second data view. When β is zero, the training objective of MIB is equivalent to the Multi-View InfoMax target, since the representation has no incentive to discard any information. When 0 < β ≤ 1 the sufficiency constrain is enforced, while the superfluous information is gradually removed from the representation. Values of β > 1 can in representations that violate the sufficiency constraint, since the minimization of I(x; z|v 2) is prioritized. The trade-off ing from the choice of different β is visualized in Figure 9 and compared against β-VAE. Note that in each point of the pareto-front the MIB model in a better trade-off between I(x; z) and I(y; z) when compared to β-VAE. The effectiveness of the Multi-View Information Bottleneck model is also justified by the corresponding values of predictive accuracy. Published as a conference paper at ICLR 2020

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We extend the information bottleneck method to the unsupervised multiview setting and show state of the art results on standard datasets

Represent the short summary for finding the corresponding scientific article (long)


Question: Represent the scientific article to find a short TLDR

Question: The biological plausibility of the backpropagation algorithm has long been doubted by neuroscientists. Two major reasons are that neurons would need to send two different types of signal in the forward and backward phases, and that pairs of neurons would need to communicate through symmetric bidirectional connections. We present a simple two-phase learning procedure for fixed point recurrent networks that addresses both these issues. In our model, neurons perform leaky integration and synaptic weights are updated through a local mechanism. Our learning method extends the framework of Equilibrium Propagation to general dynamics, relaxing the requirement of an energy function. As a consequence of this generalization, the algorithm does not compute the true gradient of the objective function, but rather approximates it at a precision which is proven to be directly related to the degree of symmetry of the feedforward and feedback weights. We show experimentally that the intrinsic properties of the system lead to alignment of the feedforward and feedback weights, and that our algorithm optimizes the objective function. Deep learning BID18 is the de-facto standard in areas such as computer vision BID17, speech recognition and machine translation BID3. These applications deal with different types of data and share little in common at first glance. Remarkably, all these models typically rely on the same basic principle: optimization of objective functions using the backpropagation algorithm. Hence the question: does the cortex in the brain implement a mechanism similar to backpropagation, which optimizes objective functions?The backpropagation algorithm used to train neural networks requires a side network for the propagation of error derivatives, which is vastly seen as biologically implausible BID7. One hypothesis, first formulated by , is that error signals in biological networks could be encoded in the temporal derivatives of the neural activity and propagated through the network via the neuronal dynamics itself, without the need for a side network. Neural computation would correspond to both inference and error back-propagation. This work also explores this idea. The framework of Equilibrium Propagation BID29 requires the network dynamics to be derived from an energy function, enabling computation of an exact gradient of an objective function. However, in terms of biological realism, the requirement of symmetric weights between neurons arising from the energy function is not desirable. The work presented here extends this framework to general dynamics, without the need for energy functions, gradient dynamics, or symmetric connections. Our approach is the following. We start from classical models in neuroscience for the dynamics of the neuron's membrane voltage and for the synaptic plasticity (section 3). Unlike in the Hopfield model BID16, we do not assume pairs of neurons to have symmetric connections. We then describe an algorithm for supervised learning based on these models (section 4) with minimal extra assumptions. Our model is based on two phases: at prediction time, no synaptic changes occur, whereas a local update rule becomes effective when the targets are observed. The proposed update mechanism is compatible with spike-timing-dependent plasticity, which supposedly governs synaptic changes in biological neural systems. Finally, we show that the proposed algorithm has the desirable machine learning property of optimizing an objective function (section 5). We show this experimentally (Figure 3) and we provide the beginning for a theoretical explanation. Historically, models based on energy functions and/or gradient dynamics have represented a key subject of neural network research. Their mathematical properties often allow for a simplified analysis, in the sense that there often exists an elegant formula or algorithm for computing the gradient of the objective function BID0 BID24 BID29. However, we argue in this section that 1. due to the energy function, such models are very restrictive in terms of dynamics they can model -for instance the Hopfield model requires symmetric weights, 2. machine learning algorithms do not require computation of the gradient of the objective function, as shown in this work and the work of BID19.In this work, we propose a simple learning algorithm based on few assumptions. To this end, we relax the requirement of the energy function and, at the same time, we give up on computing the gradient of the objective function. We believe that, in order to make progress in biologically plausible machine learning, dynamics more general than gradient dynamics should be studied. As discussed in section 6, another motivation for studying more general dynamics is the possible implementation of machine learning algorithms, such as our model, on analog hardware: analog circuits implement differential equations, which do not generally correspond to gradient dynamics. Most dynamical systems observed in nature cannot be described by gradient dynamics. A gradient field is a very special kind of vector field, precisely because it derives from a primitive scalar function. The existence of a primitive function considerably limits the "number of degrees of freedom" of the vector field and implies important restrictions on the dynamics. In general, a vector field does not derive from a primitive function. In particular, the dynamics of the leaky integrator neuron model studied in this work (Eq. 1) is not a gradient dynamics, unless extra (biologically implausible) assumptions are made, such as exact symmetry of synaptic weights (W ij = W ji) in the case of the Hopfield model. Machine learning relies on the basic principle of optimizing objective functions. Most of the work done in deep learning has focused on optimizing objective functions by gradient descent in the weight space (thanks to backpropagation). Although it is very well known that following the gradient is not necessarily the best option -many optimization methods based on adaptive learning rates for individual parameters have been proposed such as and Adagrad BID9 -almost all proposed optimization methods rely on computing the gradient, even if they do not follow the gradient. In the field of deep learning, "computing the gradient" has almost become synonymous with "optimizing".In fact, in order to optimize a given objective function, not only following the gradient unnecessary, but one does not even need to compute the gradient of that objective function. A weaker sufficient condition is to compute a direction in the parameter space whose scalar product with the gradient is negative, without computing the gradient itself. A major step forward was achieved by BID19. One of the contributions of their work was to dispel the long-held assumption that a learning algorithm should compute the gradient of an objective function in order to be sound. Their algorithm computes a direction in the parameter space that has at first sight little to do with the gradient of the objective function. Yet, their algorithm "learns" in the sense that it optimizes the objective function. By giving up on the idea of computing the gradient of the objective function, a key aspect rendering backpropagation biologically implausible could be fixed, namely the weight transport problem. The work presented here is along the same lines. We give up on the idea of computing the gradient of the objective function, and by doing so, we get rid of the biologically implausible symmetric connections required in the Hopfield model. In this sense, the "weight transport" problem in the backpropagation algorithm appears to be similar, at a high level, to the requirement of symmetric connections in the Hopfield model. We suggest that in order to make progress in biologically plausible machine learning, it might be necessary to move away from computing the true gradients in the weight space. An important theoretical effort to be made is to understand and characterize the dynamics in the weight space that optimize objective functions. The set of such dynamics is of course much larger than the tiny subset of gradient dynamics. We denote by s i the averaged membrane voltage of neuron i across time, which is continuous-valued and plays the role of a state variable for neuron i. We also denote by ρ(s i) the firing rate of neuron i. We suppose that ρ is a deterministic function (nonlinear activation) that maps the averaged voltage s i to the firing rate ρ(s i). The synaptic strength from neuron j to neuron i is denoted by W ij. In biological neurons a classical model for the time evolution of the membrane voltage s i is the rate-based leaky integrator neuron model, in which neurons are seen as performing leaky temporal integration of their past inputs BID8: DISPLAYFORM0 Unlike energy-based models such as the Hopfield model BID16 that assume symmetric connections between neurons, in the model studied here the connections between neurons are not tied. Thus, our model is described by a directed graph, whereas the Hopfield model is best regarded as an undirected graph (Figure 1).(a) The network model studied here is best represented by a directed graph.(b) The Hopfield model is best represented by an undirected graph. Figure 1: From the point of view of biological plausibility, the symmetry of connections in the Hopfield model is a major drawback (1b). The model that we study here is, like a biological neural network, a directed graph (1a).3.2 SPIKE-TIMING DEPENDENT PLASTICITY Spike-Timing Dependent Plasticity (STDP) is considered a key mechanism of synaptic change in biological neurons BID21 BID11 BID22. STDP is often conceived of as a spike-based process which relates the change in the synaptic weight W ij to the timing difference between postsynaptic spikes (in neuron i) and presynaptic spikes (in neuron j) BID5. In fact, both experimental and computational work suggest that postsynaptic voltage, not postsynaptic spiking, is more important for driving LTP (Long Term Potentiation) and LTD (Long Term Depression) BID6 BID20.Similarly, have shown in simulations that a simplified Hebbian update rule based on pre-and post-synaptic activity can functionally reproduce STDP: DISPLAYFORM1 Throughout this paper we will refer to this update rule (Eq. 2) as "STDP-compatible weight change" and propose a machine learning justification for such an update rule. Let s = (s 1, s 2, . . .) be the global state variable and parameter W the matrix of connection weights W ij. We write µ(W, s) the vector whose components are defined as DISPLAYFORM0 defining a vector field over the neurons state space, indicating in which direction each neuron's activity changes: DISPLAYFORM1 Since ρ(s j) = ∂µi ∂Wij (W, s), the weight change Eq. 2 can also be expressed in terms of µ in the form dW ij ∝ ∂µi ∂Wij (W, s)ds i. Note that for all i = i we have ∂µ i ∂Wij = 0 since to each synapse W ij corresponds a unique post-synaptic neuron s i. Hence dW ij ∝ ∂µ ∂Wij (W, s) · ds. We rewrite the STDP-compatible weight change in the more concise form DISPLAYFORM2 The framework and the algorithm in their general forms are described in Appendix A.To illustrate our algorithm, we consider here the supervised setting in which we want to predict an output y given an input x. We describe a simple two-phase learning procedure based on the dynamics Eq. 4 and Eq. 5 for the state and the parameter variables. This algorithm is similar to the one proposed by BID29, but here we do not assume symmetric weights between neurons. Note that similar algorithms have also been proposed by; BID13 or more recently by BID23. Our contribution in this work are theoretical insights into why the proposed algorithm works. In the supervised setting studied here, the units of the network are split in two sets: the inputs x whose values are always clamped, and the dynamically evolving units h (the neurons activity, indicating the state of the network), which themselves include the hidden layers (h 1 and h 2 here) and an output layer (h 0 here), as in Figure 2. In this context the vector field µ is defined by its components µ 0, µ 1 and µ 2 on h 0, h 1 and h 2 respectively, as follows: DISPLAYFORM0 Here the scalar function ρ is applied elementwise to the components of the vectors. The neurons h follow the dynamics DISPLAYFORM1 In this section and the next we use the notation h rather than s for the state variable. The layer h 0 plays the role of the output layer where the prediction is read. The target outputs, denoted by y, have the same dimension as the output layer h 0. The discrepancy between the output units h 0 and the targets y is measured by the quadratic cost function DISPLAYFORM2 Unlike in the continuous Hopfield model, here the feed-forward and feedback weights are not tied, and in general the state dynamics Eq. 9 is not guaranteed to converge to a fixed point. However we observe experimentally that the dynamics almost always converges. We will see in section 5 that, for a whole set of values of the weight matrix W. the dynamics of the neurons h converges. Assuming this condition to hold, the dynamics of the neurons converge to a fixed point which we denote by h 0 (beware not to confuse with the notation for the output units h 0). The prediction h 0 0 is then read out on the output layer and compared to the actual target y. The objective function (for a single training case (x, y)) that we aim to minimize is the cost at the fixed point h 0, which we write DISPLAYFORM3 Note that this objective function is the same as the one proposed by BID1 BID28. Their method to optimize J is to compute the gradient of J thanks to an algorithm which they call "Recurrent Backpropagation". Other methods related to Recurrent Backpropagation exist to compute the gradient of J -in particular the "adjoint method", "implicit differentiation" and "Backprop Through Time". These methods are biologically implausible, as argued in Appendix B.Here our approach to optimize J is to give up on computing the true gradient of J and, instead, we propose a simple algorithm based only on the leaky integrator dynamics (Eq. 4) and the STDPcompatible weight change (Eq. 5). We will show in section 5 that our algorithm computes a proxy for the gradient of J. Also, note that in its general formulation, our algorithm applies to any vector field µ and cost function C (Appendix A) The idea of Equilibrium Propagation BID29 is to see the cost function C (Eq. 10) as an external potential energy for the output units h 0, which can drive them towards their target y. Following the same idea we define the "extended vector field" µ β as DISPLAYFORM0 and we redefine the dynamics of the state variable h as DISPLAYFORM1 The real-valued scalar β ≥ 0 controls whether the output h 0 is pushed towards the target y or not, and by how much. We call β the "influence parameter" or "clamping factor".The differential equation of motion Eq. 13 can be seen as a sum of two "forces" that act on the temporal derivative of the state variable h. Apart from the vector field µ that models the interactions between neurons within the network, an "external force" −β ∂C ∂h is induced by the external potential βC and acts on the output neurons: DISPLAYFORM2 DISPLAYFORM3 The form of Eq. 14 suggests that when β = 0, the output units h 0 are not sensitive to the targets y from the outside world. In this case we say that the network is in the free phase (or first phase). When β > 0, the "external force" drives the output units h 0 towards the target y. When β 0 (small positive value), we say that the network is in the weakly clamped phase (or second phase). Also, note that the case β → ∞, not studied here, would correspond to fully clamped outputs. We propose a simple two-phase learning procedure, similar to the one proposed by BID29. In the first phase of training, the inputs are set (clamped) to the input values. The state variable (all the other neurons) follows the dynamics Eq. 9 (or equivalently Eq. 13 with β = 0) and the output units are free. We call this phase the free phase, as the system relaxes freely towards the free fixed point h 0 without any external constraints on his output neurons. During this phase, the synaptic weights are unchanged. Figure 2: Input x is clamped. Neurons h include "hidden layers" h 2 and h 1, and "output layer" h 0 that corresponds to the layer where the prediction is read. Target y has the same dimension as h 0. The clamping factor β scales the "external force" −β ∂C ∂h that attracts the output h 0 towards the target y. In the second phase, the influence parameter β takes on a small positive value β 0. The state variable follows the dynamics Eq. 13 for that new value of β, and the synaptic weights follow the STDP-compatible weight change Eq. 5. This phase is referred to as the weakly clamped phase. The novel "external force" −β ∂C ∂h in the dynamics Eq. 13 acts on the output units and drives them towards their targets (Eq. 14). This force models the observation of y: it nudges the output units h 0 from their free fixed point value in the direction of their targets. Since this force only acts on the output layer h 0, the other hidden layers (h i with i > 0) are initially at equilibrium at the beginning of the weakly clamped phase. The perturbation caused at the output layer will then propagate backwards along the layers of the network, giving rise to "back-propagating" error signals. The network eventually settles to a new nearby fixed point, corresponding to the new value β 0, termed weakly clamped fixed point and denoted h β. Our model assumes that the STDP-compatible weight change (Eq. 5) occurs during the second phase of training (weakly clamped phase) when the network's state moves from the free fixed point h 0 to the weakly clamped fixed point h β. Normalizing by a factor β and letting β → 0, we get the update rule ∆W ∝ ν(W) for the weights, where ν(W) is the vector defined as DISPLAYFORM0 The vector ν(W) has the same dimension as W. Formally ν is a vector field in the weight space. It is shown in section 5 that ν(W) is a proxy to the gradient ∂J ∂W. The effectiveness of the proposed method is demonstrated through experimental studies (Figure 3). In this section, we attempt to understand why the proposed algorithm is experimentally found to optimize the objective function J (Figure 3). We say that W is a "good parameter" if:1. for any initial state for the neurons, the state dynamics dh dt = µ (W, x, h) converges to a fixed point -a condition required for the algorithm to be correctly defined, 2. the scalar product ∂J ∂W · ν(W) at the point W is negative -a desirable condition for the algorithm to optimize the objective function J.Experiments show that the dynamics of h (almost) always converges to a fixed point and that J consistently decreases (Figure 3). This means that, during training, as the parameter W follows the update rule ∆W ∝ ν(W), all values of W that the network takes are "good parameters". In this section we attempt to explain why. ∂J ∂W AND ν Theorem 1. The gradient of J can be expressed in terms of µ and C as DISPLAYFORM0 Similarly, the vector field ν (Eq. 16) is equal to DISPLAYFORM1 In these expressions, all terms are evaluated at the fixed point h 0. and that the angle between these two vectors is directly linked to the "degree of symmetry" of the Jacobian of µ.An important particular case is the setting of Equilibrium Propagation BID29, in which the vector field µ is a gradient field µ = − ∂E ∂h, meaning that it derives from an energy function E. In this case the Jacobian of µ is symmetric since it is the Hessian of E. Indeed DISPLAYFORM0. Therefore, Theorem 1 shows that ν is also a gradient field, namely the gradient of the objective function J, that is ν = − ∂J ∂W. Note that in this setting the set of "good parameters" is the entire weight space -for all W, the dynamics dh dt = − ∂E ∂h (W, h) converges to an energy minimum, and W converges to a minimum of J since ∆W ∝ − ∂J ∂W. We argue that the set of "good parameters" covers a large proportion of the weight space and that they contain the matrices W that present a form of symmetry or "alignment". In the next subsection, we discuss how this form of symmetry may arise from the learning procedure itself. Figure 3: Example system trained on the MNIST dataset, as described in Appendix C. The objective function is optimized: the training error decreases to 0.00% in around 70 epochs. The generalization error is about 2%. Right: A form of symmetry or alignment arises between feedforward and feedback weights W k,k+1 and W k+1,k in the sense that tr(W k,k+1 · W k+1,k) > 0. This architecture uses 3 hidden layers each of dimension 512. Experiments show that a form of symmetry between feedforward and feedback weights arises from the learning procedure itself (Figure 3). Although the causes for this phenomenon aren't understood very well yet, it is worth pointing out that similar observations have been made in previous work and different settings. A striking example is the following one. A major argument against the plausibility of backpropagation in feedforward nets is the weight transport problem: the signals sent forward in the network and those sent backward use the same connections. BID19 have observed that, in the backward pass, (back)propagating the error signals through fixed random feedback weights (rather than the transpose of the feedforward weights) does not harm learning. Moreover, the learned feedforward weights W k,k+1 tend to'align' with the fixed random feedback weights W k+1,k in the sense that the trace of W k,k+1 · W k+1,k is positive. Denoising autoencoders without tied weights constitute another example of learning algorithms where a form of symmetry in the weights has been observed as learning goes on BID31.The theoretical from BID2 also shows that, in a deep generative model, the transpose of the generative weights perform approximate inference. They show that the symmetric solution minimizes the autoencoder reconstruction error between two successive layers of rectifying linear units. Our approach provides a basis for implementing machine learning models in continuous-time systems, while requirements regarding the actual dynamics are reduced to a minimum. This means that the model applies to a large class of physical realizations of vectorfield dynamics, including analog electronic circuits. Implementations of recurrent networks based on analog electronics have been proposed in the past, e.g. BID13, however, these models typically required circuits and associated dynamics to adhere to an exact theoretical model. With our framework, we provide a way of implementing a learning system on a physical substrate without even knowing the exact dynamics or microscopic mechanisms that give rise to it. Thus, this approach can be used to analog electronic system end-to-end, without having to worry about exact device parameters and inaccuracies, which inevitably exist in any physical system. Instead of approximately implementing idealized computations, the actual analog circuit, with all its individual device variations, is trained to perform the task of interest. Thereby, the more direct implementation of the dynamics might in advantages in terms of speed, power, and scalability, as compared to digital approaches. Our model demonstrates that biologically plausible learning in neural networks can be achieved with relatively few assumptions. As a key contribution, in contrast to energy-based approaches such as the Hopfield model, we do not impose any symmetry constraints on the neural connections. Our algorithm assumes two phases, the difference between them being whether synaptic changes occur or not. Although this assumption begs for an explanation, neurophysiological findings suggest that phase-dependent mechanisms are involved in learning and memory consolidation in biological systems. Theta waves, for instance, generate neural oscillatory patterns that can modulate the learning rule or the computation carried out by the network BID26. Furthermore, synaptic plasticity, and neural dynamics in general, are known to be modulated by inhibitory neurons and dopamine release, depending on the presence or absence of a target signal. BID10; BID27.In its general formulation (Appendix A), the work presented in this paper is an extension of the framework of BID29 to general dynamics. This is achieved by relaxing the requirement of an energy function. This generalization comes at the cost of not being able to compute the (true) gradient of the objective function but, rather a direction in the weight space which is related to it. Thereby, precision of the approximation of the gradient is directly related to the "alignment" between feedforward and feedback weights. Even though the exact underlying mechanism is not fully understood yet, we observe experimentally that during training the weights symmetrize to some extent, as has been observed previously in a variety of other settings BID19 BID31 BID2. Our work shows that optimization of an objective function can be achieved without ever computing the (true) gradient. More thorough theoretical analysis needs to be carried out to understand and characterize the dynamics in the weight space that optimize objective functions. Naturally, the set of all such dynamics is much larger than the tiny subset of gradient-based dynamics. Our framework provides a means of implementing learning in a variety of physical substrates, whose precise dynamics might not even be known exactly, but which simply have to be in the set of sup-ported dynamics. In particular, this applies to analog electronic circuits, potentially leading to faster, more efficient, and more compact implementations. In this Appendix, we present the framework and the algorithm in their general formulations and we prove the theoretical . We consider a physical system specified by a state variable s and a parameter variable θ. The system is also influenced by an external input v, e.g. in the supervised setting v = (x, y) where y is the target that the system wants to predict given x. Let s → µ(θ, v, s) be a vector field in the state space and C(θ, v, s) a cost function. We assume that the state dynamics induced by µ converges to a stable fixed point s 0 θ,v, corresponding to the "prediction" from the model and characterized by DISPLAYFORM0 The objective function that we want to optimize is the cost at the fixed point DISPLAYFORM1 Note the distinction between J and C: the cost function is defined for any state s whereas the objective function is the cost at the fixed point. The training objective (for a single data sample v) is find arg min DISPLAYFORM2 Equivalently, the training objective can be reformulated as a constrained optimization problem: DISPLAYFORM3 where the constraint µ (θ, v, s) = 0 is the fixed point condition. All traditional methods to compute the gradient of J (adjoint method, implicit differentiation, Recurrent Backpropagation and Backpropagation Through Time or BPTT) are thought to be biologically implausible. Our approach is to give up on computing the gradient of J and let the parameter variable θ follow a vector field ν in the parameter space which is "close" to the gradient of J.Before defining ν we first introduce the "extended vector field" DISPLAYFORM4 where β is a real-valued scalar called "influence parameter". Then we extend the notion of fixed point for any value of β. For any β we define the β-fixed point s β θ,v such that DISPLAYFORM5 Under mild regularity conditions on µ and C, the implicit function theorem ensures that, for a fixed data sample v, the funtion (θ, β) → s β θ,v is differentiable. Now for every θ and v we define the vector ν(θ, v) in the parameter space as DISPLAYFORM6 As shown in section 4, the second term on the right hand side can be estimated in a biologically realistic way thanks to a two-phase training procedure. and ∂s DISPLAYFORM7 Proof of Lemma 2. First we differentiate the fixed point equation Eq. 27 with respect to θ: DISPLAYFORM8 Rearranging the terms we get Eq. 28. Similarly we differentiate the fixed point equation Eq. 27 with respect to β: DISPLAYFORM9 Rearranging the terms we get Eq. 29.Theorem 3. The gradient of the objective function is equal to DISPLAYFORM10 and the vector field ν is equal to DISPLAYFORM11 All the factors on the right-hand sides of Eq. 32 and Eq. 33 are evaluated at the fixed point s 0 θ.Proof of Theorem 3. Let us compute the gradient of the objective function with respect to θ. Using the chain rule of differentiation we get DISPLAYFORM12 Hence Eq. 32 follows from Eq. 28 evaluated at β = 0. Similarly, the expression for the vector field ν (Eq. 33) follows from its definition (Eq. 26), the identity Eq. 29 evaluated at β = 0 and the fact that We finish by stating and proving a last . Consider the setting introduced in section 4 with the quadratic cost function C = 1 2 y − h 0 2. In the weakly clamped phase, the "external influence" −β (y − h 0) added to the vector field µ (with β 0) slightly attracts the output state h 0 to the target y. It is intuitively clear that the weakly clamped fixed point is better than the free fixed point in terms of prediction error. Proposition 5 below generalizes this property to any vector field µ and any cost function C.Proposition 4. Let s 0 be a stable fixed point of the vector field s → µ(s), in the sense that s − s 0 · µ (s) < 0 for s in the neighborhood of s 0 (i.e. the vector field at s points towards s 0). Then the Jacobian of µ at the fixed point ∂µ ∂s s 0 is negative, in the sense that DISPLAYFORM13 Proof. Let v be a vector in the state space, α > 0 a positive scalar and let s:= s 0 + αv. For α small enough, the vector s is in the region of stability of s 0. Using a first order Taylor expansion and the fixed point condition µ s 0 = 0 we get DISPLAYFORM14 as α → 0. Hence the . The following proposition shows that, unless the free fixed point s 0 θ,v is already optimal in terms of cost value, for β > 0 small enough, the nudged fixed point s β θ,v achieves lower cost value than the free fixed point. Thus, a small perturbation due to a small increment of β nudges the network towards a configuration that reduces the cost value. The inequality holds because B ADJOINT METHOD AND RELATED ALGORITHMS Earlier work have proposed various methods to compute the gradient of the objective function J (Eq. 20). One common method is the "adjoint method". In the context of fixed point recurrent neural networks studied here, the adjoint method leads to Backpropagation Through Time (BPTT) and "Recurrent Backpropagation" BID1 BID28. BPTT is the method of choice today for deep learning but its biological implausibility is obvious -it requires the network to store all its past states for the propagation of error derivatives in the second phase. Recurrent Backpropagation corresponds to a special case of BPTT where the network is initialized at the fixed point. This algorithm does not need to store the past states of the network (the state at the fixed point suffices) but it still requires neurons to send a different kind of signals through a different computational path in the second phase, which seems therefore less biologically plausible than our algorithm. Our approach is to give up on the idea of computing the gradient of the objective function. Instead we show that the STDP-compatible weight change computes a proxy to the gradient in a more biologically plausible way. For completeness, we state and prove a continuous-time version of Backpropagation Through Time and Recurrent Backpropagation. The formulas for the propagation of error derivatives (Theorem 6 and Corollary 7) will make it obvious that our algorithm is more biologically plausible than both of these algorithms. To keep notations simple, we omit to write dependences in the data sample v. We consider the dynamics ds dt = µ(θ, s) and denote by s t the state of the system at time t ≥ 0 when it starts from an initial state s 0 at time t = 0. Note that s t converges to the fixed point s 0 θ as t → ∞. We then define a family of objective functions L(θ, s 0, T):= C (θ, s T) for every couple (s 0, T) of initial state s 0 and duration T ≥ 0. This is the cost of the state at time t = T when the network starts from the state s 0 at time t = 0. Note that L(θ, s 0, T) tends to J(θ) as T → ∞ (Eq. 20).We want to compute the gradient ∂L ∂θ (θ, s 0, T) as T → ∞. For that purpose, we fix T to a large value and we consider the following quantity ∂L ∂s T −t:= ∂L ∂s (θ, s T −t, t),which represents the "partial derivative of the cost with respect to the state at time T − t". In other words this is the "partial derivative of the cost with respect to the (T − t)-th hidden layer" if one regards the network as unfolded in time (though time is continuous here). The formulas in Theorem 6 below correspond to a continuous-time version of BPTT for the propagation of the partial derivatives Computing ∂L ∂s T −t h i cannot reach values above 1. As a consequence h i must remain in the domain 0 ≤ h i ≤ 1. Therefore, rather than the standard gradient descent (Eq. 58), we will use a slightly different update rule for the state variable h: DISPLAYFORM0 This little implementation detail turns out to be very important: if the i-th hidden unit was in some state h i < 0, then Eq. 58 would give the update rule h i ← (1 −)h i, which would imply again h i < 0 at the next time step (assuming < 1). As a consequence h i would remain in the negative range forever. We use different learning rates for the different layers in our experiments. We do not have a clear explanation for why this improves performance, but we believe that this is due to the finite precision with which we approach the fixed points. The hyperparameters chosen for each model are shown in Table 1 and the are shown in Figure 3. We initialize the weights according to the Glorot-Bengio initialization BID12. Table 1: Hyperparameters. for both the 2 and 3 layered MNIST. Example system trained on the MNIST dataset, as described in Appendix C. The objective function is optimized: the training error decreases to 0.00%. The generalization error lies between 2% and 3% depending on the architecture. The learning rate is used for iterative inference (Eq. 59). β is the value of the clamping factor in the second phase. α k is the learning rate for updating the parameters in layer k. We were also able to train on MNIST using a Convolutional Neural Network (CNN). We got around 2% generalization error. The hyperparameters chosen to train this Convolutional Neural Network are shown in TAB0.

We describe a biologically plausible learning algorithm for fixed point recurrent networks without tied weights

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A theoretical analysis of a new class of RNNs, trained on neuroscience tasks, allows us to identify the role of dynamical dimensionality and cell classes in neural computations.