Question: represent the article for finding a single-sentence tldr

Question: We revisit the Recurrent Attention Model , a recurrent neural network for visual attention, from an active information sampling perspective. We borrow ideas from neuroscience research on the role of active information sampling in the context of visual attention and gaze , where the author suggested three types of motives for active information sampling strategies. We find the original RAM model only implements one of them. We identify three key weakness of the original RAM and provide a simple solution by adding two extra terms on the objective function. The modified RAM 1) achieves faster convergence, 2) allows dynamic decision making per sample without loss of accuracy, and 3) generalizes much better on longer sequence of glimpses which is not trained for, compared with the original RAM. We revisit the Recurrent Attention Model (RAM,), a recurrent neural network for visual attention, from an active information sampling perspective. The RAM, instead of processing the input image for classification in full, only takes a glimpse at a small patch of the image at a time. The recurrent attention mechanism learns where to look at to obtain new information based on the internal state of the network. After a pre-defined number of glimpses, RAM finally makes a prediction as output. Compared with the attention mechanism which now dominates the AI/NLP research such as Transformer and BERT , this recurrent attention mechanism is fundamentally different, as it is used to obtain new information (active sampling of information), rather than processing information that is already fully observed. In this paper, we identify three weaknesses of this widely-cited approach. First, the convergence of RAM training is slow. Second, RAM does not support dynamic number of glimpses per sample, but uses a fixed number of glimpses for every sample. Third and perhaps most importantly, the performance of the original RAM does not improve but rather decrease dramatically if it takes more glimpses, which is weird and against intuition. We provide a simple solution of adding two extra terms in the objective function of RAM, insipred from neuroscience research which discusses the logic and neural substrates of information sampling policies in the context of visual attention and gaze. Base on the evidence available so far, suggested three kinds of motives for the active sampling strategies of decision-making while the original RAM only implements one of them. We incorporate the other two motives in the objective function, and by doing so we 1) achieve much faster convergence and 2) instantly enbale decision making with a dynamic number of glimpse for different samples with no loss of accuracy. 3) More importantly, we find that the modified RAM generalizes much better to longer sequence of glimpses which is not trained for. Recurrent Attention Model. RAM combines a recurrent neural network with reinforcement learning to utilize visual attention for classification and is trained with policy gradient in a end-to-end manner. At each timestep, it takes a glimpse on a patch of the input image, processes the information and then uses its internal state to select the next location to focus. By adaptively selecting a sequence of patches of images and only processing the selected patches, it outperforms a convolutional neural network baseline while its computation cost is cheap and independent of the input image size. Due to page limit, we refer readers to the original paper, online tutorials and high-starred implementation on Github. The recurrent attention mechanism allows RAM to actively obtain new information for decision making. Note that though RAM is trained end-to-end through the help of the REINFORCE rule , it is not a standard reinforcement learning problem where the objective is to maximize the cumulative reward (or, minimize the cumulative regret); instead, RAM maximizes the simple reward (minimizes the simple regret). The reward is given only to the final prediction action, but not to the glimpse actions. In a sense, a glimpse action is rewarded indirectly only if it obtains useful information that helps making a correct prediction. This is very similar to the problem of pure-exploration bandits , specifically best arm identification. We think this distinguishing feature of active sampling of information for decision making is not well addressed in deep learning, in reinforcement learning, as well as in neuroscience research . Weaknesses of Recurrent Attention Model. First, the original RAM paper did not mention the exact convergence of RAM training. We inspect some Github implementations of RAM, one implementation (conan7882, 2018) takes 1000 epochs to achieve the paper accuracy; another (kevinzakka, 2017) claims to reach the paper accuracy in 30 epochs, but we find that it uses a technique to augment the test time accuracy, introduced in the followed work of RAM, by processing an input multiple times and averaging the prediction. When we disable this technique in (kevinzakka, 2017), we find that the convergence is in fact much slower. Second, RAM uses a fixed number of glimpses for each input sample, however we would like to see RAM terminates automatically as long as it is very certain about the prediction. It is possible that some samples are easier to predict and only need fewer glimpses. In the future work section of, the authors briefly mentioned that they trained a separate network to predict when to terminate. By this way RAM learns to do dynamic prediction "once it has enough information to make a confident classification". Third, RAM does not generalize well when using a larger number of glimpses. This generalization issue comes in two different settings. • When the number of glimpses is the same for training and testing, which is the experiment setting in the original RAM paper. The authors showed that RAM performs the best in MNIST trained with N = 6 glimpses per sample and more glimpses begin to weaken the accuracy. • When the number of glimpses is not the same for training and testing. We set the number of glimpses to N = 6 to train the RAM and test the performance on N = 6. We find that the accuracy does not improve but decrease dramatically (see Figure 2). In this paper we evaluate the second setting. Implications from . In , the authors reviewed recent studies on attentional learning, and highlighted the role of active sampling through visual attention and gaze for decision making. suggested three types of motives for implementing active sampling policies, " One motive relates simply to the extent to which information is expected to increase the operant rewards of a task; a second motivation is related to reducing the uncertainty of belief states; and yet a third motive may be related to the intrinsic utility or dis-utility of anticipating a positive or a negative outcome (savoring or dread). " In the context of RAM for classification, only the first motive is implemented by RAM as the reward of correct classification. We propose to add two extra terms which represents the second and third motives mentioned above, on the original objective function of RAM. We denote the MNIST dataset of in total M sample-label pairs as {X, Y} = {x, y} M; the number of class in Y is K = 10. the parameter of the RAM architecture is denoted as θ, the output K-dimension softmax vector of RAM as f θ (x), and the prediction of classification aŝ y = argmax i f θ (x) i. The original objective function. Under the distribution of all possible sequences s 1:N of actions (at length N), RAM maximizes the reward of correct classification as the objective function: J original (θ) = E p(s 1:N ;θ) [1ŷ =y] where p(s 1:N ; θ) depends on the policy. employed two tricks to maximize J original (θ) which is non-trival. First, borrowed the REINFORCE rule to maximize J original (θ) through policy gradient. Second, because the policy gradient may have a high variance, trained an auxiliary network (one fully-connected layer) called baseline network taking the internal state as input to mimic the value function, which helps the training. So the true objective function the orignal RAM maximizes is J original + J auxiliary. New objective function. We consider the new objective function to be We use J uncertainty to model the motive to reduce uncertainty. However, since the uncertainty is not available, we use a similar network like the baseline network to mimic the residual of error. In other words, we use another auxiliary network to predict the different between the classification and the ground truth, taking the internal state of RAM and the current prediction f θ (x) as input. We call this predicted diffrence as self-uncertainty, denoted as u(x), which is also a K-dimensional vector. Since we wish to minimize this uncertainty, we have J uncertainty = −λ 1 i |u(x) i | We use J intrinsic to model the intrinsic utility of anticipating outcome. The intrinsic utility of anticipating outcome is here interpreted as the intrinsic preference over certain prediction outcomes. By intrinsic, it means this preference is inherently built-in regardless of the hidden state or the gathered information and should make RAM prefer to predict a certain class if there is no enough information. As MNIST is class-balanced and no class is more important than the others, we choose to enforce this preference equally to all the classes, i.e. all the possible prediction outcomes. Technically, the intrinsic preference should encourage the output softmax vector f θ (x) to be close to a prior distribution, which is simply uniform. We incorporate this intrinsic belief as the cross entropy to a uniform distribution on the output K-dimension vector f θ (x): This should regularize RAM to not make any over-confident prediction if there is no enougth information. This should be helpful, especially in the early stage of training, when the current active information sampling policy is not so good so that no useful information is gathered. Note that for simplicity, we merge the objective of the self-uncertainty network and the objective of the baseline network into J auxiliary. The entire modified RAM could be trained end-to-end just as the orignal one. How to do dynamic decision-making in test-time. The introduction of self-uncertainty has a side-effect. It gives us an opportunity to enable a dynamic number of glimpses for diffrence samples because now we have an uncertainty measure. Borrowing ideas from pure-exploration bandits, we use self-uncertainty to construct a upper and lower'bound' for each class, and let RAM terminate if the lower bound of the highest class is higher than the upper bound of the rest classes. Given a timestep t, denote the prediction as f t θ (x) and the self-uncertainty as u t (x), the upper and lower bound for a class i is where the exploration rate β is a hyperparameter. We take i * to be the class of the highest i * = argmax i f t θ (x) i. RAM terminates when the following condition is met: Lower(i *) > max i =i * Upper(i). Given a larger β, RAM will take more glimpses and when β = 0, RAM will terminate in only one glimpses. We evaluate on MNIST dataset as in the orignal RAM paper. We set the train-time number of glimpses N = 6 for it achieves the best test-time accuracy in. Implementation details see the source code 1. We first show in Figure 1 that the two new terms in the objective both contribute to a faster convergence. We test four cases 1) the orignal objective, 2) add the J intrinsic, 3) add J uncertainty, 4) add both new terms. We see in Figure 1 that both of our new objective in isolation help a faster learning and together give the fastest convergence. As in Figure 2, we test the trained models with varying number of glimpses. (We want to emphasize that the focus is not the absolute performance, but rather the generalization on more glimpses than train time.) We fisrt evaluate the non-dynamic case (fixed number for all samples). The performance of the original RAM decrease dramatically when N > 10. Adding both terms, the modified RAM does not suffer the decrease anymore even when N is large. Also, it is interesting that adding only the uncertainty term, we observe the improvement is very slight and the intrinsic term effectively stablizes the prediction accuracy given more glimpses. We also test the dynamic case by varying the exploration rate. We see that dynamic number of glimpses does not hurt the performance very much, which confirms with the hypothesis that some samples are easier to discriminate and thus need fewer glimpses. One may argue that the given longer training time or other hyperparameter tuning, RAM will eventually reach a point where it can give stable prediction accuracy on more glimpses, and the new objective only make it converge faster to that point. But during our experiments, we find with λ 2 = 0.1 the J intrinsic term can effectively stablize the prediction given more glimpses, even when trained for only 1 epoch. We observe that the l2-norm of internal states of orginal RAM becomes very large given a longer sequence of glimpses while the modified RAM with J intrinsic remains stable.

Represent the short summary for finding the corresponding scientific article (long).

Inspired by neuroscience research, solve three key weakness of the widely-cited recurrent attention model by simply adding two terms on the objective function.

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.


Question: Represent the article for finding a single-sentence TLDR

Question: Graph Neural Networks (GNNs) for prediction tasks like node classification or edge prediction have received increasing attention in recent machine learning from graphically structured data. However, a large quantity of labeled graphs is difficult to obtain, which significantly limit the true success of GNNs. Although active learning has been widely studied for addressing label-sparse issues with other data types like text, images, etc., how to make it effective over graphs is an open question for research. In this paper, we present the investigation on active learning with GNNs for node classification tasks. Specifically, we propose a new method, which uses node feature propagation followed by K-Medoids clustering of the nodes for instance selection in active learning. With a theoretical bound analysis we justify the design choice of our approach. In our experiments on four benchmark dataset, the proposed method outperforms other representative baseline methods consistently and significantly. Graph Neural Networks (GNN) (; Veličković et al., 2017; ;) have been widely applied in many supervised and semi-supervised learning scenarios such as node classifications, edge predictions and graph classifications over the past few years. Though GNN frameworks are effective at fusing both the feature representations of nodes and the connectivity information, people are longing for enhancing the learning efficiency of such frameworks using limited annotated nodes. This property is in constant need as the budget for labeling is usually far less than the total number of nodes. For example, in biological problems where a graph represents the chemical structure of a certain drug assembled through atoms, it is not easy to obtain a detailed analysis of the function for each atom since getting expert labeling advice is very expensive. On the other hand, people can carefully design a small "seeding pool" so that by selecting "representative" nodes or atoms as the training set, a GNN can be trained to get an automatic estimation of the functions for all the remaining unlabeled ones. Active Learning (AL) (; Bodó et al., 2011), following this lead, provides solutions that select "informative" examples as the initial training set. While people have proposed various methods for active learning on graphs (; ; ;), active learning for GNN has received relatively few attention in this area. and are two major works that study active learning for GNN. The two papers both use three kinds of metrics to evaluate the training samples, namely uncertainty, information density, and graph centrality. The first two metrics make use of the GNN representations learnt using both node features and the graph; while they might be reasonable with a good (well-trained) GNN model, the metrics are not informative when the label budget is limited and/or the network weights are under-trained so that the learned representation is not good. On the other hand, graph centrality ignores the node features and might not get the real informative nodes. Further, methods proposed in; only combine the scores using simple linear weighted-sum, which do not solve these problems principally. We propose a method specifically designed for GNN that naturally avoids the problems of methods above 1. Our method select the nodes based on node features propagated through the graph structure, 1 Our code will be released upon acceptance. making it less sensitive to inaccuracies of representation learnt by under-trained models. Then we cluster the nodes using K-Medoids clustering; K-Medoids is similar to the conventional K-Means, but constrains the centers to be real nodes in the graph. Theoretical and practical experiments prove the strength of our algorithm. • We perform a theoretical analysis for our method and study the relation between its classification loss and the geometry of the propagated node features. • We show the advantage of our method over Coreset by comparing the bounds. We also conjecture that similar bounds are not achievable if we use raw unpropagated node features. • We compare our method with several AL methods and obtain the best performance over all benchmark datasets. Active Learning (AL) aims at interactively choosing data points from the training pool to maximize model performances, and has been widely studied both in theory and practice . proposes to compute a Coreset over the last-layer activation of a convolutional neural network. The method is designed for general-purpose neural networks, and does not take the graph structure into account. Early works on AL with graph-structured data study non-parametric classification models with graph regularization. More recent works analyze active sampling under the graph signal processing framework . However, most of these works have focused on the denoising setting where the signal is smooth over the graphs and labels are noisy versions of node features. Similarly, optimal experimental design can also apply to graph data but primarily deals with linear regression problems, instead of nonlinear classification with discrete labels. Graph Neural Networks (GNNs) (; Veličković et al., 2017;) are the emerging frameworks in the recent years when people try to model graph-structured data. Most of the GNN variants follow a multi-layer paradigm. In each layer, the network performs a message passing scheme, so that the feature representation of a node in the next layer could be some neighborhood aggregation from its previous layer. The final feature of a single node thus comprises of the information from a multi-hop neighborhood, and is usually universal and "informative" to be used for multiple tasks. Recent works show the effectiveness of using GNNs in the AL setting. , for instance, proposes to linearly combine uncertainty, graph centrality and information density scores and obtains the optimal performance. further improves the by using learnable combination of weights with multi-armed bandit techniques. Instead of combining different metrics, in this paper, we approach the problem by clustering propagated node features. We show that our one-step active design outperforms existing methods based on learnt network represenations, in the small label setting, while not degrading in performance for larger amounts of labeled data. In this section, we describe a formal definition for the problem of graph-based active learning under the node classification setting and introduce a uniform set of notations for the rest of the paper. We are given a large graph G = (V, E), where each node v ∈ V is associated with a feature vector x v ∈ X ⊆ R d, and a label y v ∈ Y = {1, 2, ..., C}. Let V = {1, 2, ..., n}, we denote the input features as a matrix X ∈ R n×d, where each row represents a node, and the labels as a vector Y = (y 1, ..., y n). We also consider a loss function l(M|G, X, Y) that computes the loss over the inputs (G, X, Y) for a model M that maps G, X to a prediction vectorŶ ∈ Y n. Following previous works on GNN , we consider the inductive learning setting; i.e., a small part of Y is revealed to the algorithm, and we wish to minimize the loss on the whole graph l(M|G, X, Y). Specifically, an active learning algorithm A is initially given the graph G and feature matrix X. In step t of operation, it selects a subset s t ⊆ [n] = {1, 2, ..., n}, and obtains y i for every i ∈ s t. We assume y i is drawn randomly according to a distribution P y|xi supported on Y; we use η c (v) = Pr[y = c|v] to denote the probability that y = c given node v, and T. Then A uses G, X and y i for i ∈ s 0 ∪ s 1 ∪ · · · ∪ s t as the training set to train a model, using training algorithm M. The trained model is denoted as M At. If M is the same for all active learning strategies, we can slightly abuse the notation A t = M At to emphasize the focus of active learning algorithms. A general goal of active learning is then to minimize the loss under a given budget b: min where the randomness is over the random choices of Y and A. We focus on M being the Graph Neural Networks and their variants elaborated in detail in the following part. Graph Neural Networks define a multi-layer feature propagation process similar to Multi-Layer Perceptrons (MLPs). Denote the k-th layer representation matrix of all nodes as X (k), and X ∈ R n×d are the input node features. Graph Neural Networks (GNNs) differ in their ways of defining the recursive function f for the next-layer representation: where Θ k is the parameter for the k-th layer. Naturally, the input X satisfies X = X by definition. Graph Convolution Network (GCN). A GCN has a specific form of the function f as: where ReLU is the element-wise rectified-linear unit activation function , Θ k is the parameter matrix used for transforming the size of feature representations to a different dimension and S is the normalized adjacency matrix. Specifically, S is defined as: where A is the original adjacency matrix associated with graph G and D is the diagonal degree matrix of A. Intuitively, this operation updates node embeddings by the aggregation of their neighbors. The added identity matrix I (equivalent to adding self-loops to G) acts in a similar spirit to the residual links in MLPs that bypasses shallow-layer representations to deep layers. By applying this operation in a multi-layer fashion, a GCN encourages nodes that are locally related to share similar deep-layer embeddings and prediction thereafter. For the classification task, it is normal to stack a linear transformation along with a softmax function to the representation in the final layer, so that each class could have a prediction score. That is, where softmax(x) = exp(x)/ C c=1 exp(x c) which makes the prediction scores have unit sum of 1 for all classes, and K is the total number of layers. We use the GCN structure as the fixed unified model M for all the following discussed AL strategies A. Traditionally, active learning algorithms choose one instance at a time for labeling, i.e., with |s t | = 1. However, for modern datasets where the numbers of training instances are very large, it would be extremely costly if we re-train the entire system each time when a new label is obtained. Hence we focus on the "batched" one-step active learning setting , and select the informative nodes once and for all when the algorithm starts. This is also called the optimal experimental design in the literature . Aiming to select the b most representative nodes as the batch, our target becomes: min The node selection algorithm is described in Section 4.1, followed by the loss bound analysis in Section 4.2, and the comparison with a closely related algorithm (K-Center in Coreset ) in Section 4.3. Input: Node representation matrix X, graph structure matrix G and budget We describe a generic active learning framework using distance-based clustering in Algorithm 1. It acts in two major steps: 1) computing a distance matrix or function d X,G using the node feature representations X and the graph structure G; 2) applying clustering with b centers over this distance matrix, and from each cluster select the node closest to the center of the cluster. After receiving the labels (given by matrix Y) of the selected nodes, we train a graph neural network, specifically GCN, based on X, G and Y for the node classification task. Generally speaking, different options for the two steps above would yield different performance in the down-stream prediction tasks; we detail and justify our choices below and in subsequent sections. Distance Function. Previous methods (; ;) commonly use network representations to compute the distance, i.e., for some specific k. While this can be helpful in a well-trained network, the representations are quite inaccurate in initial stages of training and such distance function might not select the representatitive nodes. Differently, we define the pairwise node distance using the L 2 norm of the difference between the corresponding propagated node features: where (M) i denotes the i-th row of matrix M, and recall that K is the total number of layers. Intuitively, this removes the effect of untrained parameters on the distance, while still taking the graph structure into account. Clustering Method. Two commonly used methods are K-Means and K-Center 2. We propose to apply the K-Medoids clustering. K-Medoids problem is similar to K-Means, but the center it selects must be real sample nodes from the dataset. This is critical for active learning, since we cannot try to label the unreal cluster centers produced by K-Means. Also, we show in Section 4.3 that K-Medoids can obtain a more favorable loss bound than K-Center. We call our method FeatProp, to emphasize the active learning strategy via node feature propagation over the input graph, which is the major difference from other node selection methods. Recall that we use (S K X) i − (S K X) j 2 to approximate the pairwise distances between the hidden representations of nodes in GCN. Intuitively, representation S K X resembles the output of a simplified GCN by dropping all activation functions and layer-related parameters in the original structure, which introduces a strong inductive bias. In other words, the selected nodes could possibly contribute to the stabilization of model parameters during the training phase of GCN. The following theorem formally shows that using K-Medoids with propagated features can lead to a low classification loss: Theorem 1 (informal). Suppose that the label vector Y is sampled independently from the distribution y i ∼ η(i), and the loss function l is bounded by [−L, L]. Then under mild assumptions, there exists a constant c 0 such that with probability 1 − δ the expected classification loss of A t satisfies To understand Theorem 1, notice that the first term the target loss of K-Medoids (sum of point-center distances), and the second term quickly decays with n, where n is the total number of nodes in graph G. Therefore the classification loss of A 0 on the entire graph G is mostly dependent on the K-Medoids loss. In practice, we can utilize existing robust initialization algorithms such as Partitioning Around Medoids (PAM) to approximate the optimal solution for K-Medoids clustering. The assumptions we made in Theorem 1 are pretty standard in the literature, and we illustrate the details in the appendix. While our share some common characteristics with Sener et al. , our proof is more involved in the sense that it relates to the translated features (S K X) i − (S K X) j 2 instead of the raw features (X) i − (X) j 2. In fact, we conjecture that using raw feature clustering selection for GCN will not in a similar bound as in: this is because GCN uses the matrix S to diffuse the raw features across all nodes in V, and the final predictions of node i will also depend on its neighbors as well as the raw feature (X) i. We could see a clearer comparison in practice in Section 5.2. Figure 1: Visualization of Theorem 1. Consider the set of selected points s and the remaining points in the dataset [n]\s. K-Medoids corresponds to the mean of all red segments in the figure, whereas K-Center corresponds to the max of all red segments in the figure. In this subsection we provide justifications on using the K-Medoids clustering method as opposed to Coreset . The Coreset approach aims to find a δ-cover of the training set. In the context of using propagated features, this means solving We can show a similar theorem as Theorem 1 for the Coreset approach: Theorem 2. Under the same assumptions as in Theorem 1, with probability 1 − δ the expected classification loss of A t satisfies It is easy to see that RHS of Eqn. is smaller than RHS of Eqn., since In other words, K-Medoids can obtain a better bound than the K-Center method (see Figure 1 for a graphical illustration). We observe superior performance of K-Medoid clustering over K-Center clustering in our experiments as well (see Section 5.2). We evaluate the node classification performance of our selection method on the Cora, Citeseer, and PubMed network datasets . We further supplement our experiment with an even denser network dataset CoraFull (Bojchevski & Günnemann, 2017) to illustrate the performance differences of the comparing approaches on a large-scale setting. We evaluate the Macro-F1 of the methods over the full set of nodes. The sizes of the budgets are fixed for all benchmark datasets. Specifically, we choose to select 10, 20, 40, 80 and 160 nodes as the budget sizes. After selecting the nodes, a two-layer GCN 3, with 16 hidden neurons, is trained as the prediction model. We use the Adam optimizer with a learning rate of 0.01 and weight decay of 5 × 10 −4. All the other hyperparameters are kept as in the default setting (β 1 = 0.9, β 2 = 0.999). To guarantee the convergence of the GCN, the model trained after 200 epochs is used to evaluate the metric on the whole set. We compared the following methods: • Random: Choosing the nodes uniformly from the whole vertex set. • Degree: Choosing the nodes with the largest degrees. Note that this method does not consider the information of node features. • Uncertainty: Similar to the methods in , we put the nodes with maxentropy into the pool of instances. • Coreset : This method performs a K-Center clustering over the last hidden representations in the network. If time allows (on Cora and Citeseer), a robust mixture integer programming method as in (dubbed CoresetMIP) is adopted. We also apply a time-efficient approximation version (Coreset-greedy) for all of the datasets. The center nodes are then selected into the pool. • AGE : This method linearly combines three metrics -graph centrality, information density, and uncertainty and select nodes with the highest scores. • ANRMAB : This method enhances AGE by learning the combination weights of metrics through an exponential multi-arm-bandit updating rule. • FeatProp: This is our method. We perform a K-Medoids clustering to the propogated features (Eqn.), where X is the input node features. In the experiment, we adopts an efficient approximated K-Medoids algorithm which performs K-Means until convergence and select nodes cloesest to centers into the pool. In our experiments, we start with a small set of nodes (5 nodes) sampled uniformly at random from the dataset as the initial pool. We run all experiments with 5 different random seeds and report the averaged classification accuracy as the metric. We plot the accuracy vs the number of labeled points. For approaches (Uncertainty, Coreset, AGE and ANRMAB) that require the current status/hidden representations from the classification model, a fully-trained model built from the previous budget pool is returned. For example, if the current budget is 40, the model trained from 20 examples selected by the same AL method is used. Figure 2, our method outperforms all the other baseline methods in most of the compared settings. It is noticeable that AGE and ANRMAB which use uncertainty score as their sub-component can achieve better performances than Uncertainty and are the second best methods in most of the cases. We also show an averaged Macro-F1 with standard deviation across different number of labeled nodes in Table 3. It is interesting to find that our method has the second smallest standard deviation (Degree is deterministic in terms of node selection and the variance only comes from the training process) among all methods. We conjecture that this is due to the fact that other methods building upon uncertainty may suffer from highly variant model parameters at the beginning phase with very limited labeled nodes. Efficiency. We also compare the time expenses between our method and Coreset, which also involves a clustering sub-routine (K-Center), in Table 2. It is noticeable that in order to make Coreset more stable, CoresetMIP uses an extreme excess of time comparing to Coreset-greedy in the same setting. An interesting fact we could observe in Figure 2 is that CoresetMIP and Coreset-greedy do not have too much performance difference on Citeseer, and Coreset-greedy is even better than CoresetMIP on Cora. This is quite different from the in image classification tasks with CNNs . This phenomenon distinguishes the difference between graph node classification with traditional classification problems. We conjecture that this is partially due to the fact that the nodes no longer preserve independent embeddings after the GCN structure, which makes the original analysis of Coreset not applicable. Ablation study. It is crucial to select the proper distance function and clustering subroutine for FeatProp (Line 1 and Line 2 in Algorithm 1). As is discussed in Section 4.3, we test the differences with the variant of using the L2 distance from the final layer of GCN as the distance function and the one by setting K-Medoids choice with a K-Center replacement. We compare these algorithms in Figure 3. As is demonstrated in the figure, the K-Center version (blue line) has a lower accuracy than the original FeatProp approach. This observation is compatible with our analysis in Section 4.3 as K-Medoids comes with a tighter bound than K-Center in terms of the classification loss. Furthermore, as final layer representations are very sensitive to the small budget case, we observe that the network representation version (orange line) also generally shows a much deteriorated performance at the beginning stage. Though FeatProp is tailored for GCNs, we could also test the effectiveness of our algorithm over other GNN frameworks. Specifically, we compare the methods over a Simplified Graph Convolution (SGC) and obtain similar observations. Due to the space limit, we put the detailed in the appendix. We study the active learning problem in the node classification task for Graph Convolution Networks (GCNs). We propose a propagated node feature selection approach (FeatProp) to comply with the specific structure of GCNs and give a theoretical characterizing the relation between its classification loss and the geometry of the propagated node features. Our empirical experiments also show that FeatProp outperforms the state-of-the-art AL methods consistently on most benchmark datasets. Note that FeatProp only focuses on sampling representative points in a meaningful (graph) representation, while uncertainty-based methods select the active nodes from a different criterion guided by labels, how to combine that category of methods with FeatProp in a principled way remains an open and yet interesting problem for us to explore. We also evaluate the methods using the metric of Micro-F1 in Table 4 C be the prediction for node i under input G, X, and (M) i,c be the c-th element of (M) i (i.e., the prediction for class c). In order to show Theorem 1, we make the following assumptions: Assumption 1. We assume A 0 overfits to the training data. Specifically, we assume the following two conditions: i) A 0 has zero training loss on s 0; ii) for any unlabeled data (x i, x j) with i ∈ s 0 and j ∈ s 0, we have (A 0) i,yj ≤ (A 0) j,yj and (A 0) i,c ≥ (A 0) j,c for all c = y j. The second condition states that A 0 achieves a high confidence on trained samples and low confidence on unseen samples. We also assume that the class probabilities are given by a ground truth GCN; i.e., there exists a GCN M * that predicts Pr[Y i = c] on the entire training set. This is a common assumption in the literature, and shows that gradient descent provably achieves zero training loss and a precise prediction in polynomial time. Assumption 2. We assume l is Lipschitz with constant λ and bounded in [−L, L]. The loss function is naturally Lipschitz for many common loss functions such as hinge loss, mean squared error, and cross-entropy if the model output is bounded. This assumption is widely used in DL theory (e.g., ;). Assumption 3. We assume that there exists a constant α such that the sum of input weights of every neuron is less than α. Namely, we assume i |(Θ K) i,j | ≤ α. This assumption is also present in . We note that one can make i |(Θ K) i,j | arbitrarily small without changing the network prediction; this is because dividing all input weights by a constant t will also divide the output by a constant t. Assumption 4. We assume that ReLU function activates with probability 1/2. This is a common assumption in analyzing the loss surface of neural networks, and is also used in (; ;). This assumption also aligns with observations in practice that usually half of all the ReLU neurons can activate. With these assumptions in place, we are able to prove Theorem 1. Theorem 1 (restated). Suppose Assumptions 1-4 hold, and the label vector Y is sampled independently from the distribution y v ∼ η(v) for every v ∈ V. Then with probability 1 − δ the expected classification loss of A t satisfies Proof. Fix y j for j ∈ s 0 and therefore the ing model A 0. Let i ∈ V \ s 0 be any node and j ∈ s 0. We have For the first term we have The last inequality holds from the Lipschitz continuity of l. Now from Assumption 1, we have otherwise. Now taking the expection w.r.t the randomness in ReLU we have Here E σ represents taking the expectation w.r.t ReLU. Now for we have The inequality follows from. Now for the second loss in we use the property that M * computes the ground truth: We now use the fact that ReLU activates with probability 1/2, and compute the expectation: Here E σ means that we compute the expectation w.r.t randomness in σ (ReLU) in M *. The last inequality follows from definition of α, and that l ∈ [−L, L]. Combining the two parts to and let j = argmin (S K X) i − (S K X) j, we obtain Consider the following process: we first get G, X (fixed data) as input, which induces η(i) for i ∈ [n]. Note that M * gives the ground truth η(i) for every i so distributions η(i) ≡ η X,G (i) are fixed once we obtain G, X 4. Then the algorithm A choose the set s 0 to label. After that, we randomly sample y j ∼ η(j) for j ∈ s 0 and use the labels to train model A 0. At last, we randomly sample y i ∼ η(i) and obtain loss l(A 0 |G, X, Y). Note that the sampling of all y i for i ∈ V \ s 0 is after we fix the model A 0, and knowing exact values of y j for j ∈ s 0 does not give any information of y i (since η(i) is only determined by G, X). Now we use Hoeffding's inequality (Theorem 3) with Z i = l((A 0) i, y i ); we have −L ≤ Z i ≤ L by our assumption, and recall that |V \ s 0 | = n − b. Let δ be the RHS of, we have that with probability 1 − δ, Now plug in, multiply both sides by (n − b) and rearrange. We obtain that i∈V \s 0 Now note that since the random draws of y i is completely irrelevant with training of A 0, we can also sample y i together with y j for j ∈ s 0 after receiving G, X and before the training of A 0 (A does not have access to the labels anyway). So holds for the random drawings of all y's. Now divide both sides of by n and use, we have The same proof as Theorem 1 applies for Theorem 2 using the max of distances instead of averaging. We therefore omit the details here. We attach the Hoeffding's inequality here for the completeness of our paper. 4 To make a rigorous argument, we get the activation of M * in this step, meaning that we pass through the randomness of σ in M *.

This paper introduces a clustering-based active learning algorithm on graphs.

Represent the TLDR for finding the corresponding scientific article!

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation. However, conditioning CNFs on signals of interest for conditional image generation and downstream predictive tasks is inefficient due to the high-dimensional latent code generated by the model, which needs to be of the same size as the input data. In this paper, we propose InfoCNF, an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information. Since the partitioning strategy (slightly) increases the number of function evaluations (NFEs), InfoCNF also employs gating networks to learn the error tolerances of its ordinary differential equation (ODE) solvers for better speed and performance. We show empirically that InfoCNF improves the test accuracy over the baseline while yielding comparable likelihood scores and reducing the NFEs on CIFAR10. Furthermore, applying the same partitioning strategy in InfoCNF on time-series data helps improve extrapolation performance. Invertible models are attractive modelling choice in a range of downstream tasks that require accurate densities including anomaly detection and model-based reinforcement learning . These models enable exact latent-variable inference and likelihood estimation. A popular class of invertible models is the flow-based generative models (; ; ;) that employ a change of variables to transform a simple distribution into more complicated ones while preserving the invertibility and exact likelihood estimation. However, computing the likelihood in flow-based models is expensive and usually requires restrictive constraints on the architecture in order to reduce the cost of computation. Recently, introduced a new type of invertible model, named the Continuous Normalizing Flow (CNF), which employs ordinary differential equations (ODEs) to transform between the latent variables and the data. The use of continuous-time transformations in CNF, instead of the discrete ones, together with efficient numerical methods such as the Hutchinson's trace estimator , helps reduce the cost of determinant computation from O(d 3) to O(d), where d is the latent dimension. This improvement opens up opportunities to scale up invertible models to complex tasks on larger datasets where invertibility and exact inference have advantages. Until recently, CNF has mostly been trained using unlabeled data. In order to take full advantage of the available labeled data, a conditioning method for CNF -which models the conditional likelihood, as well as the posterior, of the data and the labels -is needed. Existing approaches for conditioning flow-based models can be utilized, but we find that these methods often do not work well on CNF. This drawback is because popular conditioning methods for flow-based models, such as in , make use of the latent code for conditioning and introduce independent parameters for different class categories. However, in CNF, for invertibility, the dimension of the latent code needs to be the same as the dimension of the input data and therefore is substantial, which in many unnecessary parameters. These additional but redundant parameters increase the complexity of the model and hinder learning efficiency. Such overparametrization also has a negative impact on other flow-based generative models, as was pointed out by , but is especially bad in the case of CNF. This is because the ODE solvers in CNF are sensitive to the complexity of the model, and the number of function evaluations that the ODE solvers request in a single forward pass (NFEs) increases significantly as the complexity of the model increases, thereby slowing down the training. This growing NFEs issue has been observed in unconditional CNF but to a much lesser extent . It poses a unique challenge to scale up CNF and its conditioned variants for real-world tasks and data. Our contributions in this paper are as follows: Contribution 1: We propose a simple and efficient conditioning approach for CNF, namely InfoCNF. Our method shares the high-level intuition with the InfoGAN , thus the eponym. In InfoCNF, we partition the latent code into two separate parts: class-specific supervised code and unsupervised code which is shared between different classes (see Figure 1). We use the supervised code to condition the model on the given supervised signal while the unsupervised code captures other latent variations in the data since it is trained using all label categories. The supervised code is also used for classification, thereby reducing the size of the classifier and facilitating the learning. Splitting the latent code into unsupervised and supervised parts allows the model to separate the learning of the task-relevant features and the learning of other features that help fit the data. We later show that the cross-entropy loss used to train InfoCNF corresponds to the mutual information between the generated image and codes in InfoGAN, which encourages the model to learn disentangled representations. We explore the speed-up achievable in InfoCNF by tuning the error tolerances of the ODE solvers in the model. ODE solvers can guarantee that the estimated solution is within a given error tolerance of the true solution. Increasing this tolerance enhances the precision of the solution but in more iterations by the solver, which leads to higher NFEs and longer training time. However, when training a neural network, it might not be necessary to achieve high-precision activation, i.e. the solution of the corresponding ODE, at each layer. Some noise in the activations can help improve the generalization and robustness of the network (; ; ; a). With carefully selected error tolerances, InfoCNF can gain higher speed and better performance. However, the process of manually tuning the tolerances is time-consuming and requires a large amount of computational budget. To overcome this limitation, we propose a new method to learn the error tolerances of the ODE solvers in InfoCNF from batches of input data. This approach employs learnable gating networks such as the convolutional neural networks to compute good error tolerances for the ODE solvers. We study methods to improve the large-batch training of InfoCNF including tuning and learning the error tolerances of the ODE solvers, as well as increasing the learning rates. We conduct experiments on CIFAR10 and show that InfoCNF equipped with gating networks outperforms the baseline Conditional Continuous Normalizing Flow (CCNF) in test error and NFEs in both small-batch and large-batch training. In small-batch training, InfoCNF improves the test error over the baseline by 12%, and reduces the NFEs by 16%. When trained with large batches, InfoCNF attains a reduction of 10% in test error while decreasing the NFEs by 11% compared to CCNF. InfoCNF also achieves a slightly better negative log-likelihood (NLL) score than the baseline in large-batch training, but attains a slightly worse NLL score in small-batch training. In order to better understand the impact of the gating approach to learn the error tolerances, we compare InfoCNF with and without the gating networks. In small-batch training, InfoCNF with gating networks achieves similar classification and density estimation performance as the same model without the gating networks, but reduces the NFEs by more than 21%. When trained with large batches, gating networks help attain a reduction of 5% in test error and a small improvement in NLLs. We also confirm the benefits of our gating approach on unconditional CNF and observe that on CIFAR10 learning the error tolerances helps reduce the NFEs by 15% while preserving the NLL. Furthermore, we explore the potential benefit of the partitioning strategy for time-series data. In our experiments, when the latent code is partitioned in the baseline LatentODE , the model achieves better performance in curve extrapolation. supervised signals (e.g., the class labels), and the model's parameters respectively. The superscript i is used to indicate the layer i. For example, x i is the set of input features into layer i. Invertible flow-based generative models such as RealNVP and Glow have drawn considerable interest recently. These models are composed of bijective transforms whose Jacobian matrices are invertible with tractable determinants. Let f (z; θ) denote such a transform applied on the latent variable z to generate the data x. Because of its bijective structure, the transform f not only allows exact inference of z given x but also enables exact density evaluation via the change of variable formula: The exact inference and exact density evaluation are preserved when stacking the bijective transforms into a deep generative model. The chain formed by the successive probability distributions generated by these transforms is called a normalizing flow, and the ing generative models are called flow-based generative models. The distribution of the latent code z at the top of the model is usually chosen to be a factorized standard Gaussian to simplify the computation, and the parameters θ are learned by maximizing the exact log-likelihood log p(X; θ) where X is the training set containing all training data x. While flow-based generative models enjoy nice properties, the requirements to ensure the invertibility and tractable computation restrict the expressive power of the model. Recently, the Continuous Normalizing Flows (CNFs) have been explored in to bypass the restrictive requirements in the flow-based generative models and allow the models to be expressive for more complicated tasks. CNF defines the invertible transforms via continuous-time dynamics. It models the latent variable z and the data x as values of a continuous-time variable z(t) at time t 0 and t 1, respectively. Given z, CNF solves the initial value problem to find x The change in log-density under this model follows the instantaneous change of variables formula Thus, CNF reduces the O(d 3) cost of computing the determinant to the O(d 2) cost of computing the trace. Taking advantage of the Hutchinson's trace estimator , this computation cost can be reduced to O(d) where d is the dimension of latent code z. Conditional CNF: To the best of our knowledge, there has not been a conditioning method particularly designed for CNF. However, since CNF belongs to the flow-based generative model family, conditioning methods for flow-based models can also be applied on CNF. In particular, the conditioning approach via a Gaussian Mixture Model (GMM) and an auxiliary classifier proposed in has been widely used. We refer to CNF equipped with this type of conditioning as the Conditional Continuous Normalizing Flow (CCNF). We use CCNF as the baseline for comparison in our experiments. In CCNF, the distribution of the latent code z follows a GMM whose means and scales are functions of the conditional signal y and parameterized by simple neural networks. Furthermore, an additional predictive model is applied on z to model the distribution p(y|z) via an auxiliary predictive task Here, q φ (y) and q θ (z) are usually chosen to be neural networks whose parameters are learned during training. While this conditioning method has been shown to enable flow-based models to do conditional image generation and perform predictive tasks such as object classification in , it is in fact rather inefficient. This is because the size of the latent code z in flow-based generative models is the same as the size of the input image and therefore often large. As a , the conditioning network q φ used to synthesize z and the predictive network q θ applied on z introduce a significant amount of additional parameters for the model to learn. InfoCNF: InfoCNF only uses a portion of the latent code z for conditioning. In particular, InfoCNF splits z into two non-overlapping parts -supervised latent code z y and unsupervised latent code z u -such that z = [z y, z u]. The supervised latent code z y captures the salient structured semantic features of the data distribution. In particular, denote the set of structured latent variables which account for those semantic features by y 1, y 2, · · ·, y L. For simplicity and efficient computation, we assume that these latent variables follow a factored distribution such that The supervised code z y is the concatenation of z y 1, z y 2, · · ·, z y L where z y i is the code that captures the latent variable y i. We use z y for conditioning the model. Similar to the conditional Glow, the distribution p(z y) is modeled by a GMM, N (z y |µ(y), σ 2 (y)), whose centers and scales are functions of the conditional signal y. As in Eq., these functions are parameterized by a neural network q φ (y). The posterior p(y|z) is then approximated by another neural network q θ (z y) applied on z y to solve the corresponding predictive task. The unsupervised code z u ∼ N (z u |0, I) can be considered as source of incompressible noise which accounts for other latent variations in the data. We learn InfoCNF by optimizing the supervised loss from q θ (z y) and the conditional log-likelihood log p(x|y) of the model. The learning objective of InfoCNF is given by where L Xent (ŷ, y) is the cross-entropy loss between the estimated labelŷ and the ground truth label y. β is the weighting factor between the cross-entropy loss L Xent (ŷ, y) and the conditional log-likelihood loss L NLL (x|y). L NLL (x|y) is given by where k are indices for layers in the network and log p(z y |y), log p(z u) are calculated from the formula for log-likelihood of a Gaussian distribution. In our notation, we set z K = z and z 0 = x. For each integral of the trace of the Jacobian in Eqn. 2, without generality, we choose t 0 = 0 and t 1 = 1. The mutual information between the generated images and the codes in InfoGAN is approximated by a variational lower bound via an "auxiliary" distribution, which is chosen to be a neural network. Since InfoCNF is an invertible model, the generated images from the model given the codes matches the input images. Thus, maximizing the mutual information between the generated images and the codes is equivalent to maximizing the cross-entropy loss between the estimated label and the ground truth label, which is the loss L Xent (ŷ, y) in Eqn. 4. Thanks to the invertibility of InfoCNF, we can eliminate the need of using an additional "auxiliary" network. Compared to CCNF, InfoCNF needs slightly fewer parameters since the size of the supervised code z y is smaller than the size of z. For example, in our experiments, z y is only half the size of z, and InfoCNF requires 4% less parameters than CCNF. This removal of unnecessary parameters helps facilitate the learning. As discussed in Section 4.3, our experiments on CIFAR10 suggest that InfoCNF requires significantly less NFEs from the ODE solvers than CCNF. This evidence indicates that the partition strategy in InfoCNF indeed helps alleviate the difficulty during the training and improves the learning of the model. Tuning the error tolerances: We explore the possibility of improving InfoCNF by tuning the error tolerances of the ODE solvers in the model. The advantage of this approach is two-fold. First, it reduces the number of function evaluations (NFEs) by the ODE solvers and, therefore, speeds up the training. Second, we hypothesize that the numerical errors from the solvers perturb the features and gradients, which provides additional regularization that helps improve the training of the model. We extend our approach of tuning the error tolerances of the ODE solvers by allowing the model to learn those tolerances from the data. We propose InfoCNF with learned tolerances, which associates each ODE solver in InfoCNF with a gating network that computes the error tolerance of the solver such that the model achieves the best accuracy and negative log-likelihood with the minimal NFEs. These gates are learnable functions that map input data or features into the tolerance values. In our experiments, we use CNNs for the gates (see Figure 1). The error tolerance decides how many iterations the solvers need to find the solution. This process is discrete and non-differentiable, which creates a unique challenge for training InfoCNF with learned tolerances. We exploit the reinforcement learning approach to solve this non-differentiable optimization and learn the parameters of the gating networks. In particular, at each gate, we formulate the task of learning the gating network as a policy optimization problem through reinforcement learning to find the optimal error tolerances. In InfoCNF, we assume that the error tolerances can be modeled by a Gaussian distribution. The sample sequence of the error tolerances drawn from our policy starting with input x is defined as is the sequence of network layers with parameters θ and are the outputs of the gating network at layer i. Here, the policy is to decide which error tolerance to use and is defined as a function from input x i to the probability distribution over the values of the error tolerances, π(. The CNN in the gating network is used to estimate the parameters of the probability distribution over the values of the error tolerances. We choose the rewards function R i = −NFE i, the negative of the number of function evaluations by the solver at layer i, so that our policy tries to fit the model well and do good classification while requiring less computation. The overall objective is given as: where the α balance between minimizing the prediction/NLL loss and maximizing the rewards. Employing REINFORCE , we can derive the gradients The first part of Eq. 7 is gradients of the cross-entropy and NLL loss while the second part corresponds to the REINFORCE gradients in which r i is the cumulative future rewards given the error tolerance estimated by the gating networks. This combination of supervised, unsupervised, and reinforcement learning encourages the model to achieve good performance on classification and density estimation while demanding less number of function evaluations by the ODE solvers. In this section, we empirically demonstrate the advantage of InfoCNF over the baseline CCNF when trained on CIFAR10. Throughout the experiments, we equip InfoCNF with the gating networks to learn the error tolerances unless otherwise stated. Compared to CCNF, InfoCNF achieves significantly better test errors, smaller NFEs, and better (in large-batch training) or only slightly worse (in smallbatch training) NLL. Furthermore, we observe that learning the error tolerances of the solvers helps improve InfoCNF in all criteria except for a slightly worse NLL in small-batch training and a similar NFEs in large-batch training. We also describe how we evaluate our model to make sure that reported are not biased by the numerical error from the ODE solvers in section 4.2. Dataset: We validate the advantages of our models on the CIFAR10 dataset. Uniform noise is added to the images, and during training, the data are randomly flipped horizontally. We use the FFJORD multiscale architecture composed of multiple flows as in for our experiments. The network details can be found in Appendix B. When conditioning the model, we use separate 1-layer linear networks for the classifier q θ and the condition encoder q φ. The parameters of the networks are initialized to zeros. In InfoCNF, we use half of the latent code for conditioning. We apply a dropout of rate 0.5 on the linear classifier. Training: We train both CCNF and InfoCNF with the Adam optimizer . When using the batch size of 900, we train for 400 epochs with learning rate of 0.001 which was decayed to 0.0001 after 250 epochs. When using the batch size of 8,000, we train for 400 epochs with learning rate of 0.01 which was decayed to 0.001 at epoch 120. The adaptive ODE solvers perform numerical integration and therefore have errors inherent in their outputs. When evaluating the models, we need to take these numerical errors into account. Since our method of learning the error tolerances is purely for training, a reasonable evaluation is to set the error tolerances to a small value at test time and report the on the test set. In order to find which small value of the error tolerance to use for evaluation, we train InfoCNF on 1-D synthetic data. Since a valid probability density function needs to integrate to one, we take InfoCNF trained on the synthetic 1-D data sampled from a mixture of three Gaussians and compute the area under the curve using Riemann sum at different error tolerance values starting from the machine precision of 10 −8 (see Appendix C for more details). Figure 2a shows that the numerical errors from InfoCNF is negligible when the tolerance is less than or equal to 10 −5. Thus, in our experiments, we set tolerances to 10 −5 at test time. In other to validate that a tolerance of 10 −5 still yields negligible numerical errors on complex datasets like CIFAR10, we also evaluate our trained models using tolerances of 10 −6, 10 −7, and 10 −8. We observe that when using those values for tolerances, our trained models yield the same test errors and NLLs as when using tolerance of 10 −5. The NFEs of the ODE solvers reported in our experiments are computed by averaging the NFEs over training epochs. Figure 2b shows the learned error tolerances from InfoCNF trained on CIFAR10 using small batches of size 900. We validate that the learned tolerances from the train and test sets are similar and thus the learned tolerances from our model do not only overfit the training data. Evaluation using the learned error tolerances with different batch sizes is discussed in Appendix D. Solver Tolerance Layer Solver Tolerance Figure 3a shows that compared to CCNF, InfoCNF improves the test classification error on CIFAR10 by 12% when training with small batches. Interestingly, the advantage of InfoCNF over CCNF still holds when we train the models with large batch size of 8k (10% improvement in test classification error, see Figure 3d). While InfoCNF facilitates classification, it also attains similar NLLs compared to CCNF in small-batch training and better NLLs in large-batch training (Figure 3b, e). Figure 3c and f show the evolution of NFEs during training of InfoCNF and CCNF. In small-batch experiments, InfoCNF is much more efficient than CCNF in the first 240 epochs but then the NFEs of InfoCNF increases and exceeds the NFEs of CCNF. Overall, the NFEs from InfoCNF during training is 16% less than the NFEs from CCNF. When training with large batches, InfoCNF requires 11% less NFEs than CCNF. We study the benefits of using gating networks to learn the error tolerances by comparing InfoCNF with learned error tolerances and with error tolerances set to 10 −5. When training both models with large batches, InfoCNF with learned tolerances achieves the test error of 4% less than the test error attained by the same model with fixed error tolerances. InfoCNF with learned tolerances also yields slightly better NLLs (3.64 vs. 3.67). In small-batch training, both models achieve similar test errors and NLLs. Unlike with the test error and likelihood, the InfoCNF with learned tolerances significantly reduces the NFEs compared to InfoCNF with fixed error tolerances when both are trained with small batches-a reduction of 21%. In large-batch training, the NFEs from both models are similar. In summary, when InfoCNF with learned tolerances has the advantage over InfoCNF with tolerances set to 10 −5, it is a notable improvement. Otherwise, InfoCNF with learned tolerances is as good as or only slightly worse than the its counterpart. We summarize the comparison between InfoCNF and CCNF in Table 1. Automatic Tuning vs. Manual Tuning: In Figure 3d, e, and f, InfoCNF is trained with the manuallytuned ODE solvers' tolerances since otherwise the models are hard to train. Thus, for large-batch training, we are comparing the learned tolerances in InfoCNF with the manually tuned tolerances. Furthermore, Figure 10 in Appendix F shows similar comparison for small-batch training. In both experiments, we observe that our automatic approach learns the tolerances which outperform the manually-tuned ones in both classification and density estimation while being only slightly worse in term of NFEs. Also, our automatic approach via reinforcement learning requires much less time and computational budget to find the right values for the tolerances compared to the manual tuning. Unconditional CNF: Inspired by the success of InfoCNF, we explore whether the same reinforcement learning approach still work for unconditional models. We compare the baseline CNF in with CNF which uses the CNN gates to learn the error tolerance of the ODE solvers in small-batch training setting. While both models yield similar NLLs (3.37 bits/dim), CNF with learned tolerances significantly reduces the NFEs by 15% compared to the baseline CNF (see Figure 4). Training neural networks using large batches is much faster than small batches, but suffers from poor generalization . In order to conduct meaningful experiments with large-batch training, we study methods to improve the performance of CCNF and InfoCNF. Our experiments confirm that tuning the error tolerance of the ODE solvers, together with using larger learning rate as suggested in , helps enhance the performance of both InfoCNF and CCNF, ing in better test error, lower NLLs, and smaller NFEs (see Figure 5). We study the potential benefit of applying the conditioning method via partitioning the latent code in InfoCNF on time-series data. In this experiment, we choose the LatentODE as the baseline model and conduct the experiment on the synthetic bi-directional spiral dataset based on the one proposed in. In particular, we first generate a fixed set of 5,000 2-dimensional spirals of length 1,000 as ground truth data: 2,500 curves are clockwise and the other 2,500 curves are counter-clockwise. These spirals have different parameters. We then randomly sample one spiral of 200 equally-spaced time steps from each of these ground truth spirals. We add Gaussian noise of mean 0 and standard deviation 0.3 to these small spirals to form the training set for our experiments. The test set is generated in the similar way. We train the LatentODE with and without our conditioning strategy for trajectory fitting and test the trained models for trajectory fitting and extrapolation on data in test set. More details on the dataset, network architecture, and training details are provided in Appendix E. We observe that the LatentODE equipped with our conditioning strategy outperforms the baseline Latent ODE on trajectory fitting and extrapolation tasks, especially in unseen domains. Conditional generative models: applies a generalized linear model on the latent code of a flow-based generative model to compute both p(x) and p(y|x) exactly. Their method does not consider splitting the latent code and is complimentary to our partitioning approach. The conditioning approach proposed in is close to InfoCNF. However, in contrast to this method, InfoCNF does not penalize the mismatch between the joint distribution of the supervised and the unsupervised code p(z y, z u) and the product of their marginal distributions p(z y)p(z u). InfoCNF also does not minimize the MMD distance between the distribution of the backward prediction against the prior data distribution. Instead, we maximize the likelihood of the input image given the label p(x|y) = p(z y |y)p(z u). uses an encoder to generate the partitioned latent code for Glow. This encoder is trained by optimizing an adversarial loss as in the generative adversarial networks (GANs) so that its generated latent codes are indistinguishable from the latent codes computed by Glow for real data. Our InfoCNF directly splits the latent code without using an extra complicated architecture like GANs, which might introduce more instability into the ODE solvers and slow down the training. Adaptive computation: The use of gating networks and reinforcement learning to learn a policy for adaptive computation has been studied to enhance the efficiency of the neural networks. (b ;c) employ gating networks to decide which blocks to skip in residual networks. develops a reinforcement learning framework to automatically select compression techniques for a given DNN based on the usage demand. Other works including (; ; ;) study methods to select the number of evaluations in recurrent and residual networks. To the best of our knowledge, our InfoCNF with learned tolerances is the first that learns the error tolerances of the ODE solvers in CNF. Large-batch training: Various methods have been proposed to improve the generalization of neural networks trained with large batches by scaling the learning rate and scheduling the batch size. Among them are (; ; ;). These methods are complimentary to our approach of tuning the error tolerances. We have developed an efficient framework, namely InfoCNF, for conditioning CNF via partitioning the latent code into the supervised and unsupervised part. We investigated the possibility of tuning the error tolerances of the ODE solvers to speed up and improve the performance of InfoCNF. We invented InfoCNF with gating networks that learns the error tolerances from the data. We empirically show the advantages of InfoCNF and InfoCNF with learned tolerances over the baseline CCNF. Finally, we study possibility of improving large-batch training of our models using large learning rates and learned error tolerances of the ODE solvers. We validate InfoCNF on the MNIST dataset. On MNIST, InfoCNF with learned error tolerances, InfoCNF with fixed error tolerances, and the baseline CCNF achieve similar NLLs and test errors. However, the InfoCNF with learned and fixed error tolerances are 1.5x and 1.04x faster than the baseline CCNF, respectively (416 NFEs/epoch vs. 589 NFEs/epoch vs. 611 NFEs/epochs). We include the detailed in Table 2. All experiments are conducted with batch size 900, and the are averaged over 3 runs. Below is the MNIST images generated by InfoCNFwith learned error tolerances. For experiments on CIFAR10, we use the FFJORD multiscale architecture composed of multiple flows as in for our experiments. The network uses 4 scale blocks, each of which contain 2 flows, a "squeeze" operator, and then other 2 flows. Each flow is made of 3 convolutional layers with 64 filters whose kernel size is 3. The squeeze operators are applied between flows to down-sample the spatial resolution of the images while increasing the number of channels. We apply the softplus nonlinearity at each layer. This architecture tries to reduce the dimensionality of the latent representation at each level while preserving invertibility. It was based on the multiscale architecture in , which has been widely used as the base architecture for invertible models. Also, we parameterize q θ and q φ by separate linear networks. C SYNTHETIC 1-D USED TO ESTIMATE THE VALUE OF ERROR TOLERANCES OF THE ODE SOLVERS AT TEST TIME As mentioned in Section 4.2, in order to estimate the value of error tolerances of the ODE solvers which yields negligible numerical errors during test time, we train InfoCNF on a 1-D synthetic dataset and calculate the area under the curve at different error tolerance values starting from the machine precision of 10 −8. Examples in this dataset are sampled from the mixture of Gaussians shown by the blue curve in Figure 8. The orange curve in Figure 8 represents the distribution learned by our InfoCNF when the error tolerances are set to 10 −5. Groundtruth distribution Learned distribution Figure 8: Distribution from which we sample the 1-D synthetic data for computing the value of error tolerances used to evaluate the trained InfoCNF (blue curve) and the distribution learned by InfoCNF (orange curve) We explore if the error tolerances computed from batches of input data can be used to evaluate the trained InfoCNF. First, we repeat the experiment on 1-D synthetic data described in Section 4.2 and Appendix C above using the learned error tolerances instead of the fixed error tolerances. We observe that the numerical error in this case is 0.00014, which is small enough. We further use the learned error tolerances to evaluate our trained InfoCNF on CIFAR10 with small-batches. Distribution of these error tolerances at different layers can be found in Figure 2b. The test error and NLL we obtain are 20.85 ± 1.48 and 3.566 ± 0.003, which are close enough to the obtained when setting the error tolerances to 10 −5 (test error and NLL in this case are 20.99 ± 0.67 and 3.568 ± 0.003, respectively, as shown in Table 1). Furthermore, when using the learned error tolerances to evaluate the trained model, we observe that the trained InfoCNF achieves similar classification errors, negative log-likelihoods (NLLs), and number of function evaluations (NFEs) with various small values for the batch size (e.g. 1, 500, 900, 1000, 2000). However, when we use large batch sizes for evaluation (e.g. 4000, 6000, 8000), those metrics get worse. This sensitivity to test batch size is because the error tolerances in InfoCNF are computed for each batch of input data. E DATASET, NETWORK ARCHITECTURES, AND TRAINING DETAILS FOR EXPERIMENTS ON SYNTHETIC TIME-SERIES DATA E.1 DATASET In the experiments on time-series data in Section 4.6, we use the synthetic bi-directional spiral dataset based on the one proposed in. In particular, we first generate a fixed set of 5,000 2-dimensional spirals of length 1,000 as ground truth data: 2,500 curves are clockwise and the other 2,500 curves are counter-clockwise. These spirals have different parameters. The equations of the ground truth spirals are given below: Counter-Clockwise: R = a + b × t; x = R × cos(t) + 5; y = R × sin(t). Here a and b serve as the system parameters and are sampled from the Gaussian distributions N (1.0, 0.08) and N (0.25, 0.03), respectively. We then randomly sample one spiral of 200 equally-spaced time steps from each of these ground truth spirals. We add Gaussian noise of mean 0 and standard deviation 0.3 to these small spirals to form the training set for our experiments. The test set is generated in the similar way. In our experiments, the baseline is the LatentODE model in. The RNN encoder that estimates the posterior q φ (z t 0 |x t 0, x t 1, · · ·, x t N) from the observations is fully connected and has 25 hidden units. The latent state z t i are 5-dimensional vectors. Different from the LatentODE, in our model, the first three dimensions of the initial latent state z t 0 are for the supervised code z y and the other two dimensions are for the unsupervised code z u. Like the LatentODE, given the initial latent state z t 0, our model computes the future states z t 1, z t 2, · · ·, z t N by solving the equation The dynamic function f is parameterized with a one-hidden-layer network of 20 hidden units. Also, the decoder that reconstructsx t i from z t i is another one-hidden-layer network of 20 hidden units. Additionally, in our model, the conditioning function q φ and the supervised function q θ are linear networks. In our experiments, the conditioning/supervised signal y are the values of the parameters a, b, and the direction of the spiral (i.e. clockwise or counter-clockwise). We train the model with the Adam optimizer . We use batch training with the learning rate of 0.001. The training is run for 20,000 epochs. We show the test error, NLLs, and the NFEs of InfoCNF with manually-tuned error tolerances for small-batch training on CIFAR10 (the yellow curves) in the Figure 10 in comparison with the from CCNF, InfoCNF, and InfoCNF with learned tolerances in the main text. As can be seen, InfoCNF with learned tolerances, which learns the error tolerances, is still as good as InfoCNF with manually-tuned error tolerances (except for the slightly worse NFEs) when trained with small batches. The NLLs discussed in the main text are the conditional negative log-likelihoods − log p(x|y). We would also like to compare the marginal negative log-likelihoods − log p(x) of CCNF, InfoCNF, and InfoCNF with learned tolerances. Figure 11 shows that like in the case of the conditional NLLs, InfoCNF with learned tolerances in better marginal NLLs in large-batch training but slightly worse NLLs in small-batch training compared to InfoCNF and CCNF. Improving Large-Batch Training of InfoCNF and CCNF: We would like to understand if using large learning rate and tuned error tolerances of the ODE solvers help improve the marginal NLLs in large-batch training. Figure 12 confirms that both InfoCNF and CCNF trained with large learning rate and tuned error tolerances yield much better marginal NLLs in large-batch training compared to the baseline training method which uses small learning rate and constant error tolerances. CCNF without large learning rate and tuned tolerance InfoCNF without large learning rate and tuned tolerance Figure 12: Test marginal NLLs of InfoCNF and CCNF trained on CIFAR10 using large batch size with and without large learning rate and manually-tuned error tolerance. Given promising from training our models with large batches, we would like to compare large-batch with small-batch training in term of speed. Currently, the CNF-based models studied in our paper yield better test classification errors and negative log-likelihoods on CIFAR10 when trained with small batches than with large batches. However, since large-batch training attains smaller NFEs than small-batch training, we would like to explore if the model trained with large batches can reach certain test classification errors and NLLs faster than the same model trained with small batches. In our experiments, we choose to study InfoCNF with learned tolerances since it yields the best test error and NLLs in large-batch training while requiring small NFEs. We compare InfoCNF with learned tolerances trained with large batches and with small batches. We also compare with the baseline CCNF trained with small batches. Figure 13 shows that InfoCNF with learned tolerances trained with large batches achieves better test error than CCNF trained with small batches while converging faster. However, it still lags behind CCNF trained with small batches in term of NLLs and InfoCNF with learned tolerances trained with small batches in both test error and NLLs. This suggests future work for improving large-batch training with the CNF-based models. Since InfoCNF with learned tolerances reduces the NFEs while improving the test error and NLL, we would like to explore the possibility of using InfoCNF with learned tolerances for training larger models to gain better performance in classification and density estimation. We train a InfoCNF with learned tolerances with 4 flows per scale block (compared to 2 flows per scale block as in the original model) on CIFAR10 using large batches and compare the with the original InfoCNF with learned tolerances and the baseline CCNF. Here we follow the same notation as in to describe the models. We call the large InfoCNF with learned tolerances with 4 flows per scale block the 2x-InfoCNF with learned tolerances. Figure 14 shows that the 2x-InfoCNF with learned tolerances yields better test errors and NLLs while increasing the NFEs compared to both InfoCNF and CCNF.

Represent the short summary for finding the corresponding scientific article (long):

We propose the InfoCNF, an efficient conditional CNF that employs gating networks to learn the error tolerances of the ODE solvers

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.