Question: represent the article for finding a single-sentence tldr:

Question: We propose a novel subgraph image representation for classification of network fragments with the target being their parent networks. The graph image representation is based on 2D image embeddings of adjacency matrices. We use this image representation in two modes. First, as the input to a machine learning algorithm. Second, as the input to a pure transfer learner. Our from multiple datasets are that 1. deep learning using structured image features performs the best compared to graph kernel and classical features based methods; and, 2. pure transfer learning works effectively with minimum interference from the user and is robust against small data. With the advent of big data, graphical representation of information has gained popularity. Being able to classify graphs has applications in many domains. We ask, "Given a small piece of a parent network, is it possible to identify the nature of the parent network (Figure1)?" We address this problem using structured image representations of graphs. Adjacency matrices are notoriously bad for machine learning. It is easy to see why, from the unstructured image of a small fragment of a road network, in figure (a) below. Though the road network is structured, the random image would convey little or no information to machine learning algorithms (in the image, a black pixel at position (i, j) corresponds to an edge between nodes i and j). Reordering the vertices (figure (b) below) gives a much more structured image for the same subgraph as in (a). Now, the potential to learn distinguishing properties of the subgraph is evident. We propose to exploit this very observation to solve a basic graph problem (see Figure 1). The datasets mentioned in Figure 1 are discussed in Section 2.4. We stress that both images are lossless representations of the same adjacency matrix. We use the structured image to classify subgraphs in two modes: (i) Deep learning models on the structured image representation as input. (ii) The structured image representation is used as input to a transfer learner (Caffe: see Section 2.3) in a pure transfer learning setting without any change to the Caffe algorithm. Caffe outputs top-k categories that best describe the image. For real world images, these Caffe-descriptions are human friendly as seen in Figure 2a. However, for network-images, Caffe Figure 2: An image of a dog and a structured image of a Facebook graph sample vs their corresponding maximally specific classification vectors returned by Caffe gives a description which doesn't really have intuitive meaning (Figure 2b). We map the Caffedescriptions to vectors. This allows us to compute similarity between network images using the similarity between Caffe description-vectors (see Section 2). The significant difference between our work and previous approaches is that we transform graph classification into image classification. We propose an image representation for the adjacency matrix. We use this representation as input to machine learning algorithms for graph classification, yielding top performance. We further show that this representation is powerful enough to serve as input to a pure transfer learner that has been trained in a completely unrelated image domain. The Adjacency Matrix Image Representation. Given a sample subgraph from a parent network, the first step is to construct the image representation. We illustrate the workflow below. Image Embedding Image Structuring We use the novel method proposed in which produces an adjacency matrix that is invariant to permutations of the vertices in the adjacency matrix. The image is simply a "picture" of this permutation-invariant adjacency matrix. Deep Learning Using the Adjacency Matrix Image Representation. We train deep image classifiers (discussed in Section 2.2) on our image representation as in Figure 3. Figure 3: Classification of structured image embeddings using Deep LearningWe compared performance with several methods, including graph kernel classifiers and classifiers based on standard topological features of the graph. Our image representation performs best. Transfer Learning Using the Adjacency Matrix Image Representation. When data is scarce or there are many missing labels, a popular option is transfer learning to leverage knowledge from some other domain. Typically the other domain is closely related to the target application. It is unusual for learning in a completely unrelated domain to be transferable to a new target domain. We show that our image representation is powerful enough that one can directly transfer learn from the real world image domain to the network domain (two completely unrelated domains). That is, our image representation provides a link between these two domains enabling classification in the graph domain to leverage the wealth of techniques available to the image domain. The image domain has mature pre-trained models based on massive data. For example, the opensource Caffe deep learning framework is a convolutional neural network trained on the ImageNet data which can recognize everyday objects like chairs, cats, dogs etc. BID11 ). We use Caffe as is. Caffe is a black box that provides a distribution over image classes which we refer to as the Caffe-classification vector. The Caffe are then mapped back into the source domain using a distance-based heuristic e.g. Jaccard distance and K-nearest neighbors as in Figure 4. Classification (Jaccard plus K-NN)Figure 4: Classification of structured images using the classification vectors obtained from Caffe Images, not graphs, are passed through the Caffe deep neural network, and as we shall show, one can get good performance from as little as 10% of the training data used in the ab initio machine learning approach. It is quite stunning that such little training data together with un-tweaked transfer learning from a completely unrelated domain can perform so well. The reason is that our image representation provides very structured images (human-recognizable) for real world networks. Though these images are not traditional images like those used in training Caffe, Caffe still maps the different structured images to different distributions over its known classes, hence we are able to transfer this knowledge from Caffe to graph classification. introduced the problem we study: Can one identify the parent network from a small subgraph? How much does local information reveal about the parent graph at the global level? We approach the problem from the supervised setting and the unsupervised transfer learning setting. There is previous work on similar problems using graph kernels BID9; BID10; BID12 ). Such methods use kernels to compute similarities between graphs and then algorithms like SVM for classification. Choosing kernels is not straightforward and is certainly not a one-size-fits-all process. Further, these kernel methods do not scale well for very large graphs. We compare with one such method proposed by BID21.Another approach is to constuct feature vectors from topological attributes of the subgraph BID17 ). The topological characteristics of social networks have been extensively studied by BID0 BID3 BID22. The shortcomings of this approach are that it is difficult to come up with a master set of features that can be used to represent graphs from different domains. For example, assortativity could be an important feature in social networks while being of little significance in case of road networks. It is hard to identify beforehand what features need to be computed for a given problem, thus leading to a trial and error scenario. One of the methods we compare is logistic regression. Transfer learning is useful when a classification task in one domain can leverage knowledge learned in a related domain (see BID18 for an extensive survey). BID20 introduced a method called self-taught learning which takes advantage of irrelevant unlabeled data to boost performance. BID30 discuss heterogeneous transfer learning where they use information from text data to improve image classification performance. BID19 create a new representation from kernel distances to large unlabeled data points before performing image classification using a small subset of reference prototypes. The rest of the paper is organized as follows. In Section 2 we give more details of our approaches to subgraph classification. Section 3 present the comparing the performance of our approach with other approaches. We conclude in Section 4 with some possible future directions. Given fragments of a large network we first obtain the structured image representation. In the transfer setting, we pass these training samples to the classifiers and record the . In the unsupervised setting, we feed the training samples to Caffe framework and obtain the label-vectors. To classify a test subgraph, we first obtain its label-vector through Caffe, compute the distance between the test vector and the training vectors, and classify using majority class of the nearest−k vectors. We explain each step in detail below. An adjacency matrix of a graph can be thought of as a monochrome image with 1s corresponding to dark pixels and 0s corresponding to white pixels. This observation, however, is of limited practical use since it is not permutation invariant. We utilize a novel technique for producing a permutationinvariant ordering of the adjacency matrix, first given in. The authors describe several ways to sort an adjacency matrix like page rank, degree based sorting, etc. They show that the BFS-like approach introduced works best. The ordering starts with the node of highest degree (ties are broken using k-neighborhood size for k = 2, then k = 3, . . .). Subsequently, the ordering proceeds based on a combination of shortest paths and degrees. Details can be found in. The ordering scheme in permutation-invariant adjacency matrices from which we obtain structured images. We observe that the image embeddings have enough structure so that even the human eye can distinguish between different networks without much effort. This is the intuition behind our approach. Neural networks are highly successful in recognizing real world objects that have high structural properties. For example, all dogs have similar features although they are individually different. Our primary objective is to use deep learning to learn to classify subgraphs to their parent networks using our image representation of the subgraph. However, we also tested a wide variety of other methods as outlined in the table below (more details are in the Appendix, including the references). All these methods are tested in a standard supervised learning framework where n training examples (x i, y i) are given (the input x i and y i the target label). For DBN, CNN and SdA, the input x i is our image representation of the subgraph. For DCNN and GK, the input x i is the graph itself, represented as an adjacency matrix. For LR, the input x i is a set of 15 classical features (assortativity, clustering coefficient, etc.). Caffe is a deep learning framework developed by BID8, Berkeley AI Research and by community contributors with expressive and modular architecture in mind. It has been extensively used in image classification and filter visualization, learning handwritten digital data, and style recognition among other things as seen in. We use a pre-trained model that is trained on a crowd-sourced labeled data set ImageNet. As of 2016, ImageNet had more than 10 million hand-annotated images. The massive volume combined with a deep convolutional neural network gives us fine-grained discriminatory power for images. Given an image, the output of our Caffe-based image classification function is a vector of (label, label-probability) tuples, sorted in decreasing order of probabilities. An example of the output for a real image is shown in Figure 2a. Although Caffe has not been trained on image embeddings of graphs, such images nonetheless produce vectors that have sufficient discriminatory information that we extract using the post processing step (see Section 2.3.1). An example of a vector corresponding to a Facebook network sample is shown in Figure 2b.Caffe provides either maximally accurate or maximally specific classification. We use the maximally specific categorization option in Caffe for our work. Further, while we have shown cardinality-5 vectors for brevity in Figure 2, we use cardinality-10 vectors in our experiments. Caffe provides a set of label vectors L i for each training network x i. Each label vector is a tuple of (label, label-probability pairs) as deemed by Caffe. In this work, we ignore the probabilities, and treat each vector as an unordered list of labels (strings). Each training vector also has a ground truth parent label. We use Jaccard similarity to compute a similarity metric between two label-vectors L j and L k: DISPLAYFORM0 We leave for future work the use of more sophisticated metrics which could use the probabilities from Caffe -our goal is to demonstrate the potential of even this simplest possible approach. For a test graph, we get the label-vector T from Caffe and then compute the k nearest training vectors using the Jaccard distance d(L i, T) to each training vector L i and classify using the majority class C among these k nearest training vectors (ties are broken randomly). The test example is correctly classified if and only if its ground truth matches C. One advantage of the k nearest neighbor approach is that it seamlessly extends to an arbitrary number of parent network classes. We used a variety of datasets ranging from citation networks to social networks to e-commerce networks (see TAB3 and the brief descriptions in the Appendix). Our deep networks learn signatures from the image representation of each graph class. To gein some insight into these network signatures, we show the top principal component of the images of each network. This process is very similar to the one carried out in BID24. We grouped all the samples into their respective categories. Then, we vectorized each sample; i.e., we reshaped each sample from n × n to 1 × n 2. We then performed principal component analysis on the vectorized dataset. We show the top principal component for each network in FIG2.In FIG2, we show the structured image representations of sample 64-node subgraphs from each of the 9 datasets. These are adjacency matrices that have gone through the structuring process described in Section 2.1. Observe that the images for different graphs are well structured and quite different. This is why the deep networks are able to perform well at classifying the subgraphs, as we will see later in Section 3. Our approach leverages the structure as well as the distinctness of the image representations. DISPLAYFORM0 In this section we will describe our experimental setup and present the we obtained from the two approaches we have described in the earlier sections. We perform the graph classification task using the above mentioned 9 parent networks. We perform a random walk on each of these networks 5, 000 times and we stop when we get the required number of nodes per sample denoted by n. We carry out this exercise 4 times and set n to 8, 16, 32 and 64 respectively. So, with 9 networks and 5, 000 samples per network, we create 4 datasets with 45, 000 samples each. Each dataset is of the size 45, 000 × n × n. For a given dataset, we randomly chose 33% of the dataset for validation and set aside 33% of the dataset for testing. The accuracy score is defined as the ratio of the sum of the principal diagonal entries of the confusion matrix over the sum of all the entries of the confusion matrix (sometimes called the error matrix). A confusion matrix C is such that C i,j is equal to the number of observations known to be in class i but predicted (confused) to be in class j. We report the best accuracy score for each classifier in the following CNN was the best performing classifier while DCNN and GK were the poorest performers. This shows that off-the-shelf deep learning with graph image features with no tuning of the classifier is better than any other method we tested. FIG3 summarizes the performance of all the methods we tested concisely. As expected, we obtained higher accuracy in classification as the number of nodes per sample n increased. Note that straight line in the figure refers to the accuracy achieved for random guessing of the classes. We would like to point out the fact that even with only 8 nodes, we were able to do significantly better than random while being better than graph kernel methods and the feature based logistic regression classifier. We would like to note that although LR performs okay, it is very hard to choose the features when graphs come from different domains. One set of features that worked best in one scenario may not be the best in another. So, when using an untuned CNN that requires very little effort from the user handily outperforms these cumbersome methods, it is an easy choice to make. This is one of the biggest observations of our study. We also mixed the samples with different n's to create hybrid datasets. We observed that the performance of the classification was better when the mixture had more samples with higher n. This is inline with our expectation and the in FIG3. Interested readers can refer to the Appendix for more details. Graph kernel and feature-based methods performed better than DCNN, but not as well as the image embedding based methods. Kernel methods are complex and usually slow and the fact that they have performed poorly do not make them attractive. One may notice that the accuracy scores for LR are very comparable to SdA. However, we would like to remark that LR is a lossy method since it approximates the graph and boils it down to a handful of features. It is very hard to decide which features must be used and the choice may vary for graphs in different domains in order to get optimal . However, our approach is completely lossless. The structured image representation of the graph has every bit of the information the adjacency matrix does. So, we would not have to make any compromises to get the best out of the data. We would like to make a note about DCNN. Out of the 4 neural network classification models we have used in this work, DCNN is the only one that takes a graph as an input instead of images. In fact, it takes two inputs: an adjacency matrix and a design matrix. The design matrix contains information about each node in the adjacency matrix. For example, information like average degree, clustering co-efficient etc. can be provided in the design matrix. In order to make the comparison between DCNN and the other classification models as fair as possible, we specified the values of the pixels in the image embedding in our design matrices throughout our experiments. When other information (assortativity, centrality etc.) were provided we observed no significant increase in performance. This is because these properties can be calculated from the graph which is already an input. The neural network is expected to have learned these features already. In this section we present our experimental of our second approach to graph classification: transfer learning. We show that our transfer learning approach is highly resilient to sparse training data. We achieve a respectable accuracy even when only 10% of the data was used for training. Caffe can be treated as a black-box that requires very little interference from the user. This is significant because when one does not have access to ample data to train their own neural networks, transfer learning can be a very quick and effective fix to get the job done. These are the other two big observations to take home from our work. We also carried out the following experiments: classification. The detailed are relegated to the Appendix, but we note that accuracy scores were in the high 90s and 80s for (a) and around the 60s for (b). While the multi-class scores are not as glamorous as those presented in Section 3.1, they are still worth mentioning. FIG4 plots the accuracy numbers from the (a) set of experiments as we progressively increase the proportion of data used for training. Each point on the x-axis shows the percentage of available data that was used for training, the reminder used for testing. Note that the reduction of training percentage hardly impacts accuracy except for a slight dip when only 10% is used for training. Most learning techniques, especially deep neural networks are sensitive to training data volume. The relative insensitivity of our approach is likely due to the fact that we leverage pre-trained recognition engine in the image domain which has already been trained with a massive volume of images. This shows that the transfer learning approach is very robust and resilient to sparse training data. Finally, we study the impact of k, the neighborhood size for the majority rule, on accuracy. We show the detailed analysis in the Appendix but the take away fact is that as long as k > 15, we do not have to worry too much about tuning k, showing once again that the approach is robust. Our experiments overwhelmingly show that the structured image representation of graphs achieves successful graph classification with ease. The image representation is lossless, that is the image embeddings contain all the information in the corresponding adjacency matrix. Our also show that even with very little information about the parent network, Deep network models are able to extract network signatures. Specifically, with just 64-node samples from networks with up to 1 million nodes, we were able to predict the parent network with > 90% accuracy while being significantly better than random with only 8-node samples. Further, we demonstrated that the image embedding approach provides many advantages over graph kernel and feature-based methods. We also presented an approach to graph classification using transfer learning from a completely different domain. Our approach converts graphs into 2D image embeddings and uses a pre-trained image classifier (Caffe) to obtain label-vectors. In a range of experiments with real-world data sets, we have obtained accuracies from 70% to 94% for 2-way classification and 61% for multi-way classification. Further, our approach is highly resilient to training-to-test ratio, that is, can work with sparse training samples. Our show that such an approach is very promising, especially for applications where training data is not readily available (e.g. terrorist networks).Future work includes improvements to the transfer learning by improving the distance function between label-vectors, as well as using the probabilities from Caffe. Further, we would also look to generalize this approach to other domains, for example classifying radio frequency map samples using transfer learning. Deep Belief Network (DBN) DBNs (see:) consist of multiple layers of unsupervised Restricted Boltzmann Machines (RBMs) where the output of each RBM is used as input to the next. BID7 ). The RBMs can be trained greedily and a supervised back-propagation step can be used for fine-tuning. Typically, DBNs have an input layer, hidden layer(s) and a final output layer. The input layer contains a node for each of the entries in the feature vector. For example, if the input is an image of size 8 × 8, then there will be 8 × 8 = 64 nodes in the input layer. The hidden layers consist of RBMs where the output of each of RBMs are used as input to the next. Finally, the output layer contains a node for each class. The probabilities of each class label is returned. The one with the highest probability is chosen as the overall classification for the given input. Convolutional Neural Network (CNN) CNNs are a category of neural networks that have been proven to be very effective in image classification tasks. Although they have been extensively used on real world images, we believe it is essential to include CNNs in our experiments. The building blocks of a CNN are convolution layers, non-linear layers such as Rectified Linear Units (ReLU), pooling layers and fully connected layers for classification. Stacked De-Noising Auto-Encoder (SdA) We implement the stacked de-noising auto-encoder (SdA) based on greedy training algorithm presented in. In a regular multi-layer deep neural network, each layer is trained to "reconstruct" the input from the previous layer. Then, the system is fine tuned by using back-propagation. In SdA, instead of the original input, a noisy input is fed to the system. This model, introduced by BID1, works on the graphs themselves rather than the image embeddings of their adjacency matrices. DCNNs provide a flexible representation of graphical data that encodes node features, edge features, and purely structural information with little preprocessing. DCNNs learn diffusion-based representations from graph-structured data which is made possible by the new diffusion-convolution operation. Graph Kernel Approach We use a graph kernel introduced in BID21 which uses a rapid feature extraction scheme based on the Weisfeiler-Lehman test of isomorphism on graphs. It maps the original graph to a sequence of graphs, whose node attributes capture topological and label information. We use the code provided here Ghisu as-is in our experiments. Feature-based Classification We compute 15 classic features for every graph sample and use just these to perform classification using logistic regression. The said features are: transitivity, average clustering co-efficient, average node connectivity, edge connectivity, average eccentricity, diameter, average shortest path, average degree, fraction of single-degree nodes, average closeness centrality, central points, density, average neighbor degree and top two eigen values of the adjacency matrix. Citation This citation network is from Arxiv HEP-PH (high energy physics phenomenology). If a paper i cites paper j, the graph contains a directed edge from i to j. There are 34, 546 nodes with 421, 578 edges. See: BID14; BID4 Facebook This social network contains "friends lists" from Facebook. There are 4039 people (nodes) and there is an undirected edge between nodes if they are friends. There are 88, 234 such edges. See: BID13 Road Network This is a road network of Pennsylvania. Intersections and endpoints are represented by nodes, and the roads connecting these intersections are represented by undirected edges. There are 1, 088, 092 nodes and 1, 541, 898 edges in this network. See: BID16 Web Nodes in this network represent web pages and directed edges represent hyperlinks between them. There are 875, 713 nodes and 5, 105, 039 edges. See: BID16 Wikipedia This is a Wikipedia hyperlink graph. A condensed version of Wikipedia was used in the collection of this dataset. There are 4, 604 articles (nodes) with 119, 882 links (edges) between them. See: BID25; BID26 Amazon This is a product co-purchase network of amazon.com. The nodes are products sold on amazon.com. There is an undirected edge between two products if they are frequently co-purchased. There are 334, 863 nodes and 925, 872 edges. See: BID15 DBLP This is a co-authorship network. It has authors for its nodes and there is an undirected edge between them if they have co-authored at least one paper. There are 317, 080 nodes and 1, 049, 866 We present the detailed for different mixtures of datasets that we experimented with in the supervised setting. Although the performance deteriorates when different n's are mixed, the relative ordering of the methods w.r.t. their performances remains the same. Figure 10 shows the variation of accuracy when k is varied in steps of 8 from k = 7 to k = 47, with the base case k = 15 used for the tabulated above included for comparison. As can be seen, except for a couple of cases with k = 7 providing a somewhat lower accuracy, the variation is within 1% of the base case. Thus, as long as k > 15, we do not have to worry too much about tuning k, showing that the approach is robust.

Represent the short summary for finding the corresponding scientific article (long)

We convert subgraphs into structured images and classify them using 1. deep learning and 2. transfer learning (Caffe) and achieve stunning results.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: This paper explores the use of self-ensembling for visual domain adaptation problems. Our technique is derived from the mean teacher variant (Tarvainen et. al 2017) of temporal ensembling (Laine et al. 2017), a technique that achieved state of the art in the area of semi-supervised learning. We introduce a number of modifications to their approach for challenging domain adaptation scenarios and evaluate its effectiveness. Our approach achieves state of the art in a variety of benchmarks, including our winning entry in the VISDA-2017 visual domain adaptation challenge. In small image benchmarks, our algorithm not only outperforms prior art, but can also achieve accuracy that is close to that of a classifier trained in a supervised fashion. The strong performance of deep learning in computer vision tasks comes at the cost of requiring large datasets with corresponding ground truth labels for training. Such datasets are often expensive to produce, owing to the cost of the human labour required to produce the ground truth labels. Semi-supervised learning is an active area of research that aims to reduce the quantity of ground truth labels required for training. It is aimed at common practical scenarios in which only a small subset of a large dataset has corresponding ground truth labels. Unsupervised domain adaptation is a closely related problem in which one attempts to transfer knowledge gained from a labeled source dataset to a distinct unlabeled target dataset, within the constraint that the objective (e.g.digit classification) must remain the same. Domain adaptation offers the potential to train a model using labeled synthetic data -that is often abundantly available -and unlabeled real data. The scale of the problem can be seen in the VisDA-17 domain adaptation challenge images shown in FIG3. We will present our winning solution in Section 4.2.Recent work BID28 ) has demonstrated the effectiveness of self-ensembling with random image augmentations to achieve state of the art performance in semi-supervised learning benchmarks. We have developed the approach proposed by BID28 to work in a domain adaptation scenario. We will show that this can achieve excellent in specific small image domain adaptation benchmarks. More challenging scenarios, notably MNIST → SVHN and the VisDA-17 domain adaptation challenge required further modifications. To this end, we developed confidence thresholding and class balancing that allowed us to achieve state of the art in a variety of benchmarks, with some of our coming close to those achieved by traditional supervised learning. Our approach is sufficiently flexble to be applicable to a variety of network architectures, both randomly initialized and pre-trained. Our paper is organised as follows; in Section 2 we will discuss related work that provides context and forms the basis of our technique; our approach is described in Section 3 with our experiments and in Section 4; and finally we present our in Section 5. In this section we will cover self-ensembling based semi-supervised methods that form the basis of our approach and domain adaptation techniques to which our work can be compared. Recent work based on methods related to self-ensembling have achieved excellent in semisupervised learning scenarious. A neural network is trained to make consistent predictions for unsupervised samples under different augmentation BID23, dropout and noise conditions or through the use of adversarial training. We will focus in particular on the self-ensembling based approaches of BID13 and BID28 as they form the basis of our approach. BID13 present two models; their Π-model and their temporal model. The Π-model passes each unlabeled sample through a classifier twice, each time with different dropout, noise and image translation parameters. Their unsupervised loss is the mean of the squared difference in class probability predictions ing from the two presentations of each sample. Their temporal model maintains a per-sample moving average of the historical network predictions and encourages subsequent predictions to be consistent with the average. Their approach achieved state of the art in the SVHN and CIFAR-10 semi-supervised classification benchmarks. BID28 further improved on the temporal model of BID13 by using an exponential moving average of the network weights rather than of the class predictions. Their approach uses two networks; a student network and a teacher network, where the student is trained using gradient descent and the weigthts of the teacher are the exponential moving average of those of the student. The unsupervised loss used to train the student is the mean square difference between the predictions of the student and the teacher, under different dropout, noise and image translation parameters. There is a rich body of literature tackling the problem of domain adaptation. We focus on deep learning based methods as these are most relevant to our work. Auto-encoders are unsupervised neural network models that reconstruct their input samples by first encoding them into a latent space and then decoding and reconstructing them. BID5 describe an auto-encoder model that is trained to reconstruct samples from both the source and target domains, while a classifier is trained to predict labels from domain invariant features present in the latent representation using source domain labels. BID0 reckognised that samples from disparate domains have distinct domain specific characteristics that must be represented in the latent representation to support effective reconstruction. They developed a split model that separates the latent representation into shared domain invariant features and private features specific to the source and target domains. Their classifier operates on the domain invariant features only. BID4 propose a bifurcated classifier that splits into label classification and domain classification branches after common feature extraction layers. A gradient reversal layer is placed between the common feature extraction layers and the domain classification branch; while the domain classification layers attempt to determine which domain a sample came from the gradient reversal operation encourages the feature extraction layers to confuse the domain classifier by extracting domain invariant features. An alternative and simpler implementation described in their appendix minimises the label cross-entropy loss in the feature and label classification layers, minimises the domain cross-entropy in the domain classification layers but maximises it in the feature layers. The model of runs along similar lines but uses separate feature extraction sub-networks for source and domain samples and train the model in two distinct stages. BID21 ) use tri-training (Zhou & Li; feature extraction layers are used to drive three classifier sub-networks. The first two are trained on samples from the source domain, while a weight similarity penalty encourages them to learn different weights. Pseudo-labels generated for target domain samples by these source domain classifiers are used to train the final classifier to operate on the target domain. Generative Adversarial Networks (GANs; BID6) are unsupervised models that consist of a generator network that is trained to generate samples that match the distribution of a dataset by fooling a discriminator network that is simultaneously trained to distinguish real samples from generates samples. Some GAN based models -such as that of BID24 -use a GAN to help learn a domain invariant embedding for samples. Many GAN based domain adaptation approaches use a generator that transforms samples from one domain to another. BID1 propose a GAN that adapts synthetic images to better match the characteristics of real images. Their generator takes a synthetic image and noise vector as input and produces an adapted image. They train a classifier to predict annotations for source and adapted samples alonside the GAN, while encouraing the generator to preserve aspects of the image important for annotation. The model of BID25 consists of a refiner network (in the place of a generator) and discriminator that have a limited receptive field, limiting their model to making local changes while preserving ground truth annotations. The use of refined simulated images with corresponding ground truths ed in improved performance in gaze and hand pose estimation. BID20 present a bi-directional GAN composed of two generators that transform samples from the source to the target domain and vice versa. They transform labelled source samples to the target domain using one generator and back to the source domain with the other and encourage the network to learn label class consistency. This work bears similarities to CycleGAN, by.A number of domain adaptation models maximise domain confusion by minimising the difference between the distributions of features extracted from source and target domains. minimises the difference between the feature covariance matrices for a mini-batch of samples from the source and target domains. and BID16 minimise the Maximum Mean Discrepancy metric BID7. described adaptive batch normalization, a variant of batch normalization BID11 ) that learns separate batch normalization statistics for the source and target domains in a two-pass process, establishing new state-of-the-art . In the first pass standard supervised learning is used to train a classifier for samples from the source domain. In the second pass, normalization statistics for target domain samples are computed for each batch normalization layer in the network, leaving the network weights as they are. Our model builds upon the mean teacher semi-supervised learning model of BID28, which we will describe. Subsequently we will present our modifications that enable domain adaptation. The structure of the mean teacher model of BID28 -also discussed in section 2.1 -is shown in FIG1. The student network is trained using gradient descent, while the weights of the teacher network are an exponential moving average of those of the student. During training each input sample x i is passed through both the student and teacher networks, generating predicted class probability vectors z i (student) andz i (teacher). Different dropout, noise and image translation parameters are used for the student and teacher pathways. During each training iteration a mini-batch of samples is drawn from the dataset, consisting of both labeled and unlabeled samples. The training loss is the sum of a supervised and an unsupervised component. The supervised loss is cross-entropy loss computed using z i (student prediction). It is masked to 0 for unlabeled samples for which no ground truth is available. The unsupervised component is the self-ensembling loss. It penalises the difference in class predictions between student (z i) and teacher (z i) networks for the same input sample. It is computed using the mean squared difference between the class probability predictions z i andz i.Laine & Aila FORMULA0 and BID28 found that it was necessary to apply a timedependent weighting to the unsupervised loss during training in order to prevent the network from getting stuck in a degenerate solution that gives poor classification performance. They used a function that follows a Gaussian curve from 0 to 1 during the first 80 epochs. In the following subsections we will describe our contributions in detail along with the motivations for introducing them. We minimise the same loss as in BID28; we apply cross-entropy loss to labeled source samples and unsupervised self-ensembling loss to target samples. As in BID28, self-ensembling loss is computed as the mean-squared difference between predictions produced by the student (z T i) and teacher (z T i) networks with different augmentation, dropout and noise parameters. The models of BID28 and of BID13 were designed for semisupervised learning problems in which a subset of the samples in a single dataset have ground truth labels. During training both models mix labeled and unlabeled samples together in a minibatch. In contrast, unsupervised domain adaptation problems use two distinct datasets with different underlying distributions; labeled source and unlabeled target. Our variant of the mean teacher model -shown in FIG1 -has separate source (X Si) and target (X T i) paths. Inspired by the work of, we process mini-batches from the source and target datasets separately (per iteration) so that batch normalization uses different normalization statistics for each domain during training.1. We do not use the approach of as-is, as they handle the source and target datasets separtely in two distinct training phases, where our approach must train using both simultaneously. We also do not maintain separate exponential moving averages of the means and variances for each dataset for use at test time. As seen in the'MT+TF' row of Table 1, the model described thus far achieves state of the art in 5 out of 8 small image benchmarks. The MNIST → SVHN, STL → CIFAR-10 and Syn-digits → SVHN benchmarks however require additional modifications to achieve good performance. We found that replacing the Gaussian ramp-up factor that scales the unsupervised loss with confidence thresholding stabilized training in more challenging domain adaptation scenarios. For each unlabeled sample x T i the teacher network produces the predicted class probabilty vectorz T ijwhere j is the class index drawn from the set of classes C -from which we compute the confidencẽ f T i = max j∈C (z T ij); the predicted probability of the predicted class of the sample. Iff T i is below the confidence threshold (a parameter search found 0.968 to be an effective value for small image benchmarks), the self-ensembling loss for the sample x i is masked to 0.Our working hypothesis is that confidence thresholding acts as a filter, shifting the balance in favour of the student learning correct labels from the teacher. While high network prediction confidence does not guarantee correctness there is a positive correlation. Given the tolerance to incorrect labels reported by BID13, we believe that the higher signal-to-noise ratio underlies the success of this component of our approach. The use of confidence thresholding achieves a state of the art in the STL → CIFAR-10 and Syn-digits → SVHN benchmarks, as seen in the'MT+CT+TF' row of Table 1. While confidence thresholding can in very slight reductions in performance (see the MNIST ↔ USPS and SVHN → MNIST ), its ability to stabilise training in challenging scenarios leads us to recommend it as a replacement for the time-dependent Gaussian ramp-up used in BID13. We explored the effect of three data augmentation schemes in our small image benchmarks (section 4.1). Our minimal scheme (that should be applicable in non-visual domains) consists of Gaussian noise (with σ = 0.1) added to the pixel values. The standard scheme (indicated by 'TF' in Table 1) was used by BID13 and adds translations in the interval [−2, 2] and horizontal flips for the CIFAR-10 ↔ STL experiments. The affine scheme (indicated by 'TFA') adds random affine transformations defined by the matrix in, where N (0, 0.1) denotes a real value drawn from a normal distribution with mean 0 and standard deviation 0.1. DISPLAYFORM0 The use of translations and horizontal flips has a significant impact in a number of our benchmarks. It is necessary in order to outpace prior art in the MNIST ↔ USPS and SVHN → MNIST benchmarks and improves performance in the CIFAR-10 ↔ STL benchmarks. The use of affine augmentation can improve performance in experiments involving digit and traffic sign recognition datasets, as seen in the'MT+CT+TFA' row of Table 1. In contrast it can impair performance when used with photographic datasets, as seen in the the STL → CIFAR-10 experiment. It also impaired performance in the VisDA-17 experiment (section 4.2). With the adaptations made so far the challenging MNIST → SVHN benchmark remains undefeated due to training instabilities. During training we noticed that the error rate on the SVHN test set decreases at first, then rises and reaches high values before training completes. We diagnosed the problem by recording the predictions for the SVHN target domain samples after each epoch. The rise in error rate correlated with the predictions evolving toward a condition in which most samples are predicted as belonging to the'1' class; the most populous class in the SVHN dataset. We hypothesize that the class imbalance in the SVHN dataset caused the unsupervised loss to reinforce the'1' class more often than the others, ing in the network settling in a degenerate local minimum. Rather than distinguish between digit classes as intended it seperated MNIST from SVHN samples and assigned the latter to the'1' class. We addressed this problem by introducing a class balance loss term that penalises the network for making predictions that exhibit large class imbalance. For each target domain mini-batch we compute the mean of the predicted sample class probabilities over the sample dimension, ing in the mini-batch mean per-class probability. The loss is computed as the binary cross entropy between the mean class probability vector and a uniform probability vector. We balance the strength of the class balance loss with that of the self-ensembling loss by multiplying the class balance loss by the average of the confidence threshold mask (e.g. if 75% of samples in a mini-batch pass the confidence threshold, then the class balance loss is multiplied by 0.75). We would like to note the similarity between our class balance loss and the entropy maximisation loss in the IMSAT clustering model of BID10; IMSAT employs entropy maximisation to encourage uniform cluster sizes and entropy minimisation to encourage unambiguous cluster assignments. Our implementation was developed using PyTorch (Chintala et al.) and is publically available at http://github.com/Britefury/self-ensemble-visual-domain-adapt. Our can be seen in Table 1. The'train on source' and'train on target' report the target domain performance of supervised training on the source and target domains. They represent the exepected baseline and best achievable . The'Specific aug.' experiments used data augmentation specific to the MNIST → SVHN adaptation path that is discussed further down. The small datasets and data preparation procedures are described in Appendix A. Our training procedure is described in Appendix B and our network architectures are described in Appendix D. The same network architectures and augmentation parameters were used for domain adaptation experiments and the supervised baselines discussed above. It is worth noting that only the training sets of the small image datasets were used during training; the test sets used for reporting scores only. MNIST ↔ USPS (see FIG2 . MNIST and USPS are both greyscale hand-written digit datasets. In both adaptation directions our approach not only demonstrates a significant improvement over prior art but nearly achieves the performance of supervised learning using the target domain ground truths. The strong performance of the base mean teacher model can be attributed to the similarity of the datasets to one another. It is worth noting that data augmentation allows our 'train on source' baseline to outpace prior domain adaptation methods. CIFAR-10 ↔ STL (see FIG2). CIFAR-10 and STL are both 10-class image datasets, although we removed one class from each (see Appendix A.2). We obtained strong performance in the STL → CIFAR-10 path, but only by using confidence thresholding. The CIFAR-10 → STL are more interesting; the'train on source' baseline performance outperforms that of a network trained on the STL target domain, most likely due to the small size of the STL training set. Our self-ensembling outpace both the baseline performance and the'theoretical maximum' of a network trained Table 1: Small image benchmark classification accuracy; each is presented as mean ± standard deviation, computed from 5 independent runs. The abbreviations for components of our models are as follows: MT = mean teacher, CT = confidence thresholding, TF = translation and horizontal flip augmentation, TFA = translation, horizontal flip and affine augmentation, * indicates minimal augmentation.on the target domain, lending further evidence to the view of BID23 and BID13 that self-ensembling acts as an effective regulariser. Syn-Digits → SVHN (see FIG2 . The Syn-Digits dataset is a synthetic dataset designed by BID4 to be used as a source dataset in domain adaptation experiments with SVHN as the target dataset. Other approaches have achieved good scores on this benchmark, beating the baseline by a significant margin. Our improves on them, reducing the error rate from 6.9% to 2.9%; even slightly outpacing the 'train on target' 3.4% error rate achieved using supervised learning. Syn-Signs → GTSRB (see FIG2). Syn-Signs is another synthetic dataset designed by BID4 to target the 43-class GTSRB (German Traffic Signs Recognition Benchmark; BID26) dataset. Our approach halved the best error rate of competing approaches. Once again, our approaches slightly outpaces the'train on target' supervised learning upper bound. SVHN → MNIST (see FIG2). Google's SVHN (Street View House Numbers) is a colour digits dataset of house number plates. Our approach significantly outpaces other techniques and achieves an accuracy close to that of supervised learning. MNIST → SVHN (see FIG2). This adaptation path is somewhat more challenging as MNIST digits are greyscale and uniform in terms of size, aspect ratio and intensity range, in contrast to the variably sized colour digits present in SVHN. As a consequence, adapting from MNIST to SVHN required additional work. Class balancing loss was necessary to ensure training stability and additional experiment specific data augmentation was required to achieve good accuracy. The use of translations and affine augmentation (see section 3.3) in an accuracy score of 37%. Significant improvements ed from additional augmentation in the form of random intensity flips (negative image), and random intensity scales and offsets drawn from the intervals [0.25, 1.5] and [−0.5, 0.5] respectively. These hyper-parameters were selected in order to augment MNIST samples to match the intensity variations present in SVHN, as illustrated in FIG2. With these additional modifications, we achieve a that significantly outperforms prior art and nearly achieves the accuracy of a supervised classifier trained on the target dataset. We found that applying these additional augmentations to the source MNIST dataset only yielded good ; applying them to the target SVHN dataset as well yielded a small improvement but was not essential. It should also be noted that this augmentation scheme raises the performance of the'train on source' baseline to just above that of much of the prior art. The VisDA-2017 image classification challenge is a 12-class domain adaptation problem consisting of three datasets: a training set consisting of 3D renderings of sketchup models, and validation and test sets consisting of real images (see FIG3) drawn from the and YouTube BoundingBoxes BID19 datasets respectively. The objective is to learn from labeled computer generated images and correctly predict the class of real images. Ground truth labels were made available for the training and validation sets only; test set scores were computed by a server operated by the competition organisers. While the algorithm is that presented above, we base our network on the pretrained ResNet-152 BID9 ) network provided by PyTorch (Chintala et al.), rather than using a randomly initialised network as before. The final 1000-class classification layer is removed and replaced with two fullyconnected layers; the first has 512 units with a ReLU non-linearity while the final layer has 12 units with a softmax non-linearity. Results from our original competition submissions and newer using two data augmentation schemes are presented in Table 2. Our reduced augmentation scheme consists of random crops, random horizontal flips and random uniform scaling. It is very similar to scheme used for ImageNet image classification in BID9. Our competition configuration includes additional augmentation that was specifically designed for the VisDA dataset, although we subsequently found that it makes little difference. Our hyper-parameters and competition data augmentation scheme are described in Appendix C.1. It is worth noting that we applied test time augmentation (we averaged predictions form 16 differently augmented images) to achieve our competition . We present resuts with and without test time augmentation in Table 2. Our VisDA competition test set score is also the of ensembling the predictions of 5 different networks. We have presented an effective domain adaptation algorithm that has achieved state of the art in a number of benchmarks and has achieved accuracies that are almost on par with traditional supervised learning on digit recognition benchmarks targeting the MNIST and SVHN datasets. The Table 2: VisDA-17 performance, presented as mean ± std-dev of 5 independent runs. Full are presented in TAB6 in Appendix C. ing networks will exhibit strong performance on samples from both the source and target domains. Our approach is sufficiently flexible to be usable for a variety of network architectures, including those based on randomly initialised and pre-trained networks. stated that the self-ensembling methods presented by -on which our algorithm is based -operate by label propagation. This view is supported by our , in particular our MNIST → SVHN experiment. The latter requires additional intensity augmentation in order to sufficiently align the dataset distributions, after which good quality label predictions are propagated throughout the target dataset. In cases where data augmentation is insufficient to align the dataset distributions, a pre-trained network may be used to bridge the gap, as in our solution to the VisDA-17 challenge. This leads us to conclude that effective domain adaptation can be achieved by first aligning the distributions of the source and target datasets -the focus of much prior art in the field -and then refining their correspondance; a task to which self-ensembling is well suited. The datasets used in this paper are described in Some of the experiments that involved datasets described in TAB3 required additional data preparation in order to match the resolution and format of the input samples and match the classification target. These additional steps will now be described. MNIST ↔ USPS The USPS images were up-scaled using bilinear interpolation from 16 × 16 to 28 × 28 resolution to match that of MNIST.CIFAR-10 ↔ STL CIFAR-10 and STL are both 10-class image datasets. The STL images were down-scaled to 32 × 32 resolution to match that of CIFAR-10. The'frog' class in CIFAR-10 and the'monkey' class in STL were removed as they have no equivalent in the other dataset, ing in a 9-class problem with 10% less samples in each dataset. Syn-Signs → GTSRB GTSRB is composed of images that vary in size and come with annotations that provide region of interest (bounding box around the sign) and ground truth classification. We extracted the region of interest from each image and scaled them to a resolution of 40 × 40 to match those of Syn-Signs. MNIST ↔ SVHN The MNIST images were padded to 32 × 32 resolution and converted to RGB by replicating the greyscale channel into the three RGB channels to match the format of SVHN.B SMALL IMAGE EXPERIMENT TRAINING B.1 TRAINING PROCEDURE Our networks were trained for 300 epochs. We used the Adam BID12 gradient descent algorithm with a learning rate of 0.001. We trained using mini-batches composed of 256 samples, except in the Syn-digits → SVHN and Syn-signs → GTSRB experiments where we used 128 in order to reduce memory usage. The self-ensembling loss was weighted by a factor of 3 and the class balancing loss was weighted by 0.005. Our teacher network weights t i were updated so as to be an exponential moving average of those of the student s i using the formula t i = αt i−1 + (1 − α)s i, with a value of 0.99 for α. A complete pass over the target dataset was considered to be one epoch in all experiments except the MNIST → USPS and CIFAR-10 → STL experiments due to the small size of the target datasets, in which case one epoch was considered to be a pass over the larger soure dataset. We found that using the proportion of samples that passed the confidence threshold can be used to drive early stopping BID18 ). The final score was the target test set performance at the epoch at which the highest confidence threshold pass rate was obtained. C VISDA-17 C.1 HYPER-PARAMETERS Our training procedure was the same as that used in the small image experiments, except that we used 160 × 160 images, a batch size of 56 (reduced from 64 to fit within the memory of an nVidia 1080-Ti), a self-ensembling weight of 10 (instead of 3), a confidence threshold of 0.9 (instead of 0.968) and a class balancing weight of 0.01. We used the Adam BID12 gradient descent algorithm with a learning rate of 10 −5 for the final two randomly initialized layers and 10 DISPLAYFORM0 for the pre-trained layers. The first convolutional layer and the first group of convolutional layers (with 64 feature channels) of the pre-trained ResNet were left unmodified during training. Reduced data augmentation:• scale image so that its smallest dimension is 176 pixels, then randomly crop a 160 × 160 section from the scaled image • No random affine transformations as they increase confusion between the car and truck classes in the validation set • random uniform scaling in the range [0.75, 1.333]• horizontal flipping Competition data augmentation adds the following in addition to the above:• random intensity/brightness scaling in the range [0.75, 1.333]• random rotations, normally distributed with a standard deviation of 0.2π• random desaturation in which the colours in an image are randomly desaturated to greyscale by a factor between 0% and 100% • rotations in colour space, around a randomly chosen axes with a standard deviation of 0.05π• random offset in colour space, after standardisation using parameters specified by 10 × 10 × 192 Dropout, 50%10 × 10 × 192 Conv 3 × 3 × 384, pad 1, batch norm 10 × 10 × 384 Conv 3 × 3 × 384, pad 1, batch norm 10 × 10 × 384 Conv 3 × 3 × 384, pad 1, batch norm 10 × 10 × 384 Max-pool, 2x25 × 5 × 384 Dropout, 50%5 × 5 × 384 Global pooling layer 1 × 1 × 384 Fully connected, 43 units, softmax 43 Table 8: Syn-signs → GTSRB architecture

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Self-ensembling based algorithm for visual domain adaptation, state of the art results, won VisDA-2017 image classification domain adaptation challenge.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.