Question: Represent the scientific article to find a short TLDR

Question: Unsupervised monocular depth estimation has made great progress after deep learning is involved. Training with binocular stereo images is considered as a good option as the data can be easily obtained. However, the depth or disparity prediction show poor performance for the object boundaries. The main reason is related to the handling of occlusion areas during the training. In this paper, we propose a novel method to overcome this issue. Exploiting disparity maps property, we generate an occlusion mask to block the back-propagation of the occlusion areas during image warping. We also design new networks with flipped stereo images to induce the networks to learn occluded boundaries. It shows that our method achieves clearer boundaries and better evaluation on KITTI driving dataset and Virtual KITTI dataset. Monocular depth estimation becomes an active research topic as deep learning is applied in various computer vision tasks. It has many applications, from navigation through to scene understanding. A single traditional camera can be a cheaper alternative to the expensive LIDAR sensor for automotive cars if accurate estimation can be achieved. Meanwhile, single camera simplifies the design of depth estimation solution which can be adopted quite widely at a low cost. One straight-forward way to train deep depth estimation models is to use ground truth depth images as the supervision signals BID1. However, supervised deep learning method is eager for massive data with ground truth. Collecting large datasets with ground truth depth in varied real scenarios is challenge and expensive. Instead, training using stereo images without depth label is an alternative option. BID7 proposed a method to exploit the left-right consistency of stereo images to tackle the monocular depth estimation, which achieved quite promising . However, the depth predicted by their method has blurred boundaries. The issue is mainly due to the occlusions during the image warping. Though it can be alleviated in some extent with proper post processing, the fundamental problem is not well addressed. In this paper, we propose a new method to overcome the blurred boundaries when using stereo pairs to train the monocular depth model. An example is illustrated in FIG0. During the image warping, we generate an occlusion mask using the disparity map to block the inappropriate back-propagation gradients for occlusion areas. However, the mask only cannot guarantee clear boundaries as there is no constrain for the masked areas. Then we design new networks to fully exploit the information of stereo images. With flipped stereo pairs, the network is induced to learn clear boundaries for occlusion areas. Our method provides a solution to the fundamental learning difficulty of occluded areas introduced by image warping in depth estimation. Empirical evaluation on KITTI driving dataset BID6 ) and Virtual KITTI dataset BID4 ) demonstrates the effectiveness of our approach. Moreover, we find the depth label of KITTI 2015 is usually very sparse near the object boundaries, which is not very sensitive to evaluate the clearness of boundaries. Large amounts of multi-view based approaches have been proposed such as stereo matching BID22 ), difference view point BID3 ) or temporal sequence BID18 ). Here we briefly review work based on single view depth estimation which is usually harder since reasoning depth from monocular colored image is an ill-posed problem. The most intuition way is treating monocular depth estimation problem as a supervised problem by taking RGB images as inputs and Lidar depth points as ground truth. BID21 proposed a method known as Make3d which breaks the image into homogeneous patches. For each small homogeneous patch, Saxena et al used a Markov Random Field (MRF) to infer a set of plane parameters that capture both the 3D location and 3D orientation. However, this approach has a hard time capturing thin structures since the predictions are made locally. BID1 first exploited CNN in a coarse to fine manner. BID13 BID12 proposed a network jointly explore the capacity of deep CNN and continuous conditional random field (CRF). BID10 incorporated semantic segmentations in to single-view depth estimation task since they are closely tied to the property of perspective geometry. BID11 proposed a residual network with up-sampling module using fully convolutional architecture. Also, reverse Huber loss was introduced. However, large amount of high-quality labelled data is needed, which is hard to require in practice. To overcome the lack of high quality labelled data, several semi-supervised and fully unsupervised methods have been proposed. BID2 first proposed a view synthesis based method called DeepStereo which generates new view image from nearby image. BID24 FIG2 proposed a method which generates the right image through probability distribution over all the possible disparities for each pixel. BID5 first proposed a warp based method by aligning reconstructed image with the ground truth left image as described in 3.1. However, their loss is not fully differentiable. BID7 improve this methods by introducing a novel loss. BID19 extended the network into two separate channel with 6 or 12 losses which improves the . Based on Garg et al, BID9 proposed a semi-supervised methods that exploited both the sparse Lidar points as supervision and stereo pairs as unsupervion signals. The semisupervised method was further improved by BID15, they decoupled the monocular depth prediction problem into two procedure, a view synthesis procedure followed by stereo matching. Recently, several work using only monocular temporal sequence comes out which enables more training data such as video sequence on YouTube. BID26 proposed a network that predicts depth and camera pose separately. Using the predicted depth and camera pose, relative temporal image can be reconstructed by image warping with which final loss can be constructed. BID16 performed a novel 3D loss that enforces consistency of the estimated 3D point clouds and ego-motion across consecutive frames and combined it with 2D photometric loss. proposed a differentiable implementation of Direct Visual Odometry (DVO) and a novel depth normalization strategy. However, all the temporal sequence based training meet the same problem of object motion. This problem can be alleviated by including stereo pairs during training known as trinocular training BID25 BID17 BID8 ). However, all these warp based methods have the difficulty of learning occluded area which would infect the . Godard et al achieves state of art of unsupervised monocular depth estimation with only stereo images. Follow Godard et al, we proposed monocular depth estimation network with novel mask methods which can be trained end-to-end and without ground-truth label. Our method is superior to Godard et al in quality with clearer boundaries, especially on dense evaluation dataset such as virtual-KITTI BID4 ). In general, our goal is to learn a network that can predict a pixel-wise dense depth map from single colored image(DISPLAYFORM0 However, all supervised methods have a hard time with acquiring large dense labelled data in real scenarios. Thus, several unsupervised methods have been proposed to overcome the obstacle. Among these methods, training by image reconstruction using rectified stereo pairs became more and more popular currently due to its high accuracy and easy accessibility of training data. BID5, Left-right consistency network proposed by Godard et al, Our refined network without shared parameters and our refined network with shared parameters First proposed by BID5, the monodepth estimation network takes left image (I l) as input and outputs the disparity aligned with the right image (d r) which can be used to sample from the left image (I l) to reconstruct the right image (Ĩ r) during training. Thus, image reconstruction loss can be constructed between the reconstructed right image(Ĩ r) and original right image(I r). When testing, only one colored image are required, the predicted disparity can be converted to depth simply using depth = b * f /disparity, where b and f are given baseline and camera focal length respectively. It is worth to mention that disparities (d l and d r) are a scalar per pixel as the images are rectified. The network was further improved by BID7 by introducing left-right consistency loss and refined encoder-decoder network. Given the input left image, the network predicts both the left and right disparities simultaneously which enables constructing the left-right consistency. This consistency restriction leads to more accurate and less artifacts. Also, fully differentiable backward bilinear sampling was used to reconstruct the left image which makes the model easier to optimize. With better network architecture and better loss, Godard et al achieved the state of art of unsupervised monodepth estimation only with rectified stereo pairs, and even outperforming supervised methods. Their network architectures are shown in FIG1.However, there still are unsatisfactory artifacts at the occlusion boundaries showing blurred ramps on the left side of image and of the occluders. Even with the post-processing step which weighted sums the flipped disparity of the flipped input image and the disparity of the input image, the blurred ramps are still visible near the objects, especially for those near the cameras with higher disparities as illustrated in FIG0. Though common backward warping using bilinear sampler is fully differentiable which enables us to train the model end to end, some undesirable duplicates and artifacts are introduced during the warping process because of occlusions according to BID14. In FIG2, we use SYTHIA dataset BID20 ), a synthesized virtual driving dataset to illustrate. Though ground truth disparities are used for warping, there still exists obvious duplicates and even a huge black region on the left of the reconstructed image(Ĩ l) because of the occlusion. If those inevitable artifacts and duplicates are back propagated during the training process, unwanted high losses will be introduced forcing the network learn to blur in those regions, as the blurriness(disparity ramps) in the occluded regions will make the reconstructed image show stretched patterns (Fig 7) which are more similar to original ground truth image compared to duplicates and large black regions FIG2 ).. We used a warping mask which is generated automatically from disparities to block the back-propagation of those artifacts. The final output is shown in (e), where the white regions are masked. In order to block the back propagation process of those warping induced artifacts, we designed an algorithm which can mask the occlusion region automatically. In backward bilinear sampling process, a disparity map was used to sample from the source image. The intuition here is that, if any pixel in the source image has never been sampled during the bilinear sampling process, this pixel should only be visible in the source image and should be masked when reconstructing this source image later on. Thus, the mask methods takes, say, left disparity as input while generating the mask of reconstructed right image and vice versa. The pseudo code is shown in Algorithm 1 and the mask schematic diagram is shown in Fig 4. However, the mask method alone cannot guarantee the clearness since no constrain is added in masked region. Specifically, though the masks block the back propagation of duplicates and artifacts induced by warping, they also block the further learning process of those regions. Once the disparities start to blur, hardly can we correct the network back to clearness. To solve this problem, we refined the network architecture and introduced a flip-over training scheme. Though the mask blocks the process of further learning of the blurred regions, we find that the blurred side are definite which can be exploited to reactive the learning process. For example, when the network is only trained on left images and takes corresponding right images as ground truth, the disparity ramps (blurred regions) will only appear on the left side of the occluders. So, if we randomly flipped the input images horizontally, the disparity ramps will still appear on the left side. When flipped the output disparities back for warping, the blurred regions will appear on the right side where no mask is added. Examples are shown in the last column of Fig 7. Thus, those blurred regions will make distortions when reconstructing images, and back propagate despite the 3 for x,y in grid(h,w) do DISPLAYFORM0 13 end 14 end 15 return mask l, mask r mask. Also, because the flip-over scheme is performed randomly, any blurriness on the definite side will receive punishment on average which restrict the prediction to be clear. It is worth to mention that flip-over scheme will not affect masking process and we still use masks to block the back propagation of duplicates. The flip-over schematic diagram is shown in Fig. 4.However, the predicted disparities of the right branch won't make any sense if we take the flipped left image as input and are totally mismatched with the ground truths as is shown in Fig 4. As a , we delete the predicting branch of the right disparity and add another encoder-decoder network which takes right images as input and predicts right disparities as is shown in FIG1 This doubled encoder-decoder network enables us to preform left-right consistency loss at the cost of doubling the training parameters. However, it won't slow down the test speed since only one branch of encoder-decoder network is used when testing. We also tried another network architecture with shared the encoder-decoder network which achieves comparable as non-shared network while halves the training time. More details can be found in 6.1 We use similar training loss as BID7. The losses are performed on four different scale and are finally summed as the total loss. C = 4 s=1 C s. For each scale, three different losses including appearance match loss(C ap), disparity smoothness loss(C ds) and LR consistency loss(C lr) are performs as follows. DISPLAYFORM0 Intuitively, appearance matching loss measures photometric error between the reconstructed image(Ĩ l ij)) and the ground truth image(I l ij), which is defined as the weighted sum of L1 and SSIM shown as follows, DISPLAYFORM1 Disparity smoothness loss aims to force the smoothness of predicted disparities through Figure 4: Schematic diagram. The orange squares refer to an object such as pedestrians or vehicles. The blue and black squares refer to blurred disparity region and masked region respectively. Left: Naive masking process. Warping mask(black) is overlap with blurred region(blue). As a , masks will block the learning of those region. Right: Flip-over training scheme. We flipped the left images(f (I l)) as input and flipped the output disparities(f (d(f (I l)))) back to reconstruct the image(Ĩ l). Different from the diagram shown in left, the blurred region will switch to the opposite side to avoid the mask. As a , losses will be introduced leading to clear boundaries. We can also observed that the right branch (flipped predicted right disparity of flipped left image(f (d(I r)))) is totally mismatch with the ground truth right image, so we delete this branch and add another encoder-decoder branch as shown in FIG1. DISPLAYFORM2 Finally, L1 penalty is added to constrict the left-right consistency, which further reduce the artifacts, DISPLAYFORM3 Our network is based on BID7 and is implemented in Tensorflow (Abadi et al.). The network contains encoder network based on VGG16 or Resnet50 and decoder network with 7 upconvolutional layers, 7 merging layers and 4 disparity prediction layers. With 7 skip connections, the network can better handles features at different scales. Different from Godard et al, we modified the channel number of disparity prediction layers to predict only one disparity instead of two. Also, we tuned the default hyper parameters α and α ds. The rest are the same with Godard et al with α ap = 1, α lr = 1. The learning rate λ remains 10 −4 for the first 30 epoch and halves every 10 epoch when trained for 50 epoch. We also tried batch norm but might lead to unstable . Data augmentation is performed on the fly similar as Godard et al including flipping and color shifting. We trained our model on rectified stereo image pairs in KITTI and Cityscapes dataset and evaluated mainly on KITTI split BID6 ), Eigen split BID1 ) and virtual-KITTI datasets BID4 ). Also, we find the problem of KITTI 2015 evaluation such as sparsity and man-made defects which infects our evaluation . When testing on dense datasets like virtual-KITTI, the superiority becomes more obvious. For comparison, we use the same split as BID7 high quality disparity image issued by official KITTI dataset. It is worth mentioned that though these disparity images are of better quality than projected velodyne laser point and with CAD models inserted in cars, most part of ground truth are extremely sparse, especially in terms of occluded regions. As the red boxes in the FIG3 shown, there is some blank space on the left side of image and occluded regions which is not involved in the evaluation. Thus, those blank space just covers the shortcomings the disparity ramps on the occluded regions. As a , our shows less superiority over Godard et al on KITTI stereo 2015 dataset. As shown in TAB2, we use the metrics from BID1 and D1-all metrics from KITTI. Our model achieves comparable with Godard et al when both trained on KITTI dataset only for 50 epoch, and superior when both trained longer for 100 epoch. BID6 ). K means the model is only trained on KITTI dataset, while CS + K means the model is trained on Cityscapes dataset and then finetune on KITTI dataset. Also, pp means post processing step, and more details about post processing can be found in 6.3. For a fair comparison, we train the network proposed by Godard et al and our non-shared network both to 50 and 100 epoch, and find that our network has larger improvement and better performance than BID7 when trained longer to 100 epoch. We guess that our network requires longer time to converge. We choose different hyperparameters from Godard et al for our network. Though it is unfair to evaluation on sparse KITTI dataset, our still outperforms that of Godard et al. Similar to Godard et al, we use the test split of 697 images proposed by BID1. Each image contains 3D points captured by velodyne laser which was used to generate ground truth depth. We keep the same 22600 stereo pairs for training. According to Godard et al, all evaluation are done under the Garg crop BID5 ) except for Eigen for a fair comparison. We also present the uncropped which our models' superiority become more obvious, because it will crop the disparity ramps on the left side for evaluation which boosting Godard et al's . Also, ground truths are captured by velodyne Lidar thus rather sparse which would reduce our superiority over Godard et al. The evaluation is in TAB4 4.3 VIRTUAL-KITTITo prevent sparsity induced inaccurate , we evaluate models on the Virtual-KITTI dataset BID6 ). Virtual-KITTI dataset contains 50 monocular videos generated from five different virtual worlds in urban settings under different weather conditions with corresponding pixel-level ground truth depth. With the same resolution, scenes and camera parameters as KITTI 2015, Virtual-KITTI dataset can be implement naively when testing. Since KITTI 2015 dataset we used for training does not cover those weathers, only 2126 labbeled images without weather condi-Method Supervised Dataset Abs Rel Sq Rel RMSE RMSE log δ < 1.25 δ < 1.25 2 δ < 1.25 3 BID1 BID1. K is KITTI and CS is cityscapes dataset for training. We use the evaluation provided in BID7. For a fair comparison, we use the crop the same as BID5 except for BID1 and apply the same hyper-parameters as Godard et al on our model. Besides, we set the maximun evaluation depth to 50 meters(cap 50m) in the second row which is the same as BID5, while others remain 80 meters. Also, we compared uncropped with Godard et al, on which the superiority become more obvious.tion are used for evaluation. The evaluation is shown in TAB5: Evaluation on virtual-KITTI. Once our ground truth become dense, our model out performance other models for its sharp and clear boundaries on predicted depth map. Even we crop the black edges of Godard et al (the left 10% of the whole disparity map) in the second row, our model is still superior to BID7 (the state of art unsupervised monocular depth estimation using left and right information only). We evaluate on 3 different methods: naive, crop the black edge, both crop the edge and set the maximun evaluation depth to 50 meters In this work, we present an occlusion mask and filp-over training scheme to enable effective learning of object boundaries when using image warping. With our new network, our model achieves state of art using only stereo images. Moreover, as warping based image reconstruction is commonly used in depth estimation problem, our method provides a solution to the fundamental difficulty of occluded areas introduced by image warping. In the future, our method can be incorporated with more accurate network trained on trinocular data (temporal Stereo sequence) such as BID25, BID17 and BID8, which would further boost the accuracy.6 SUPPLEMENTARY MATERIALS The shared weight network is similar to the non-shared weight network except for the shared weight. The architecture is shown in 2. Fortunately, the shared weight model naturally handled the problem. Because the blurriness always occur in the occluded region, say, disparity ramps (burriness) in left disparity maps will only appear on the left side of the occluders, the network cannot distinguish which side the input image belongs to and which side to blur when trained on the left and right image simultaneously with shared weights. Thus, any wrong blurriness (blurriness on the wrong side, say, right blurriness in the left disparity maps) will lead to punishment in raising reconstructed loss since no mask was adding on the other side (wrong side, say, right side). As a , the network achieves comparable as non-shared network while halves the training time. However, shared method is slightly inferior to non-shared network but still out performance Godard FIG0, Godard et al used non-linear weight. Because our model do not predict black edge, so we simply average them as our post-processing. Here we use the network by BID7 to illustrate how flip-over scheme could solve the problem of back-propagate, so the predicted disparities are rather blurred. In Fig 7, from left to right: left disparity(d l), reconstructed left image(Ĩ l), corresponding mask(mask l), ground truth(I l), flipped disparity of flipped input imagef (d(f (I l))). To minimize the reconstruction error, the network predicts blurred forcing the network to sample from the , which leads to more similar stretched patterns instead of high loss duplicates as shown in FIG2. However, the blurred regions are aligned with the mask, which freezes the fine-tuning process of blurred regions. To solve the problem, we randomly flip the colored image as input(Input = f (I l)), then flipped the output back as final disparities for warping(f (d(f (I l)))). As shown in the last column, the blurred regions are no longer aligned with the masks which enables further learning process. As the training process goes on, the boundaries will become clearer as shown in FIG0. We also use schematic diagram to illustrate in Fig. 4 Figure 7: Example of blurred and flip-over scheme.

This paper propose a mask method which solves the previous blurred results of unsupervised monocular depth estimation caused by occlusion

We propose a novel way to incorporate conditional image information into the discriminator of GANs using feature fusion that can be used for structured prediction tasks.


Question: Represent the scientific article to find a short TLDR!

Question: Graph classification is currently dominated by graph kernels, which, while powerful, suffer some significant limitations. Convolutional Neural Networks (CNNs) offer a very appealing alternative. However, processing graphs with CNNs is not trivial. To address this challenge, many sophisticated extensions of CNNs have recently been proposed. In this paper, we reverse the problem: rather than proposing yet another graph CNN model, we introduce a novel way to represent graphs as multi-channel image-like structures that allows them to be handled by vanilla 2D CNNs. Despite its simplicity, our method proves very competitive to state-of-the-art graph kernels and graph CNNs, and outperforms them by a wide margin on some datasets. It is also preferable to graph kernels in terms of time complexity. Code and data are publicly available. Graphs, or networks, are rich, flexible, and universal structures that can accurately represent the interaction among the components of many natural and human-made complex systems BID11. For instance, graphs have been used to describe and analyze the interplay among proteins within cells and the internal structure of proteins themselves BID3, the organization of the brain BID6, the World Wide Web BID26, textual documents BID22, and information propagation through a population BID18. Consequently, graph mining has attracted significant attention in machine learning and artificial intelligence, and is still today a very active area of investigation. A central graph mining task is that of graph classification (not to be mistaken with node classification). Its applications range from determining whether a protein is an enzyme or not in bioinformatics to categorizing documents in NLP and analyzing a social network. Graph classification is the task of interest in this study. The state-of-the-art in graph classification is currently dominated by a family of methods referred to as graph kernels. Graph kernels compute the similarity between two graphs as the sum of the pairwise similarities between some of their substructures, and then pass the similarity matrix computed on the entire dataset to a kernel-based supervised algorithm such as the Support Vector Machine BID8 to learn soft classification rules. Graph kernels mainly vary based on the substructures they use, which include random walks BID12, shortest paths, and subgraphs BID29, to cite only a few. While graph kernels have been very successful, they suffer significant limitations:1. High time complexity. This problem is threefold: first, populating the kernel matrix requires computing the similarity between every two graphs in the training set (say of size N), which amounts to N (N −1) /2 operations. The cost of training therefore increases much more rapidly than the size of the dataset. Second, computing the similarity between a pair of graphs (i.e., performing a single operation) is itself polynomial in the number of nodes. For instance, the time complexity of the shortest path graph kernel is O(|V 1 | 2 |V 2 | 2) for two graphs (V 1, V 2), where |V i | is the number of nodes in graph V i. Processing large graphs can thus become prohibitive, which is a serious limitation as big networks abound in practice. Finally, finding the support vectors is O(N 2) when the C parameter of the SVM is small and O(N 3) when it gets large BID4, which can again pose a problem on big datasets.2. Disjoint feature and rule learning. With graph kernels, the computation of the similarity matrix and the learning of the classification rules are two independent steps. In other words, the features are fixed and not optimized for the task.3. Graph comparison is based on small independent substructures. As a , graph kernels focus on local properties of graphs, ignoring their global structure . They also underestimate the similarity between graphs and suffer unnecessarily high complexity (due to the explosion of the feature space), as substructures are considered to be orthogonal dimensions BID35. We propose a very simple approach to turn a graph into a multi-channel image-like structure suitable to be processed by a traditional 2D CNN. It can be broken down into 3 steps, summarized in FIG0. The first step involves embedding the nodes of the graph. The embedding space is compressed with PCA at step 2. We then repeatedly extract 2D slices from the compressed space and compute a 2D histogram for each slice. The "image" representation of the graph is finally given by the stack of its 2D histograms (each histogram making for a channel). Note that the dimensionality of the final representation of a graph does not depend on its number of nodes or edges. Big and small graphs are represented by images of the same size. Our method addresses the limitations of graph kernels in the following ways:1. High time complexity. By converting all graphs in a given dataset to representations of the same dimensionality, and by using a classical 2D CNN architecture for processing those graph representations, our method offers constant time complexity at the instance level, and linear time complexity at the dataset level. Moreover, state-of-the-art node embeddings can be obtained for a given graph in linear time (w.r.t. the size of the graph), for instance with node2vec BID14. 2. Disjoint feature and rule learning. Thanks to the 2D CNN classifier, features are learned directly from the raw data during training to optimize performance on the downstream task. 3. Graph comparison is based on small independent substructures. Our approach capitalizes on state-of-the-art graph node embedding techniques that capture both local and global properties of graphs. In addition, we remove the need for handcrafted features. Convolutional Neural Networks (CNNs) are feedforward neural networks specifically designed to work on regular grids. A regular grid is the d-dimensional Euclidean space discretized by parallelo-topes (rectangles for d = 2, cuboids for d = 3, etc.). In CNNs, each neuron in a given layer receives input from a neighborhood of the neurons in the previous layer BID20. Those neighborhoods, or local receptive fields, allow CNNs to compose higher-level features from lower-level features, and thus to capture patterns of increasing complexity in a hierarchical way. Regular grids satisfy the spatial dependence 2 property, which is the fundamental premise on which local receptive fields and hierarchical composition of features in CNNs hold. Traditionally, a graph G(V, E) is encoded as its adjacency matrix A or Laplacian matrix L. A is a square matrix of dimensionality |V | × |V |, symmetric in the case of undirected graphs, whose (i, j) th entry A i,j is equal to the weight of the edge e i,j between nodes v i and v j, if such an edge exists, or to 0 otherwise. On the other hand, the Laplacian matrix L is equal to D − A, where D is the diagonal degree matrix. One could initially consider passing one of those structures as input to a 2D CNN. However, unlike in images, where close pixels are more strongly correlated than distant pixels, adjacency and Laplacian matrices are not associated with spatial dimensions and the notion of Euclidean distance, and thus do not satisfy the spatial dependence property. As will be detailed next, we capitalize on graph node embeddings to address this issue. Step 1: Graph node embeddings. There is local correlation in the node embedding space. In that space, the Euclidean distance between two points is meaningful: it is inversely proportional to the similarity of the two nodes they represent. For instance, two neighboring points in the embedding space might be associated with two nodes very distant in the graph, but playing the same structural role (e.g., of flow control), belonging to the same community, or sharing some other common property. Step 2: Alignment and compression with PCA. As state-of-the-art node embedding techniques (such as node2vec) are neural, they are stochastic. Dimensions are thus recycled from run to run, meaning that a given dimension will not be associated with the same latent concepts across graphs, or across several runs on the same graph. Therefore, to ensure that the embeddings of all the graphs in the collection are comparable, we apply PCA and retain the first d D principal components (where D is the dimensionality of the original node embedding space). PCA also serves an information maximization (compression) purpose. Compression is desirable as it greatly reduces the shape of the tensors fed to the CNN (for reasons that will become clear in what follows), and thus complexity, at the expense of a negligible loss in information. Step 3: Computing and stacking 2D histograms. We finally repeatedly extract 2D slices from the d-dimensional PCA node embedding space, and turn those planes into regular grids by discretizing them into a finite, fixed number of equally-sized bins, where the value associated with each bin is the count of the number of nodes falling into that bin. In other words, we represent a graph as a stack of d/2 2D histograms of its (compressed) node embeddings 3. As illustrated in FIG1, the first histogram is computed from the coordinates of the nodes in the plane made of the first two principal directions, the second histogram from directions 3 and 4, and so forth. Note that using adjacent and following PCA dimensions is an arbitrary choice. It ensures at least that channels are sorted according to the amount of information they contain. Using computer vision vocabulary, bins can be viewed as pixels, and the 2D slices of the embedding space as channels. However, in our case, instead of having 3 channels (R,G,B) like with color images, we have d/2 of them. That is, each pixel (each bin) is associated with a vector of size d/2, whose entries are the counts of the nodes falling into that bin in the corresponding 2D slice of the embedding space. Finally, the resolution of the image is determined by the number of bins of the histograms, which is constant for a given dataset across all dimensions and channels.4 EXPERIMENTAL SETUP We implemented a variant of LeNet-5 with which we reached 99.45% accuracy on the MNIST handwritten digit classification dataset. As illustrated in FIG2 for an input of shape, this simple architecture deploys four convolutional-pooling layers (each repeated twice) in parallel, with respective region sizes of 3, 4, 5 and 6, followed by two fully-connected layers. Dropout BID31 ) is employed for regularization at every hidden layer. The activations are ReLU functions (in that, our model differs from LeNet-5), except for the ultimate layer, which uses a softmax to output a probability distribution over classes. For the convolutionpooling block, we employ 64 filters at the first level, and as the signal is halved through the max pooling layer, the number of filters in the subsequent convolutional layer is increased to 96 to compensate for the loss in resolution. We used node2vec BID14, which applies the very fast Skip-Gram language model BID23 to truncated biased random walks performed on the graph. node2vec scales linearly with the number of nodes in the network. We leveraged the igraph module BID9 and the publicly available high performance C++ implementation 4 of node2vec. We conducted experiments on the publicly available 5 datasets from BID35 which we briefly describe in what follows, and in Table 1. In all datasets, graphs are unweighted, undirected, with unlabeled nodes, and the task is to predict the class they belong to. Classes are mutually exclusive. REDDIT-B, REDDIT-5K, and IMDB-B are perfectly balanced, whereas REDDIT-12K and COLLAB feature a maximum class imbalance ratio of 1:5 and 1:3.4, respectively. Note that graphs with less than 10 nodes were removed because having 5 channels requires at least a 10-dimensional embedding space, which is impossible to obtain with less than 10 nodes. However, this represented only a few graphs per dataset, for some datasets. In all REDDIT datasets, a graph corresponds to a thread where nodes represent users, and there is an edge between two nodes if one of the two users responded to a comment from the other user. More precisely, graphs in REDDIT-B are labeled according to whether they were constructed from Q&A communities or discussion communities, and REDDIT-5K and REDDIT-12K respectively feature graphs from 5 and 11 forums dedicated to specific topics (those forums are known as "subreddits").In COLLAB, graphs are hop-1 neighborhoods of researchers from a scientific collaboration network (two researchers are linked if they co-authored a paper), and are labeled according to the subfield of Physics the corresponding researcher belongs to. Finally, the IMDB-B dataset features hop-1 neighborhoods of actors and actresses selected from two movie collaboration networks corresponding to specific genres (action and romance), in which two actors are linked if they starred in the same movie. Graphs are labeled according to the genre they were sampled from. We refer the reader to the original paper for more information about the datasets. We compared our model to two state-of-the-art graph kernels, the graphlet kernel BID29 ) and the Weisfeiler-Lehman (WL) kernel BID30. The graphlet kernel computes the similarity between two graphs as the cosine of their count vectors. These vectors encode how many subgraphs of size up to a certain threshold can be found in each graph (each entry is an occurrence count). We sampled 2000 graphlets of size up to 6 from each graph. The WL kernel is actually a framework that operates on top of any graph kernel accepting node labels and boosts its performance by using the relabeling procedure of the WL test of isomorphism. More precisely, following the computation of the kernel value between the two graphs, vertex labels are updated based on the labels of their neighbors. This two-step process repeats for a certain number of iterations. The final kernel value is the sum of the values at each iteration. Since our graphs have unlabeled nodes, we set the degrees of the nodes as their labels. Furthermore, we used the WL framework with the subtree graph kernel BID12, as it is very efficient with this kernel BID30. For both baselines, we used a C-SVM classifier 6 BID27. The C parameter of the SVM and the number of iterations in WL were jointly optimized on a 90-10 % partition of the training set of each fold by searching the grid (10 −4, 10 4, len = 10); (2, 7, step = 1). All experiments involved 10-fold cross validation where each fold was repeated 3 times. We used Xavier initialization BID13, a batch size of 32, and for regularization, a dropout rate of 0.3 and early stopping with a patience of 5 epochs. The categorical cross-entropy loss was optimized with Adam BID16 In our initial experiments involving spectral embeddings, the coordinates of any node in any dimension belonged to the [−1, 1] range, due to the eigenvectors being unit-normed. Furthermore, inspired by the MNIST images which are 28 × 28 in size, and on which we initially tested our 2D CNN architecture, we decided to learn 2D histograms featuring 28 bins in each direction. This gave us a resolution of 28 /(1−(−1)), that is, 14 pixels per unit (or simply 14:1). As it was giving good , we stuck to similar values in our final experiments making use of neural embeddings. With the p and q parameters of node2vec held constant and equal to 1, we conducted a search on the coarse grid; to get more insights about the impact of resolution and number of channels (respectively). On a given dataset, image size is calculated as the range |max(coordinates) − min(coordinates)| multiplied by the resolution, where "coordinates" are the node loadings flattened across all dimensions of the embedding space. For instance, on COLLAB with a resolution of 9:1, image size is equal to 37 × 37, since |2.78 − (−1.33)| × 9 ≈ 37. Optimal values for each dataset are summarized in Table 2.With the best resolution and number of channels, we then tuned the return and in-out parameters p and q of node2vec. Those parameters respectively bias the random walks towards exploring larger areas of the graph or staying in local neighborhoods, allowing the embeddings to encode a similarity that interpolates between structural equivalence (two nodes acting as, e.g., flow controllers, are close to each other) and homophily (two nodes belonging to the same community are close to each other). We Table 2: Best resolution, number of channels, and (p, q) for each dataset. The classification accuracy of our approach and the baselines we implemented are reported in TAB2. Even though we did not re-implement those models, we also display for comparison purposes the performance reported in BID35 (Deep Graph Kernels) and BID24 (Graph CNN, PSCN k = 10), since the experimental setting is the same. Our approach shows significantly better than all baselines on the REDDIT-12K and REDDIT-B datasets, with large improvements of 6.81 and 2.82 in accuracy over the best performing competitor, respectively. We also reach best performance on the REDDIT-5K dataset, with an improvement in accuracy of 1.34 over the best performing baseline. However, the difference is not statistically significant. Finally, on the IMDB-B dataset, we get third place, very close (≤ 1.2) to the top performers, and again without statistically significant differences. Actually, the only dataset on which a baseline proved significantly better than our approach is COLLAB (WL graph kernel). Even if not directly comparable, we report in TAB3 kernel matrix computation time for the two graph kernel baselines, along with the time required by our 2D CNN model to perform one pass over the entire training set, i.e., the time per epoch. With respects to time complexity, our method is superior to graph kernels on several counts: first, unlike graph kernels, the time required by the 2D CNN to process one training example is constant (all images for a given dataset have the same size), while computing the kernel value for a pair of graphs depends on their size (polynomial in the number of nodes). It is true that a prerequisite for our approach is an embedding for all the graphs in the dataset, but node2vec scales linearly with the number of nodes in the graph. Therefore, on big graphs, our method is still usable, while graph kernels may not be. Also, node2vec is easily parallelizable over the collection, so one can take advantage of multi-core CPUs to considerably speed up the process. Second, with a 2D CNN, the time necessary to go through the entire training set only increases linearly with the size of the set, while populating the kernel matrix is quadratic, and finding the support vectors is then again at least quadratic. This means that on large datasets, our approach is also preferable to graph kernels. Examples of 2D CNN architectures much more complex than ours applied to millions of images in reasonable time abound in the recent computer vision literature. Processing such big datasets with graph kernels would simply be intractable. In addition, not only do neural models allow processing big datasets, but their performance also tends to significantly improve in the presence of large quantities of training data. Motivated by the outstanding performance recently reached by Convolutional Neural Networks (CNNs) in computer vision, e.g. BID34 BID19, many research efforts have been devoted to generalizing CNNs to graphs. Indeed, CNNs offer a very appealing alternative to kernel-based methods. The parsimony achieved through weight sharing makes them very efficient, their time complexity is constant for each training example and linear with respect to the size of the dataset, and the extra expressiveness they bring might translate to significant accuracy gains. However, since convolution and pooling are natively defined for regular, low-dimensional grids such as images (2D Euclidean space discretized by rectangles), generalizing CNNs to graphs, which are irregular, non-Euclidean objects, is far from trivial. Possible solutions that can be found in the literature fall into two broad categories: spatial and spectral techniques BID5. Spectral approaches BID10 BID17 invoke the convolution theorem from signal processing theory to perform graph convolutions as pointwise multiplications in the Fourier domain of the graph. The basis used to send the graph to the Fourier domain is given by the SVD decomposition of the Laplacian matrix of the graph, whose eigenvalues can be viewed as "frequencies". By contrast, spatial methods BID24 BID33 operate directly on the graph structure. For instance, in BID24, the algorithm first determines the sequence of nodes for which neighborhood graphs (of equal size) are created. To serve as receptive fields, the neighborhood graphs are then normalized, i.e., mapped to a vector space with a linear order, in which nodes with similar structural roles in the neighborhood graphs are close to each other. Normalization is the central step, and is performed via a labeling procedure. A 1D CNN architecture is finally applied to the receptive fields. While the aforementioned sophisticated frameworks have made great strides, we showed in this paper that graphs can also be processed by vanilla 2D CNN architectures. This is made possible by the novel graph representation we introduce, which encodes graphs as stacks of 2D histograms of their node embeddings. Compared to the more complex approaches that involve different fundamental architectural and/or operational modifications, the main advantage of our method is its simplicity. Crucially, we show that this simplicity can be obtained without giving up accuracy: we indeed outperform graph CNN baselines by a wide margin on some datasets, and are very close elsewhere. Replacing the raw counts by the empirical joint probability density function, either by normalizing the histograms, or with a Kernel Density Estimate, significantly deteriorated performance. This suggests that keeping the absolute values of the counts is important, which makes sense, because some categories might be associated with larger or smaller graphs, on average. Therefore, preventing the model from using size information is likely to decrease accuracy. We also observed that increasing the number of channels to more than 5 does not yield better (which makes sense, as channels contain less and less information), but that reducing this number improves performance in some cases, probably because it plays a regularization role. The main contribution of our study is a novel method for representing graphs as multi-channel image-like structures from their node embeddings, that allows them to be processed by 2D CNNs. How the embeddings are computed, and which 2D CNN architecture is used, does not matter. We hold this flexibility to be a major strength. First, the embedding-agnostic nature of our method means that it can be seamlessly extended to directed, weighted, or labeled graphs with continuous or categorical node/edge attributes, simply by using an embedding algorithm that accepts such graphs, e.g., BID21. The independence of our approach with respect to the image classification model used is another advantage. Here, we employed a vanilla 2D CNN architecture as it was offering an excellent trade-off between accuracy and simplicity, but more recent models, such as the one of BID15, may yield even better . Above all, performance should improve as graph node embedding algorithms and CNN architectures for images improve in the future. Even though are very good out-of-the-box in most cases, finding an embedding algorithm that works well, or the right combination of parameters for a given dataset, can require some efforts. For instance, on COLLAB, we hypothesize that our are inferior to that observed on the other datasets because optimizing p and q for COLLAB may require more than a coarse grid search, or because node2vec may not be well-suited to very dense graphs such as the ones found in COLLAB. The main contribution of this paper is to show that CNN architectures designed for images can be used for graph processing in a completely off-the-shelf manner, simply by representing graphs as stacks of two-dimensional histograms of their node embeddings. Despite the simplicity of our approach, indicate that it is very competitive to state-of-the-art graph kernels and graph CNN models, sometimes outperforming them by a wide margin. Furthermore, these good were obtained with limited parameter tuning and by using a basic 2D CNN model. From a time complexity perspective, our approach is preferable to graph kernels too, allowing to process larger datasets featuring bigger graphs.

Represent the TLDR for finding the corresponding scientific article

We introduce a novel way to represent graphs as multi-channel image-like structures that allows them to be handled by vanilla 2D CNNs.

We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.