# embeddings_queue

created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc29-6db4-11ee-8c2a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-18", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "Instead, if a pixel falls within the bounding box of a detection, we assign the pixel to\nthat instance with a probability proportional to the rescored detection (obtained from the\nprobability of the latent Yvariable after inference) and the semantic segmentation con\ufb01dence\nfor that class, as shown in Equation 4\nPr(vi=k) =/braceleftBigg1\nZ(Y,Q)Qi(lk)Pr(Yk=1)ifi\u2208Bk\n0 otherwise .(4)\nHere, viis a multinomial random variable indicating the \u201cidenti\ufb01ed instance\u201d at pixel iand\ntakes on labels from D,Qi(l)is the output of the initial category-level segmentation stage\nof our network and denotes the probability of pixel itaking the label l\u2208L, and Z(Y,Q)\nnormalises the probability in case there are multiple bounding boxes overlapping the same\npixel. For this formulation to be valid, we also add another detection, d0, denoting background pixels that are not overlapped by any detection. This then acts as the unary potentials\nof another CRF with the energy:\nE(v) =\u2211\ni\u03c8U\ni(vi)+\u2211\ni<j\u03c8P"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc2a-6db4-11ee-a84c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-19", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "of another CRF with the energy:\nE(v) =\u2211\ni\u03c8U\ni(vi)+\u2211\ni<j\u03c8P\ni,j(vi,vj) \u03c8U\ni(vi) =\u2212lnPr(vi). (5)\nThis Instance CRF contains unary, \u03c8U\ni(vi), and densely-connected pairwise terms, \u03c8P\ni,j(vi,vj),\nencouraging appearance and spatial consistency [18]. These priors are valid in the case of\ninstance segmentation as well. We then perform mean \ufb01eld inference on this \ufb01nal Instance\nCRF to get our \ufb01nal output. Note that this \ufb01nal, pairwise CRF is dynamic \u2013 the D+1 labels\nARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 7\nthat it deals with are different for every image, where Dis equal to the number of object detections obtained after non-maximal suppression. This differs with the CRFs considered in\nsemantic segmentation literature which always have a \ufb01xed number of labels for all images"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc2b-6db4-11ee-84e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-20", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "semantic segmentation literature which always have a \ufb01xed number of labels for all images\nin the dataset [17, 18, 35, 37]. Zheng et al. [37] showed how the iterative mean \ufb01eld inference algorithm can be unrolled into a series of differentiable operations and be implemented\nas a recurrent neural network. We employ a similar method so that our dynamic CRF is fully\ndifferentiable and can thus be incorporated into a neural network. This is achieved by using\nCRF weights that are not class-speci\ufb01c, and can thus generalise to any number of labels per\nimage. A label here refers to an instance associated with a detection and has no semantic\nmeaning. For example, in Fig. 1, label \u201c1\u201d was associated with a dining table, whilst in Fig.\n2 it is a sheep. It therefore does not make sense to have class-speci\ufb01c weights in any case.\nThis approach to instance segmentation is bottom-up, in the sense that we \ufb01rst perform\ncategory-level segmentation of each pixel before predicting instances. However, the fact\nthat we used object detection cues in performing our semantic segmentation helps us to\nsubsequently reason about instances. Moreover, the fact that we used results from both\nsemantic segmentation and object detection makes sense since instance segmentation can be"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc2c-6db4-11ee-abb7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-21", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "subsequently reason about instances. Moreover, the fact that we used results from both\nsemantic segmentation and object detection makes sense since instance segmentation can be\nregarded as being at the intersection of these two more common problems.\n4 Experimental Results and Analysis\nWe \ufb01rst describe the dataset and our experimental setup (\u00a74.1), before examining the ef\ufb01cacy\nof our detection potentials (\u00a74.2) and comparing our approach with other recent instance\nsegmentation methods (\u00a74.3).\n4.1 Experimental Setup\nFollowing previous work, we evaluate our instance segmentation performance on the PASCAL VOC 2012 [8] validation set which comprises of 1449 images with high-quality annotations. There is no test server for instance segmentation to evaluate on the test set. We\nuse the standard APrmeasure for evaluation [13], which computes the mean average precision under different Intersection over Union (IoU) scores between the predicted and ground\ntruth segmentations (rather than IoU between bounding boxes, as in object detection). In object detection, a predicted bounding box is considered correct if the IoU between it and the\nground-truth bounding box is greater than 0.5. Here, we consider different overlap thresholds\nbetween the predicted and ground-truth segments, since higher overlap thresholds require the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc2d-6db4-11ee-9c7f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-22", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "ground-truth bounding box is greater than 0.5. Here, we consider different overlap thresholds\nbetween the predicted and ground-truth segments, since higher overlap thresholds require the\nsegmentations to be more precise and are therefore better indicators of segmentation performance. We also quote the APr\nvolmeasure, which is the mean APrscore for overlap thresholds\nranging from 0 .1 to 0 .9 in increments of 0 .1 [13]. The detailed annotations in the VOC 2012\ndataset allows us to reliably evaluate the AProf our method at high overlap thresholds.\nWe \ufb01rst trained a network for semantic segmentation, in a similar manner to [27] and\n[37]. Although the trained models from these two authors have been made publicly available,\nwe could not use them since they had been trained on part of the VOC validation set. We\n\ufb01rst trained a fully convolutional network [27] (\ufb01netuned from VGG-16 trained on ImageNet\n[32]) using VOC 2012 training data, augmented with images from the Semantic Boundaries\nDataset [13] which do not overlap with the VOC Validation set, and the Microsoft COCO\ndataset [26]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc2e-6db4-11ee-a874-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-23", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "Dataset [13] which do not overlap with the VOC Validation set, and the Microsoft COCO\ndataset [26].\nTo this pretrained network, we added the Higher Order CRF, and \ufb01netuned with a learning rate of 10\u221211using only VOC 2012 data which is \ufb01nely annotated. The learning rate is\n8 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION\nTable 1: Comparison of instance segmentation performance against baselines on the VOC\n2012 Validation Set of 1449 images. We report the APrmeasure at different IoU thresholds.\nMethod APrat 0.5 APrat 0.6 APrat 0.7 APr\nvol\nBaseline without detection potentials 54.6 48.5 41.8 50.0\nBaseline with detection potentials,\nbutYvariables ignored57.5 51.6 44.5 52.4\nFull system with detection potentials,\nand recalibrated detection scores from\nYvariables58.3 52.4 45.4 53.1\nlow since the loss was not normalised by the number of pixels in the training image. When\ntraining our Higher Order CRF, we use the publicly available Faster-RCNN object detection\nframework [28]. The semantic segmentation performance on the VOC validation set is a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc2f-6db4-11ee-ba0e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-24", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "training our Higher Order CRF, we use the publicly available Faster-RCNN object detection\nframework [28]. The semantic segmentation performance on the VOC validation set is a\nmean IoU of 73.4% when using only pairwise potentials in the CRF. This rises to 75.3%\nwhen using detection potentials as well. The authors of [2] observed a similar increase when\nusing detection potentials.\n4.2 Effect of detection potentials\nTo analyse the effect of our detection potentials, we evaluate our system by disabling the\nhigher order potentials in our Semantic Segmentation CRF. Because our detection potentials\nare disabled, we can no longer use the output of the latent Yvariables as our recalibrated\ndetection score, and use the original detector\u2019s con\ufb01dence score instead. As shown in Table\n1, the result is that the APrat 0.5 is 3.7% lower and the APr\nvolis 3.1% lower.\nThe detection potentials improve instance segmentation performance as they improve\nboth our initial semantic segmentation, and also recalibrate the detector\u2019s initial scores. To\ndecouple these two effects, we then include detection potentials in our Segmentation CRF,\nbut ignore the latent Yvariables output by the CRF and use the detector\u2019s original con\ufb01dence"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc30-6db4-11ee-a3c5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-25", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "but ignore the latent Yvariables output by the CRF and use the detector\u2019s original con\ufb01dence\nscore instead as the input to the instance segmentation network. The second row of Table\n1 shows that this second baseline has an APrat 0.5 and an APr\nvolthat are respectively 0.8%\nand 0.7% lower than our \ufb01nal method. The difference between the \ufb01nal method and this\nsecond baseline is due to the score recalibration performed by the detection potentials. The\ntwo baselines differ in performance since the second baseline uses a semantic segmentation\n(mean IoU of 75.3%) which is better than the other (73.4%) due to its use of detection\npotentials. The large gap in their overall instance segmentation performance emphasises\nthe importance of a good initial semantic segmentation for our method, and underlines the\nbene\ufb01ts of a bottom-up approach to instance segmentation.\nFrom the difference in performance between the baseline that does not fully make use of\ndetection potentials (Row 1 of Table 1), and our \ufb01nal method (Row 3 of Table 1), we can\nconclude that our higher order detection potentials are an integral part of our system.\n4.3 Comparison to other current methods"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc31-6db4-11ee-8327-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-26", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "conclude that our higher order detection potentials are an integral part of our system.\n4.3 Comparison to other current methods\nIn Table 2, we compare our methods to other approaches that have been evaluated on the\nsame dataset and reported APrresults at multiple IoU thresholds. The SDS method of [13]\nwas evaluated on the VOC Validation set by [4]. Our approach signi\ufb01cantly outperforms\nARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 9\nTable 2: Comparison of instance segmentation performance to other methods on the VOC\n2012 Validation Set. We report the APrmeasure at \ufb01ve different IoU thresholds.\nMethod APrat 0.5 APrat 0.6 APrat 0.7 APrat 0.8 APrat 0.9 APr\nvol\nSDS [13] 43.8 34.5 21.3 8.7 0.9 Chen et al. [4] 46.3 38.2 27.0 13.5 2.6 PFN [24] 58.7 51.3 42.5 31.2 15.7 52.3\nOurs 58.3 52.4 45.4 34.9 20.1 53.1\nboth [13] and [4] which initially perform object detection, and re\ufb01ne these detections into an"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc32-6db4-11ee-8319-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-27", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "both [13] and [4] which initially perform object detection, and re\ufb01ne these detections into an\ninstance segmentation. In fact, even our baselines in Table 1 outperform them. The model\nof [24] differs from the other two works in that the \ufb01rst stage of their pipeline is to also\nperform a category-level segmentation of the image. However, their method does not make\nuse of any information from object detectors, and our approach performs better, particularly\nat high IoU thresholds where the difference in APrat an IoU threshold of 0.9 is 4.4%. This\nindicates that our instance segmentations are more precise. The fact that our APr\nvolis only\n0.8% greater, despite our APrbeing signi\ufb01cantly larger at high thresholds suggests that the\nmethod of [24] has a higher APrat lower thresholds (below 0.5, which were not reported by\n[24]). This indicates that their method identi\ufb01es more instances than ours, but the fact that\ntheir APrat higher thresholds is lower than ours also suggests that their segmentations are\nnot as accurate.\nSome success and failure cases of our method are shown in Figure 3. Our method has\ntrouble differentiating instances which are occluded and also very visually similar to each"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc33-6db4-11ee-8ef8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-28", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "Some success and failure cases of our method are shown in Figure 3. Our method has\ntrouble differentiating instances which are occluded and also very visually similar to each\nother (Fig. 3, last two rows). The supplementary material includes a more detailed results\ntable, and more examples of success and failure cases.\n5 Conclusion\nWe have presented a simple method that effectively leverages state-of-the-art semantic segmentation and object detection networks to perform the less widely studied task of instance\nsegmentation. Our approach begins with a bottom-up semantic segmentation of the input\nimage, but is able to reason about instances since the \ufb01nal CRF layer in our semantic segmentation network incorporates information from an object detector in the form of a higherorder, detection potential. Our method produces state-of-the-art instance segmentation results, achieving considerable improvements over existing work at high APrthresholds. Our\n\ufb01nal Instance CRF \u2013 which is dynamic in its number of labels \u2013 is fully differentiable. This\nmeans that our neural network can be dynamically instantiated for every input image and\ntrained end-to-end.\nAcknowledgements\nWe thank Bernardino Romera-Paredes, Stuart Golodetz and Shuai Zheng for insightful discussions and feedback. This work was supported by the EPSRC, Clarendon Fund, ERC"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc34-6db4-11ee-8fc8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-29", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "We thank Bernardino Romera-Paredes, Stuart Golodetz and Shuai Zheng for insightful discussions and feedback. This work was supported by the EPSRC, Clarendon Fund, ERC\ngrant ERC-2012-AdG 321162-HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and EPSRC/MURI grant EP/N019474/1.\n10 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION\nInput image Object Segmentation Instance Segmentation Ground truth\nFigure 3: Some results of our system. Detector outputs are overlayed on the input image. Top\nrow: An example where our method performs well, handling occlusions and false-positive\ndetections. Second row: Our method performs well on this easy example which has no\noccluding objects of the same class. The na\u00efve method described in \u00a73.3 would have produced the same result. Third row: A failure case where the system is not able to differentiate\nvisually similar instances. Bottom row: Heavily occluded objects are dif\ufb01cult to identify.\nReferences\n[1] Pablo Arbelaez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc35-6db4-11ee-b2f7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-30", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "References\n[1] Pablo Arbelaez, Jordi Pont-Tuset, Jon Barron, Ferran Marques, and Jitendra Malik.\nMultiscale combinatorial grouping. In CVPR , pages 328\u2013335. IEEE, 2014.\n[2] Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, and Philip H. S. Torr. Higher order\nconditional random \ufb01elds in deep neural networks. In ECCV , 2016.\n[3] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L\nYuille. Semantic image segmentation with deep convolutional nets and fully connected\ncrfs. ICLR , 2015.\n[4] Yi-Ting Chen, Xiaokai Liu, and Ming-Hsuan Yang. Multi-instance object segmentation\nwith occlusion handling. In CVPR , pages 3470\u20133478, 2015.\n[5] Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object\nand stuff segmentation. In CVPR , 2015.\n[6] Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise\nconvolutional networks for semantic segmentation. In ICCV , 2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc36-6db4-11ee-8b0b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-31", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "convolutional networks for semantic segmentation. In ICCV , 2015.\nARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION 11\n[7] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via\nmulti-task network cascades. In CVPR , 2016.\n[8] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew\nZisserman. The pascal visual object classes (voc) challenge. IJCV , 2010.\n[9] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively\ntrained, multiscale, deformable part model. In CVPR , pages 1\u20138. IEEE, 2008.\n[10] Spyros Gidaris and Nikos Komodakis. Object detection via a multi-region and semantic\nsegmentation-aware cnn model. In ICCV , pages 1134\u20131142, 2015.\n[11] Ross Girshick. Fast r-cnn. In ICCV , 2015.\n[12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc37-6db4-11ee-99ec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-32", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "[13] Bharath Hariharan, Pablo Arbel\u00e1ez, Ross Girshick, and Jitendra Malik. Simultaneous\ndetection and segmentation. In ECCV , pages 297\u2013312. Springer, 2014.\n[14] Bharath Hariharan, Pablo Arbel\u00e1ez, Ross Girshick, and Jitendra Malik. Hypercolumns\nfor object segmentation and \ufb01ne-grained localization. CVPR , 2015.\n[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in\ndeep convolutional networks for visual recognition. In ECCV , 2014.\n[16] Xuming He, Richard S Zemel, and Miguel \u00c1 Carreira-Perpi\u00f1\u00e1n. Multiscale conditional\nrandom \ufb01elds for image labeling. In CVPR , volume 2, pages II\u2013695. IEEE, 2004.\n[17] Pushmeet Kohli, Lubor Ladicky, and Philip H.S. Torr. Robust higher order potentials\nfor enforcing label consistency. IJCV , 82(3):302\u2013324, 2009.\n[18] P. Kr\u00e4henb\u00fchl and V . Koltun. Ef\ufb01cient inference in fully connected CRFs with Gaussian edge potentials. In NIPS , 2011."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc38-6db4-11ee-b04d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-33", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classi\ufb01cation with\ndeep convolutional neural networks. In NIPS , pages 1097\u20131105. 2012.\n[20] Lubor Ladicky, Chris Russell, Pushmeet Kohli, and Philip HS Torr. Graph cut based\ninference with co-occurrence statistics. In ECCV , pages 239\u2013253. 2010.\n[21] L\u2019ubor Ladick\u00fd, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip H. S. Torr.\nWhat, where and how many? combining object detectors and crfs. In ECCV , pages\n424\u2013437, 2010.\n[22] Yann LeCun, L\u00e9on Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278\u20132324,\n1998.\n[23] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Zequn Jie, Jiashi Feng, Liang Lin, and\nShuicheng Yan. Reversible recursive instance-level object segmentation. arXiv preprint\narXiv:1511.04517 , 2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc39-6db4-11ee-83df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-34", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "Shuicheng Yan. Reversible recursive instance-level object segmentation. arXiv preprint\narXiv:1511.04517 , 2015.\n12 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION\n[24] Xiaodan Liang, Yunchao Wei, Xiaohui Shen, Jianchao Yang, Liang Lin, and Shuicheng\nYan. Proposal-free network for instance-level object segmentation. arXiv preprint\narXiv:1509.02636 , 2015.\n[25] Guosheng Lin, Chunhua Shen, Ian Reid, et al. Ef\ufb01cient piecewise training of deep\nstructured models for semantic segmentation. In CVPR , 2016.\n[26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r, and C Lawrence Zitnick. Microsoft coco: Common objects in\ncontext. In ECCV , pages 740\u2013755. Springer, 2014.\n[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for\nsemantic segmentation. In CVPR , 2015.\n[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc3a-6db4-11ee-afc1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-35", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "[28] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards\nreal-time object detection with region proposal networks. In NIPS , 2015.\n[29] Bernardino Romera-Paredes and Philip HS Torr. Recurrent instance segmentation. In\nECCV , 2016.\n[30] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM TOG , 2004.\n[31] Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi. Textonboost for\nimage understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV , 2009.\n[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largescale image recognition. In ICLR , 2015.\n[33] Min Sun, Byung-soo Kim, Pushmeet Kohli, and Silvio Savarese. Relating things and\nstuff via object property interactions. PAMI , 36(7):1370\u20131383, 2014.\n[34] Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc3b-6db4-11ee-b564-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1609.02583-36", "title": "Bottom-up Instance Segmentation using Deep Higher-Order CRFs", "source": "http://arxiv.org/pdf/1609.02583", "start_index": 0, "chroma:document": "[34] Jasper RR Uijlings, Koen EA van de Sande, Theo Gevers, and Arnold WM Smeulders.\nSelective search for object recognition. IJCV , 104(2):154\u2013171, 2013.\n[35] Vibhav Vineet, Jonathan Warrell, and Philip HS Torr. Filter-based mean-\ufb01eld inference\nfor random \ufb01elds with higher-order terms and product label-spaces. IJCV , 2014.\n[36] Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the Scene as a Whole: Joint\nObject Detection, Scene Classi\ufb01cation and Semantic Segmentation. In CVPR , pages\n702\u2013709, 2012.\n[37] Shuai Zheng, Sadeep Jayasumana, Bernardino Romera-Paredes, Vibhav Vineet,\nZhizhong Su, Dalong Du, Chang Huang, and Philip Torr. Conditional random \ufb01elds as\nrecurrent neural networks. In ICCV , 2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc3c-6db4-11ee-8d47-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-0", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "Aggregated Residual Transformations for Deep Neural Networks\nSaining Xie1Ross Girshick2Piotr Doll \u00b4ar2Zhuowen Tu1Kaiming He2\n1UC San Diego2Facebook AI Research\nfs9xie,ztug@ucsd.edu frbg,pdollar,kaiminghe g@fb.com\nAbstract\nWe present a simple, highly modularized network architecture for image classi\ufb01cation. Our network is constructed\nby repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has\nonly a few hyper-parameters to set. This strategy exposes a\nnew dimension, which we call \u201ccardinality\u201d (the size of the\nset of transformations), as an essential factor in addition to\nthe dimensions of depth and width. On the ImageNet-1K\ndataset, we empirically show that even under the restricted\ncondition of maintaining complexity, increasing cardinality\nis able to improve classi\ufb01cation accuracy. Moreover, increasing cardinality is more effective than going deeper or\nwider when we increase the capacity. Our models, named\nResNeXt, are the foundations of our entry to the ILSVRC\n2016 classi\ufb01cation task in which we secured 2nd place."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc3d-6db4-11ee-ad37-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-1", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "ResNeXt, are the foundations of our entry to the ILSVRC\n2016 classi\ufb01cation task in which we secured 2nd place.\nWe further investigate ResNeXt on an ImageNet-5K set and\nthe COCO detection set, also showing better results than\nits ResNet counterpart. The code and models are publicly\navailable online1.\n1. Introduction\nResearch on visual recognition is undergoing a transition from \u201cfeature engineering\u201d to \u201cnetwork engineering\u201d\n[25, 24, 44, 34, 36, 38, 14]. In contrast to traditional handdesigned features ( e.g., SIFT [29] and HOG [5]), features\nlearned by neural networks from large-scale data [33] require minimal human involvement during training, and can\nbe transferred to a variety of recognition tasks [7, 10, 28].\nNevertheless, human effort has been shifted to designing\nbetter network architectures for learning representations.\nDesigning architectures becomes increasingly dif\ufb01cult\nwith the growing number of hyper-parameters (width2, \ufb01lter sizes, strides, etc.), especially when there are many layers. The VGG-nets [36] exhibit a simple yet effective strategy of constructing very deep networks: stacking build1https://github.com/facebookresearch/ResNeXt\n2Width refers to the number of channels in a layer."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc3e-6db4-11ee-9f7e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-2", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "2Width refers to the number of channels in a layer.\n256, 1x1, 4\n4, 3x3, 4\n4, 1x1, 256\n+256, 1x1, 4\n4, 3x3, 4\n4, 1x1, 256256, 1x1, 4\n4, 3x3, 4\n4, 1x1, 256....total 32\npaths256-d in\n+256, 1x1, 64\n64, 3x3, 64\n64, 1x1, 256\n+256-d in\n256-d out\n256-d outFigure 1. Left: A block of ResNet [14]. Right : A block of\nResNeXt with cardinality = 32 , with roughly the same complexity. A layer is shown as (# in channels, \ufb01lter size, # out channels).\ning blocks of the same shape. This strategy is inherited\nby ResNets [14] which stack modules of the same topology. This simple rule reduces the free choices of hyperparameters, and depth is exposed as an essential dimension\nin neural networks. Moreover, we argue that the simplicity"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc3f-6db4-11ee-b5e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-3", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "in neural networks. Moreover, we argue that the simplicity\nof this rule may reduce the risk of over-adapting the hyperparameters to a speci\ufb01c dataset. The robustness of VGGnets and ResNets has been proven by various visual recognition tasks [7, 10, 9, 28, 31, 14] and by non-visual tasks\ninvolving speech [42, 30] and language [4, 41, 20].\nUnlike VGG-nets, the family of Inception models [38,\n17, 39, 37] have demonstrated that carefully designed\ntopologies are able to achieve compelling accuracy with low\ntheoretical complexity. The Inception models have evolved\nover time [38, 39], but an important common property is\nasplit-transform-merge strategy. In an Inception module,\nthe input is split into a few lower-dimensional embeddings\n(by 1\u00021 convolutions), transformed by a set of specialized\n\ufb01lters (3\u00023, 5\u00025,etc.), and merged by concatenation. It\ncan be shown that the solution space of this architecture is a\nstrict subspace of the solution space of a single large layer\n(e.g., 5\u00025) operating on a high-dimensional embedding.\nThe split-transform-merge behavior of Inception modules"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc40-6db4-11ee-adce-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-4", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "(e.g., 5\u00025) operating on a high-dimensional embedding.\nThe split-transform-merge behavior of Inception modules\nis expected to approach the representational power of large\nand dense layers, but at a considerably lower computational\ncomplexity.\nDespite good accuracy, the realization of Inception models has been accompanied with a series of complicating fac1arXiv:1611.05431v2  [cs.CV]  11 Apr 2017\ntors \u2014 the \ufb01lter numbers and sizes are tailored for each\nindividual transformation, and the modules are customized\nstage-by-stage. Although careful combinations of these\ncomponents yield excellent neural network recipes, it is in\ngeneral unclear how to adapt the Inception architectures to\nnew datasets/tasks, especially when there are many factors\nand hyper-parameters to be designed.\nIn this paper, we present a simple architecture which\nadopts VGG/ResNets\u2019 strategy of repeating layers, while\nexploiting the split-transform-merge strategy in an easy, extensible way. A module in our network performs a set\nof transformations, each on a low-dimensional embedding,\nwhose outputs are aggregated by summation. We pursuit a\nsimple realization of this idea \u2014 the transformations to be\naggregated are all of the same topology ( e.g., Fig. 1 (right))."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc41-6db4-11ee-9a88-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-5", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "simple realization of this idea \u2014 the transformations to be\naggregated are all of the same topology ( e.g., Fig. 1 (right)).\nThis design allows us to extend to any large number of\ntransformations without specialized designs.\nInterestingly, under this simpli\ufb01ed situation we show that\nour model has two other equivalent forms (Fig. 3). The reformulation in Fig. 3(b) appears similar to the InceptionResNet module [37] in that it concatenates multiple paths;\nbut our module differs from all existing Inception modules\nin that all our paths share the same topology and thus the\nnumber of paths can be easily isolated as a factor to be investigated. In a more succinct reformulation, our module\ncan be reshaped by Krizhevsky et al .\u2019s grouped convolutions [24] (Fig. 3(c)), which, however, had been developed\nas an engineering compromise.\nWe empirically demonstrate that our aggregated transformations outperform the original ResNet module, even\nunder the restricted condition of maintaining computational\ncomplexity and model size \u2014 e.g., Fig. 1(right) is designed\nto keep the FLOPs complexity and number of parameters of\nFig. 1(left). We emphasize that while it is relatively easy to\nincrease accuracy by increasing capacity (going deeper or\nwider), methods that increase accuracy while maintaining"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc42-6db4-11ee-9f5c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-6", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "Fig. 1(left). We emphasize that while it is relatively easy to\nincrease accuracy by increasing capacity (going deeper or\nwider), methods that increase accuracy while maintaining\n(or reducing) complexity are rare in the literature.\nOur method indicates that cardinality (the size of the\nset of transformations) is a concrete, measurable dimension that is of central importance, in addition to the dimensions of width and depth. Experiments demonstrate that increasing cardinality is a more effective way of gaining accuracy than going deeper or wider , especially when depth and\nwidth starts to give diminishing returns for existing models.\nOur neural networks, named ResNeXt (suggesting the\nnextdimension), outperform ResNet-101/152 [14], ResNet200 [15], Inception-v3 [39], and Inception-ResNet-v2 [37]\non the ImageNet classi\ufb01cation dataset. In particular, a\n101-layer ResNeXt is able to achieve better accuracy than\nResNet-200 [15] but has only 50% complexity. Moreover,\nResNeXt exhibits considerably simpler designs than all Inception models. ResNeXt was the foundation of our submission to the ILSVRC 2016 classi\ufb01cation task, in whichwe secured second place. This paper further evaluates\nResNeXt on a larger ImageNet-5K set and the COCO object"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc43-6db4-11ee-9998-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-7", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "ResNeXt on a larger ImageNet-5K set and the COCO object\ndetection dataset [27], showing consistently better accuracy\nthan its ResNet counterparts. We expect that ResNeXt will\nalso generalize well to other visual (and non-visual) recognition tasks.\n2. Related Work\nMulti-branch convolutional networks . The Inception\nmodels [38, 17, 39, 37] are successful multi-branch architectures where each branch is carefully customized.\nResNets [14] can be thought of as two-branch networks\nwhere one branch is the identity mapping. Deep neural decision forests [22] are tree-patterned multi-branch networks\nwith learned splitting functions.\nGrouped convolutions . The use of grouped convolutions\ndates back to the AlexNet paper [24], if not earlier. The\nmotivation given by Krizhevsky et al. [24] is for distributing\nthe model over two GPUs. Grouped convolutions are supported by Caffe [19], Torch [3], and other libraries, mainly\nfor compatibility of AlexNet. To the best of our knowledge,\nthere has been little evidence on exploiting grouped convolutions to improve accuracy. A special case of grouped convolutions is channel-wise convolutions in which the number\nof groups is equal to the number of channels. Channel-wise"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc44-6db4-11ee-b556-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-8", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "of groups is equal to the number of channels. Channel-wise\nconvolutions are part of the separable convolutions in [35].\nCompressing convolutional networks . Decomposition (at\nspatial [6, 18] and/or channel [6, 21, 16] level) is a widely\nadopted technique to reduce redundancy of deep convolutional networks and accelerate/compress them. Ioannouet al . [16] present a \u201croot\u201d-patterned network for reducing computation, and branches in the root are realized\nby grouped convolutions. These methods [6, 18, 21, 16]\nhave shown elegant compromise of accuracy with lower\ncomplexity and smaller model sizes. Instead of compression, our method is an architecture that empirically shows\nstronger representational power.\nEnsembling . Averaging a set of independently trained networks is an effective solution to improving accuracy [24],\nwidely adopted in recognition competitions [33]. Veit et al.\n[40] interpret a single ResNet as an ensemble of shallower\nnetworks, which results from ResNet\u2019s additive behaviors\n[15]. Our method harnesses additions to aggregate a set of\ntransformations. But we argue that it is imprecise to view\nour method as ensembling, because the members to be aggregated are trained jointly, not independently.\n3. Method\n3.1. Template"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc45-6db4-11ee-9d55-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-9", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "our method as ensembling, because the members to be aggregated are trained jointly, not independently.\n3. Method\n3.1. Template\nWe adopt a highly modularized design following\nVGG/ResNets. Our network consists of a stack of residstage output ResNet-50 ResNeXt-50 (32 \u00024d)\nconv1 112\u0002112 7\u00027, 64, stride 2 7\u00027, 64, stride 2\nconv2 56\u0002563\u00023 max pool, stride 2 3\u00023 max pool, stride 22\n641\u00021, 64\n3\u00023, 64\n1\u00021, 2563\n75\u000232\n641\u00021, 128\n3\u00023, 128, C=32\n1\u00021, 2563\n75\u00023\nconv3 28\u0002282\n641\u00021, 128\n3\u00023, 128\n1\u00021, 5123\n75\u000242\n641\u00021, 256\n3\u00023, 256, C=32\n1\u00021, 5123\n75\u00024\nconv4 14\u0002142\n641\u00021, 256\n3\u00023, 256\n1\u00021, 10243\n75\u000262\n641\u00021, 512\n3\u00023, 512, C=32\n1\u00021, 10243\n75\u00026\nconv5 7\u000272\n641\u00021, 512\n3\u00023, 512"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc46-6db4-11ee-91d9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-10", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "3\u00023, 512, C=32\n1\u00021, 10243\n75\u00026\nconv5 7\u000272\n641\u00021, 512\n3\u00023, 512\n1\u00021, 20483\n75\u000232\n641\u00021, 1024\n3\u00023, 1024, C=32\n1\u00021, 20483\n75\u00023\n1\u00021global average pool global average pool\n1000-d fc, softmax 1000-d fc, softmax\n# params. 25.5\u000210625.0\u0002106\nFLOPs 4.1\u00021094.2\u0002109\nTable 1. ( Left) ResNet-50. ( Right ) ResNeXt-50 with a 32 \u00024d\ntemplate (using the reformulation in Fig. 3(c)). Inside the brackets\nare the shape of a residual block, and outside the brackets is the\nnumber of stacked blocks on a stage. \u201c C=32\u201d suggests grouped\nconvolutions [24] with 32 groups. The numbers of parameters and\nFLOPs are similar between these two models.\nual blocks. These blocks have the same topology, and are\nsubject to two simple rules inspired by VGG/ResNets: (i)\nif producing spatial maps of the same size, the blocks share"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc47-6db4-11ee-bd5d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-11", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "subject to two simple rules inspired by VGG/ResNets: (i)\nif producing spatial maps of the same size, the blocks share\nthe same hyper-parameters (width and \ufb01lter sizes), and (ii)\neach time when the spatial map is downsampled by a factor of 2, the width of the blocks is multiplied by a factor\nof 2. The second rule ensures that the computational complexity, in terms of FLOPs (\ufb02oating-point operations, in #\nof multiply-adds), is roughly the same for all blocks.\nWith these two rules, we only need to design a template\nmodule, and all modules in a network can be determined\naccordingly. So these two rules greatly narrow down the\ndesign space and allow us to focus on a few key factors.\nThe networks constructed by these rules are in Table 1.\n3.2. Revisiting Simple Neurons\nThe simplest neurons in arti\ufb01cial neural networks perform inner product (weighted sum), which is the elementary transformation done by fully-connected and convolutional layers. Inner product can be thought of as a form of\naggregating transformation:\nDX\ni=1wixi; (1)\nwhere x= [x1; x2; :::; x D]is aD-channel input vector to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc48-6db4-11ee-b063-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-12", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "DX\ni=1wixi; (1)\nwhere x= [x1; x2; :::; x D]is aD-channel input vector to\nthe neuron and wiis a \ufb01lter\u2019s weight for the i-th chan.......\n+x1 x2 xD x3\nw1w2w3wDxFigure 2. A simple neuron that performs inner product.\nnel. This operation (usually including some output nonlinearity) is referred to as a \u201cneuron\u201d. See Fig. 2.\nThe above operation can be recast as a combination of\nsplitting, transforming, and aggregating . (i) Splitting : the\nvector xis sliced as a low-dimensional embedding, and\nin the above, it is a single-dimension subspace xi. (ii)\nTransforming : the low-dimensional representation is transformed, and in the above, it is simply scaled: wixi. (iii)\nAggregating : the transformations in all embeddings are aggregated byPD\ni=1.\n3.3. Aggregated Transformations\nGiven the above analysis of a simple neuron, we consider replacing the elementary transformation ( wixi) with\na more generic function, which in itself can also be a network. In contrast to \u201cNetwork-in-Network\u201d [26] that turns"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc49-6db4-11ee-8acf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-13", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "a more generic function, which in itself can also be a network. In contrast to \u201cNetwork-in-Network\u201d [26] that turns\nout to increase the dimension of depth, we show that our\n\u201cNetwork-in- Neuron \u201d expands along a new dimension.\nFormally, we present aggregated transformations as:\nF(x) =CX\ni=1Ti(x); (2)\nwhereTi(x)can be an arbitrary function. Analogous to a\nsimple neuron,Tishould project xinto an (optionally lowdimensional) embedding and then transform it.\nIn Eqn.(2), Cis the size of the set of transformations\nto be aggregated. We refer to Cascardinality [2]. In\nEqn.(2) Cis in a position similar to Din Eqn.(1), but C\nneed not equal Dand can be an arbitrary number. While\nthe dimension of width is related to the number of simple\ntransformations (inner product), we argue that the dimension of cardinality controls the number of more complex\ntransformations. We show by experiments that cardinality\nis an essential dimension and can be more effective than the\ndimensions of width and depth.\nIn this paper, we consider a simple way of designing the\ntransformation functions: all Ti\u2019s have the same topology."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc4a-6db4-11ee-a293-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-14", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "dimensions of width and depth.\nIn this paper, we consider a simple way of designing the\ntransformation functions: all Ti\u2019s have the same topology.\nThis extends the VGG-style strategy of repeating layers of\nthe same shape, which is helpful for isolating a few factors\nand extending to any large number of transformations. We\nset the individual transformation Tito be the bottleneckshaped architecture [14], as illustrated in Fig. 1 (right). In\nthis case, the \ufb01rst 1 \u00021 layer in eachTiproduces the lowdimensional embedding.\nequivalent\n256, 1x1, 4\n4, 3x3, 4\n4, 1x1, 256\n+256, 1x1, 4\n4, 3x3, 4\n4, 1x1, 256256, 1x1, 4\n4, 3x3, 4\n4, 1x1, 256\n(a)....total 32\npaths256-d in\n+256, 1x1, 4\n4, 3x3, 4256, 1x1, 4\n4, 3x3, 4256, 1x1, 4\n4, 3x3, 4\n+concatenate\n128, 1x1, 256256-d in\n....total 32\npaths\n(b)256, 1x1, 128"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc4b-6db4-11ee-bffa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-15", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "+concatenate\n128, 1x1, 256256-d in\n....total 32\npaths\n(b)256, 1x1, 128\n128, 3x3, 128\ngroup = 32\n128, 1x1, 256\n+256-d in\n(c)256-d out 256-d out 256-d outFigure 3. Equivalent building blocks of ResNeXt. (a): Aggregated residual transformations, the same as Fig. 1 right. (b): A block equivalent\nto (a), implemented as early concatenation. (c): A block equivalent to (a,b), implemented as grouped convolutions [24]. Notations in bold\ntext highlight the reformulation changes. A layer is denoted as (# input channels, \ufb01lter size, # output channels).\nThe aggregated transformation in Eqn.(2) serves as the\nresidual function [14] (Fig. 1 right):\ny=x+CX\ni=1Ti(x); (3)\nwhere yis the output.\nRelation to Inception-ResNet . Some tensor manipulations show that the module in Fig. 1(right) (also shown in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc4c-6db4-11ee-8659-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-16", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "where yis the output.\nRelation to Inception-ResNet . Some tensor manipulations show that the module in Fig. 1(right) (also shown in\nFig. 3(a)) is equivalent to Fig. 3(b).3Fig. 3(b) appears similar to the Inception-ResNet [37] block in that it involves\nbranching and concatenating in the residual function. But\nunlike all Inception or Inception-ResNet modules, we share\nthe same topology among the multiple paths. Our module\nrequires minimal extra effort designing each path.\nRelation to Grouped Convolutions . The above module becomes more succinct using the notation of grouped convolutions [24].4This reformulation is illustrated in Fig. 3(c).\nAll the low-dimensional embeddings (the \ufb01rst 1 \u00021 layers)\ncan be replaced by a single, wider layer ( e.g., 1\u00021, 128-d\nin Fig 3(c)). Splitting is essentially done by the grouped\nconvolutional layer when it divides its input channels into\ngroups. The grouped convolutional layer in Fig. 3(c) performs 32 groups of convolutions whose input and output\nchannels are 4-dimensional. The grouped convolutional\nlayer concatenates them as the outputs of the layer. The"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc4d-6db4-11ee-8505-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-17", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "channels are 4-dimensional. The grouped convolutional\nlayer concatenates them as the outputs of the layer. The\nblock in Fig. 3(c) looks like the original bottleneck residual block in Fig. 1(left), except that Fig. 3(c) is a wider but\nsparsely connected module.\n3An informal but descriptive proof is as follows. Note the equality:\nA1B1+A2B2= [A1; A2][B1;B2]where [;]is horizontal concatenation and [ ; ]is vertical concatenation. Let Aibe the weight of the last layer\nandBibe the output response of the second-last layer in the block. In the\ncase of C= 2, the element-wise addition in Fig. 3(a) is A1B1+A2B2,\nthe weight of the last layer in Fig. 3(b) is [A1; A2], and the concatenation\nof outputs of second-last layers in Fig. 3(b) is [B1;B2].\n4In a group conv layer [24], input and output channels are divided into\nCgroups, and convolutions are separately performed within each group.\nequivalent\n64, 3x3, 4 64, 3x3, 4 64, 3x3, 4\n....total 32\npaths"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc4e-6db4-11ee-8906-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-18", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "equivalent\n64, 3x3, 4 64, 3x3, 4 64, 3x3, 4\n....total 32\npaths\n+4, 3x3, 64\n+4, 3x3, 64 4, 3x3, 6464, 3x3, 128\n128, 3x3, 64\n+Figure 4. ( Left): Aggregating transformations of depth = 2 .\n(Right ): An equivalent block, which is trivially wider.\nWe note that the reformulations produce nontrivial\ntopologies only when the block has depth \u00153. If the block\nhas depth =2 (e.g., the basic block in [14]), the reformulations lead to trivially a wide, dense module. See the illustration in Fig. 4.\nDiscussion . We note that although we present reformulations that exhibit concatenation (Fig. 3(b)) or grouped convolutions (Fig. 3(c)), such reformulations are not always applicable for the general form of Eqn.(3), e.g., if the transformationTitakes arbitrary forms and are heterogenous.\nWe choose to use homogenous forms in this paper because\nthey are simpler and extensible. Under this simpli\ufb01ed case,\ngrouped convolutions in the form of Fig. 3(c) are helpful for\neasing implementation.\n3.4. Model Capacity"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc4f-6db4-11ee-8024-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-19", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "grouped convolutions in the form of Fig. 3(c) are helpful for\neasing implementation.\n3.4. Model Capacity\nOur experiments in the next section will show that\nour models improve accuracy when maintaining the model\ncomplexity and number of parameters. This is not only interesting in practice, but more importantly, the complexity\nand number of parameters represent inherent capacity of\nmodels and thus are often investigated as fundamental properties of deep networks [8].\nWhen we evaluate different cardinalities Cwhile preserving complexity, we want to minimize the modi\ufb01cation\nof other hyper-parameters. We choose to adjust the width of\ncardinality C 1 2 4 8 32\nwidth of bottleneck d 64 40 24 14 4\nwidth of group conv. 64 80 96 112 128\nTable 2. Relations between cardinality and width (for the template\nof conv2), with roughly preserved complexity on a residual block.\nThe number of parameters is \u001870k for the template of conv2. The\nnumber of FLOPs is \u00180.22 billion (# params \u000256\u000256 for conv2).\nthe bottleneck ( e.g., 4-d in Fig 1(right)), because it can be\nisolated from the input and output of the block. This strategy introduces no change to other hyper-parameters (depth\nor input/output width of blocks), so is helpful for us to focus"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc50-6db4-11ee-8c79-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-20", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "isolated from the input and output of the block. This strategy introduces no change to other hyper-parameters (depth\nor input/output width of blocks), so is helpful for us to focus\non the impact of cardinality.\nIn Fig. 1(left), the original ResNet bottleneck block [14]\nhas256\u000164 + 3\u00013\u000164\u000164 + 64\u0001256\u001970k parameters and\nproportional FLOPs (on the same feature map size). With\nbottleneck width d, our template in Fig. 1(right) has:\nC\u0001(256\u0001d+ 3\u00013\u0001d\u0001d+d\u0001256) (4)\nparameters and proportional FLOPs. When C= 32 and\nd= 4, Eqn.(4)\u001970k. Table 2 shows the relationship between cardinality Cand bottleneck width d.\nBecause we adopt the two rules in Sec. 3.1, the above\napproximate equality is valid between a ResNet bottleneck\nblock and our ResNeXt on all stages (except for the subsampling layers where the feature maps size changes). Table 1 compares the original ResNet-50 and our ResNeXt-50\nthat is of similar capacity.5We note that the complexity can\nonly be preserved approximately, but the difference of the\ncomplexity is minor and does not bias our results.\n4. Implementation details"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc51-6db4-11ee-a763-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-21", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "only be preserved approximately, but the difference of the\ncomplexity is minor and does not bias our results.\n4. Implementation details\nOur implementation follows [14] and the publicly available code of fb.resnet.torch [11]. On the ImageNet\ndataset, the input image is 224 \u0002224 randomly cropped\nfrom a resized image using the scale and aspect ratio augmentation of [38] implemented by [11]. The shortcuts are\nidentity connections except for those increasing dimensions\nwhich are projections (type B in [14]). Downsampling of\nconv3, 4, and 5 is done by stride-2 convolutions in the 3 \u00023\nlayer of the \ufb01rst block in each stage, as suggested in [11].\nWe use SGD with a mini-batch size of 256 on 8 GPUs (32\nper GPU). The weight decay is 0.0001 and the momentum\nis 0.9. We start from a learning rate of 0.1, and divide it by\n10 for three times using the schedule in [11]. We adopt the\nweight initialization of [13]. In all ablation comparisons, we\nevaluate the error on the single 224 \u0002224 center crop from\nan image whose shorter side is 256.\nOur models are realized by the form of Fig. 3(c). We"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc52-6db4-11ee-8f41-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-22", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "evaluate the error on the single 224 \u0002224 center crop from\nan image whose shorter side is 256.\nOur models are realized by the form of Fig. 3(c). We\nperform batch normalization (BN) [17] right after the con5The marginally smaller number of parameters and marginally higher\nFLOPs are mainly caused by the blocks where the map sizes change.volutions in Fig. 3(c).6ReLU is performed right after each\nBN, expect for the output of the block where ReLU is performed after the adding to the shortcut, following [14].\nWe note that the three forms in Fig. 3 are strictly equivalent, when BN and ReLU are appropriately addressed as\nmentioned above. We have trained all three forms and\nobtained the same results. We choose to implement by\nFig. 3(c) because it is more succinct and faster than the other\ntwo forms.\n5. Experiments\n5.1. Experiments on ImageNet-1K\nWe conduct ablation experiments on the 1000-class ImageNet classi\ufb01cation task [33]. We follow [14] to construct\n50-layer and 101-layer residual networks. We simply replace all blocks in ResNet-50/101 with our blocks.\nNotations . Because we adopt the two rules in Sec. 3.1, it is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc53-6db4-11ee-aab5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-23", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "Notations . Because we adopt the two rules in Sec. 3.1, it is\nsuf\ufb01cient for us to refer to an architecture by the template.\nFor example, Table 1 shows a ResNeXt-50 constructed by a\ntemplate with cardinality = 32 and bottleneck width =4d\n(Fig. 3). This network is denoted as ResNeXt-50 ( 32\u00024d)\nfor simplicity. We note that the input/output width of the\ntemplate is \ufb01xed as 256-d (Fig. 3), and all widths are doubled each time when the feature map is subsampled (see\nTable 1).\nCardinality vs. Width . We \ufb01rst evaluate the trade-off between cardinality Cand bottleneck width, under preserved\ncomplexity as listed in Table 2. Table 3 shows the results\nand Fig. 5 shows the curves of error vs. epochs. Comparing with ResNet-50 (Table 3 top and Fig. 5 left), the 32 \u00024d\nResNeXt-50 has a validation error of 22.2%, which is 1.7%\nlower than the ResNet baseline\u2019s 23.9%. With cardinality C\nincreasing from 1 to 32 while keeping complexity, the error\nrate keeps reducing. Furthermore, the 32 \u00024d ResNeXt also"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc54-6db4-11ee-a55b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-24", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "increasing from 1 to 32 while keeping complexity, the error\nrate keeps reducing. Furthermore, the 32 \u00024d ResNeXt also\nhas a much lower training error than the ResNet counterpart, suggesting that the gains are notfrom regularization\nbut from stronger representations .\nSimilar trends are observed in the case of ResNet-101\n(Fig. 5 right, Table 3 bottom), where the 32 \u00024d ResNeXt101 outperforms the ResNet-101 counterpart by 0.8%. Although this improvement of validation error is smaller than\nthat of the 50-layer case, the improvement of training error is still big (20% for ResNet-101 and 16% for 32 \u00024d\nResNeXt-101, Fig. 5 right). In fact, more training data\nwill enlarge the gap of validation error, as we show on an\nImageNet-5K set in the next subsection.\nTable 3 also suggests that with complexity preserved , increasing cardinality at the price of reducing width starts\nto show saturating accuracy when the bottleneck width is\n6With BN, for the equivalent form in Fig. 3(a), BN is employed after\naggregating the transformations and before adding to the shortcut.\n0 1 02 03 04 05 06 07 08 09 0\nepochs1520253035404550top-1 error (%)ResNet-50 (1 x 64d) train"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc55-6db4-11ee-8a7d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-25", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "0 1 02 03 04 05 06 07 08 09 0\nepochs1520253035404550top-1 error (%)ResNet-50 (1 x 64d) train\nResNet-50 (1 x 64d) val\nResNeXt-50 (32 x 4d) train\nResNeXt-50 (32 x 4d) val\n0 1 02 03 04 05 06 07 08 09 0\nepochs1520253035404550top-1 error (%)ResNet-101 (1 x 64d) train\nResNet-101 (1 x 64d) val\nResNeXt-101 (32 x 4d) train\nResNeXt-101 (32 x 4d) valFigure 5. Training curves on ImageNet-1K. ( Left): ResNet/ResNeXt-50 with preserved complexity ( \u00184.1 billion FLOPs, \u001825 million\nparameters); ( Right ): ResNet/ResNeXt-101 with preserved complexity ( \u00187.8 billion FLOPs,\u001844 million parameters).\nsetting top-1 error (%)\nResNet-50 1\u000264d 23.9\nResNeXt-50 2\u000240d 23.0\nResNeXt-50 4\u000224d 22.6\nResNeXt-50 8\u000214d 22.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc56-6db4-11ee-a700-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-26", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "ResNeXt-50 4\u000224d 22.6\nResNeXt-50 8\u000214d 22.3\nResNeXt-50 32\u00024d 22.2\nResNet-101 1\u000264d 22.0\nResNeXt-101 2\u000240d 21.7\nResNeXt-101 4\u000224d 21.4\nResNeXt-101 8\u000214d 21.3\nResNeXt-101 32\u00024d 21.2\nTable 3. Ablation experiments on ImageNet-1K. ( Top): ResNet50 with preserved complexity ( \u00184.1 billion FLOPs); ( Bottom ):\nResNet-101 with preserved complexity ( \u00187.8 billion FLOPs). The\nerror rate is evaluated on the single crop of 224 \u0002224 pixels.\nsmall. We argue that it is not worthwhile to keep reducing\nwidth in such a trade-off. So we adopt a bottleneck width\nno smaller than 4d in the following.\nIncreasing Cardinality vs. Deeper/Wider . Next we investigate increasing complexity by increasing cardinality C\nor increasing depth or width. The following comparison\ncan also be viewed as with reference to 2 \u0002FLOPs of the\nResNet-101 baseline. We compare the following variants"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc57-6db4-11ee-a1d5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-27", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "or increasing depth or width. The following comparison\ncan also be viewed as with reference to 2 \u0002FLOPs of the\nResNet-101 baseline. We compare the following variants\nthat have\u001815 billion FLOPs. (i) Going deeper to 200 layers. We adopt the ResNet-200 [15] implemented in [11].\n(ii)Going wider by increasing the bottleneck width. (iii)\nIncreasing cardinality by doubling C.\nTable 4 shows that increasing complexity by 2 \u0002consistently reduces error vs. the ResNet-101 baseline (22.0%).\nBut the improvement is small when going deeper (ResNet200, by 0.3%) or wider (wider ResNet-101, by 0.7%).\nOn the contrary, increasing cardinality Cshows muchsetting top-1 err (%) top-5 err (%)\n1\u0002complexity references:\nResNet-101 1\u000264d 22.0 6.0\nResNeXt-101 32\u00024d 21.2 5.6\n2\u0002complexity models follow:\nResNet- 200[15] 1\u000264d 21.7 5.8\nResNet-101, wider 1\u0002100d 21.3 5.7\nResNeXt-101 2\u000264d 20.7 5.5\nResNeXt-101 64\u00024d 20.4 5.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc58-6db4-11ee-8c3e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-28", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "ResNeXt-101 2\u000264d 20.7 5.5\nResNeXt-101 64\u00024d 20.4 5.3\nTable 4. Comparisons on ImageNet-1K when the number of\nFLOPs is increased to 2 \u0002of ResNet-101\u2019s. The error rate is evaluated on the single crop of 224 \u0002224 pixels. The highlighted factors\nare the factors that increase complexity.\nbetter results than going deeper or wider . The 2\u000264d\nResNeXt-101 ( i.e., doubling Con 1\u000264d ResNet-101 baseline and keeping the width) reduces the top-1 error by 1.3%\nto 20.7%. The 64\u00024d ResNeXt-101 ( i.e., doubling Con\n32\u00024d ResNeXt-101 and keeping the width) reduces the\ntop-1 error to 20.4%.\nWe also note that 32 \u00024d ResNet-101 (21.2%) performs\nbetter than the deeper ResNet-200 and the wider ResNet101, even though it has only \u001850% complexity. This again\nshows that cardinality is a more effective dimension than\nthe dimensions of depth and width.\nResidual connections. The following table shows the effects of the residual (shortcut) connections:\nsetting w/ residual w/o residual"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc59-6db4-11ee-9ff6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-29", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "the dimensions of depth and width.\nResidual connections. The following table shows the effects of the residual (shortcut) connections:\nsetting w/ residual w/o residual\nResNet-50 1\u000264d 23.9 31.2\nResNeXt-50 32\u00024d 22.2 26.1\nRemoving shortcuts from the ResNeXt-50 increases the error by 3.9 points to 26.1%. Removing shortcuts from its\nResNet-50 counterpart is much worse (31.2%). These comparisons suggest that the residual connections are helpful\nforoptimization , whereas aggregated transformations are\nstronger representations , as shown by the fact that they\nperform consistently better than their counterparts with or\nwithout residual connections.\nPerformance. For simplicity we use Torch\u2019s built-in\ngrouped convolution implementation, without special optimization. We note that this implementation was brute-force\nand not parallelization-friendly. On 8 GPUs of NVIDIA\nM40, training 32\u00024d ResNeXt-101 in Table 3 takes 0.95s\nper mini-batch, vs. 0.70s of ResNet-101 baseline that has\nsimilar FLOPs. We argue that this is a reasonable overhead.\nWe expect carefully engineered lower-level implementation\n(e.g., in CUDA) will reduce this overhead. We also expect"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc5a-6db4-11ee-83ff-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-30", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "We expect carefully engineered lower-level implementation\n(e.g., in CUDA) will reduce this overhead. We also expect\nthat the inference time on CPUs will present less overhead.\nTraining the 2\u0002complexity model (64 \u00024d ResNeXt-101)\ntakes 1.7s per mini-batch and 10 days total on 8 GPUs.\nComparisons with state-of-the-art results. Table 5 shows\nmore results of single-crop testing on the ImageNet validation set. In addition to testing a 224 \u0002224 crop, we\nalso evaluate a 320 \u0002320 crop following [15]. Our results compare favorably with ResNet, Inception-v3/v4, and\nInception-ResNet-v2, achieving a single-crop top-5 error\nrate of 4.4%. In addition, our architecture design is much\nsimpler than all Inception models, and requires considerably fewer hyper-parameters to be set by hand.\nResNeXt is the foundation of our entries to the ILSVRC\n2016 classi\ufb01cation task, in which we achieved 2ndplace.\nWe note that many models (including ours) start to get saturated on this dataset after using multi-scale and/or multicrop testing. We had a single-model top-1/top-5 error rates"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc5b-6db4-11ee-aae2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-31", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "of 17.7%/3.7% using the multi-scale dense testing in [14],\non par with Inception-ResNet-v2\u2019s single-model results of\n17.8%/3.7% that adopts multi-scale, multi-crop testing. We\nhad an ensemble result of 3.03% top-5 error on the test set,\non par with the winner\u2019s 2.99% and Inception-v4/InceptionResNet-v2\u2019s 3.08% [37].\n224\u0002224 320\u0002320 / 299 \u0002299\ntop-1 err top-5 err top-1 err top-5 err\nResNet-101 [14] 22.0 6.0 - ResNet-200 [15] 21.7 5.8 20.1 4.8\nInception-v3 [39] - - 21.2 5.6\nInception-v4 [37] - - 20.0 5.0\nInception-ResNet-v2 [37] - - 19.9 4.9\nResNeXt-101 ( 64\u00024d) 20.4 5.3 19.1 4.4\nTable 5. State-of-the-art models on the ImageNet-1K validation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc5c-6db4-11ee-8e60-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-32", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "Table 5. State-of-the-art models on the ImageNet-1K validation\nset (single-crop testing). The test size of ResNet/ResNeXt is\n224\u0002224 and 320\u0002320 as in [15] and of the Inception models\nis 299\u0002299.\n12345\nmini-batches \u00d71052025303540455055top-1 error (%)ResNet-101 (1 x 64d) val\nResNeXt-101 (32 x 4d) valFigure 6. ImageNet-5K experiments. Models are trained on the\n5K set and evaluated on the original 1K validation set, plotted as\na 1K-way classi\ufb01cation task. ResNeXt and its ResNet counterpart\nhave similar complexity.\n5K-way classi\ufb01cation 1K-way classi\ufb01cation\nsetting top-1 top-5 top-1 top-5\nResNet-50 1\u000264d 45.5 19.4 27.1 8.2\nResNeXt-50 32\u00024d 42.3 16.8 24.4 6.6\nResNet-101 1\u000264d 42.4 16.9 24.2 6.8\nResNeXt-101 32\u00024d 40.1 15.1 22.2 5.7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc5d-6db4-11ee-9d9a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-33", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "ResNeXt-101 32\u00024d 40.1 15.1 22.2 5.7\nTable 6. Error (%) on ImageNet-5K . The models are trained on\nImageNet-5K and tested on the ImageNet-1K val set , treated as a\n5K-way classi\ufb01cation task or a 1K-way classi\ufb01cation task at test\ntime. ResNeXt and its ResNet counterpart have similar complexity. The error is evaluated on the single crop of 224 \u0002224 pixels.\n5.2. Experiments on ImageNet-5K\nThe performance on ImageNet-1K appears to saturate.\nBut we argue that this is not because of the capability of the\nmodels but because of the complexity of the dataset. Next\nwe evaluate our models on a larger ImageNet subset that\nhas 5000 categories.\nOur 5K dataset is a subset of the full ImageNet-22K set\n[33]. The 5000 categories consist of the original ImageNet1K categories and additional 4000 categories that have the\nlargest number of images in the full ImageNet set. The 5K\nset has 6.8 million images, about 5 \u0002of the 1K set. There is\nno of\ufb01cial train/val split available, so we opt to evaluate on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc5e-6db4-11ee-86af-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-34", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "set has 6.8 million images, about 5 \u0002of the 1K set. There is\nno of\ufb01cial train/val split available, so we opt to evaluate on\nthe original ImageNet-1K validation set. On this 1K-class\nval set, the models can be evaluated as a 5K-way classi\ufb01cation task (all labels predicted to be the other 4K classes are\nautomatically erroneous) or as a 1K-way classi\ufb01cation task\n(softmax is applied only on the 1K classes) at test time.\nThe implementation details are the same as in Sec. 4.\nThe 5K-training models are all trained from scratch, and\n# params CIFAR-10 CIFAR-100\nWide ResNet [43] 36.5M 4.17 20.50\nResNeXt-29, 8\u000264d 34.4M 3.65 17.77\nResNeXt-29, 16\u000264d 68.1M 3.58 17.31\nTable 7. Test error (%) and model size on CIFAR. Our results are\nthe average of 10 runs.\nare trained for the same number of mini-batches as the 1Ktraining models (so 1/5 \u0002epochs). Table 6 and Fig. 6 show"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc5f-6db4-11ee-b75a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-35", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "are trained for the same number of mini-batches as the 1Ktraining models (so 1/5 \u0002epochs). Table 6 and Fig. 6 show\nthe comparisons under preserved complexity. ResNeXt-50\nreduces the 5K-way top-1 error by 3.2% comparing with\nResNet-50, and ResNetXt-101 reduces the 5K-way top-1\nerror by 2.3% comparing with ResNet-101. Similar gaps\nare observed on the 1K-way error. These demonstrate the\nstronger representational power of ResNeXt.\nMoreover, we \ufb01nd that the models trained on the 5K\nset (with 1K-way error 22.2%/5.7% in Table 6) perform\ncompetitively comparing with those trained on the 1K set\n(21.2%/5.6% in Table 3), evaluated on the same 1K-way\nclassi\ufb01cation task on the validation set. This result is\nachieved without increasing the training time (due to the\nsame number of mini-batches) and without \ufb01ne-tuning. We\nargue that this is a promising result, given that the training\ntask of classifying 5K categories is a more challenging one.\n5.3. Experiments on CIFAR"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc60-6db4-11ee-a30e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-36", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "argue that this is a promising result, given that the training\ntask of classifying 5K categories is a more challenging one.\n5.3. Experiments on CIFAR\nWe conduct more experiments on CIFAR-10 and 100\ndatasets [23]. We use the architectures as in [14] and replace the basic residual block by the bottleneck template\nof2\n641\u00021, 64\n3\u00023, 64\n1\u00021, 2563\n75. Our networks start with a single 3 \u00023 conv\nlayer, followed by 3 stages each having 3 residual blocks,\nand end with average pooling and a fully-connected classi\ufb01er (total 29-layer deep), following [14]. We adopt the same\ntranslation and \ufb02ipping data augmentation as [14]. Implementation details are in the appendix.\nWe compare two cases of increasing complexity based\non the above baseline: (i) increase cardinality and \ufb01x all\nwidths, or (ii) increase width of the bottleneck and \ufb01x cardinality = 1. We train and evaluate a series of networks\nunder these changes. Fig. 7 shows the comparisons of test\nerror rates vs. model sizes. We \ufb01nd that increasing cardinality is more effective than increasing width, consistent to\nwhat we have observed on ImageNet-1K. Table 7 shows the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc61-6db4-11ee-802f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-37", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "what we have observed on ImageNet-1K. Table 7 shows the\nresults and model sizes, comparing with the Wide ResNet\n[43] which is the best published record. Our model with a\nsimilar model size (34.4M) shows results better than Wide\nResNet. Our larger method achieves 3.58% test error (average of 10 runs) on CIFAR-10 and 17.31% on CIFAR-100.\nTo the best of our knowledge, these are the state-of-the-art\nresults (with similar data augmentation) in the literature including unpublished technical reports.\n4 8 16 32 64 128\n# of parameters (M)3.53.63.73.83.94.04.14.24.34.44.5test error (%)ResNet-29 (increase width)\nResNeXt-29 (increase cardinality)1x64d\n16x64d8x64d4x64d2x64d\n1x384d1x128d\n1x256d1x192dFigure 7. Test error vs. model size on CIFAR-10. The results are\ncomputed with 10 runs, shown with standard error bars. The labels\nshow the settings of the templates.\nsetting AP@0.5 AP\nResNet-50 1\u000264d 47.6 26.5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc62-6db4-11ee-bb23-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-38", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "show the settings of the templates.\nsetting AP@0.5 AP\nResNet-50 1\u000264d 47.6 26.5\nResNeXt-50 32\u00024d 49.7 27.5\nResNet-101 1\u000264d 51.1 29.8\nResNeXt-101 32\u00024d 51.9 30.0\nTable 8. Object detection results on the COCO minival set.\nResNeXt and its ResNet counterpart have similar complexity.\n5.4. Experiments on COCO object detection\nNext we evaluate the generalizability on the COCO object detection set [27]. We train the models on the 80k training set plus a 35k val subset and evaluate on a 5k val subset\n(called minival ), following [1]. We evaluate the COCOstyle Average Precision (AP) as well as AP@IoU=0.5 [27].\nWe adopt the basic Faster R-CNN [32] and follow [14] to\nplug ResNet/ResNeXt into it. The models are pre-trained\non ImageNet-1K and \ufb01ne-tuned on the detection set. Implementation details are in the appendix."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc63-6db4-11ee-bfc1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-39", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "on ImageNet-1K and \ufb01ne-tuned on the detection set. Implementation details are in the appendix.\nTable 8 shows the comparisons. On the 50-layer baseline, ResNeXt improves AP@0.5 by 2.1% and AP by 1.0%,\nwithout increasing complexity. ResNeXt shows smaller improvements on the 101-layer baseline. We conjecture that\nmore training data will lead to a larger gap, as observed on\nthe ImageNet-5K set.\nIt is also worth noting that recently ResNeXt has been\nadopted in Mask R-CNN [12] that achieves state-of-the-art\nresults on COCO instance segmentation and object detection tasks.\nAcknowledgment\nS.X. and Z.T.\u2019s research was partly supported by NSF\nIIS-1618477. The authors would like to thank Tsung-Yi\nLin and Priya Goyal for valuable discussions.\nA. Implementation Details: CIFAR\nWe train the models on the 50k training set and evaluate\non the 10k test set. The input image is 32 \u000232 randomly\ncropped from a zero-padded 40 \u000240 image or its \ufb02ipping,\nfollowing [14]. No other data augmentation is used. The"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc64-6db4-11ee-a693-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-40", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "cropped from a zero-padded 40 \u000240 image or its \ufb02ipping,\nfollowing [14]. No other data augmentation is used. The\n\ufb01rst layer is 3\u00023 conv with 64 \ufb01lters. There are 3 stages\neach having 3 residual blocks, and the output map size is\n32, 16, and 8 for each stage [14]. The network ends with a\nglobal average pooling and a fully-connected layer. Width\nis increased by 2\u0002when the stage changes (downsampling),\nas in Sec. 3.1. The models are trained on 8 GPUs with a\nmini-batch size of 128, with a weight decay of 0.0005 and\na momentum of 0.9. We start with a learning rate of 0.1\nand train the models for 300 epochs, reducing the learning\nrate at the 150-th and 225-th epoch. Other implementation\ndetails are as in [11].\nB. Implementation Details: Object Detection\nWe adopt the Faster R-CNN system [32]. For simplicity\nwe do not share the features between RPN and Fast R-CNN.\nIn the RPN step, we train on 8 GPUs with each GPU holding\n2 images per mini-batch and 256 anchors per image. We\ntrain the RPN step for 120k mini-batches at a learning rate"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc65-6db4-11ee-84be-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-41", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "2 images per mini-batch and 256 anchors per image. We\ntrain the RPN step for 120k mini-batches at a learning rate\nof 0.02 and next 60k at 0.002. In the Fast R-CNN step, we\ntrain on 8 GPUs with each GPU holding 1 image and 64\nregions per mini-batch. We train the Fast R-CNN step for\n120k mini-batches at a learning rate of 0.005 and next 60k at\n0.0005, We use a weight decay of 0.0001 and a momentum\nof 0.9. Other implementation details are as in https://\ngithub.com/rbgirshick/py-faster-rcnn .\nReferences\n[1] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Insideoutside net: Detecting objects in context with skip\npooling and recurrent neural networks. In CVPR ,\n2016.\n[2] G. Cantor. \u00a8Uber unendliche, lineare punktmannichfaltigkeiten, arbeiten zur mengenlehre aus den jahren\n1872-1884. 1884.\n[3] R. Collobert, S. Bengio, and J. Mari \u00b4ethoz. Torch: a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc66-6db4-11ee-ac9b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-42", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "1872-1884. 1884.\n[3] R. Collobert, S. Bengio, and J. Mari \u00b4ethoz. Torch: a\nmodular machine learning software library. Technical\nreport, Idiap, 2002.\n[4] A. Conneau, H. Schwenk, L. Barrault, and Y . Lecun. Very deep convolutional networks for natural language processing. arXiv:1606.01781 , 2016.\n[5] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In CVPR , 2005.\n[6] E. Denton, W. Zaremba, J. Bruna, Y . LeCun, and\nR. Fergus. Exploiting linear structure within convolutional networks for ef\ufb01cient evaluation. In NIPS ,\n2014.[7] J. Donahue, Y . Jia, O. Vinyals, J. Hoffman, N. Zhang,\nE. Tzeng, and T. Darrell. Decaf: A deep convolutional\nactivation feature for generic visual recognition. In\nICML , 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc67-6db4-11ee-ac67-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-43", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "E. Tzeng, and T. Darrell. Decaf: A deep convolutional\nactivation feature for generic visual recognition. In\nICML , 2014.\n[8] D. Eigen, J. Rolfe, R. Fergus, and Y . LeCun. Understanding deep architectures using a recursive convolutional network. arXiv:1312.1847 , 2013.\n[9] R. Girshick. Fast R-CNN. In ICCV , 2015.\n[10] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich\nfeature hierarchies for accurate object detection and\nsemantic segmentation. In CVPR , 2014.\n[11] S. Gross and M. Wilber. Training and investigating Residual Nets. https://github.com/\nfacebook/fb.resnet.torch , 2016.\n[12] K. He, G. Gkioxari, P. Doll \u00b4ar, and R. Girshick. Mask\nR-CNN. arXiv:1703.06870 , 2017.\n[13] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep\ninto recti\ufb01ers: Surpassing human-level performance\non imagenet classi\ufb01cation. In ICCV , 2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc68-6db4-11ee-81d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-44", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "into recti\ufb01ers: Surpassing human-level performance\non imagenet classi\ufb01cation. In ICCV , 2015.\n[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual\nlearning for image recognition. In CVPR , 2016.\n[15] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In ECCV , 2016.\n[16] Y . Ioannou, D. Robertson, R. Cipolla, and A. Criminisi. Deep roots: Improving cnn ef\ufb01ciency with hierarchical \ufb01lter groups. arXiv:1605.06489 , 2016.\n[17] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal\ncovariate shift. In ICML , 2015.\n[18] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank\nexpansions. In BMVC , 2014.\n[19] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc69-6db4-11ee-8982-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-45", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "[19] Y . Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long,\nR. Girshick, S. Guadarrama, and T. Darrell. Caffe:\nConvolutional architecture for fast feature embedding.\narXiv:1408.5093 , 2014.\n[20] N. Kalchbrenner, L. Espeholt, K. Simonyan, A. v. d.\nOord, A. Graves, and K. Kavukcuoglu. Neural machine translation in linear time. arXiv:1610.10099 ,\n2016.\n[21] Y .-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and\nD. Shin. Compression of deep convolutional neural\nnetworks for fast and low power mobile applications.\nInICLR , 2016.\n[22] P. Kontschieder, M. Fiterau, A. Criminisi, and S. R.\nBul`o. Deep convolutional neural decision forests. In\nICCV , 2015.\n[23] A. Krizhevsky. Learning multiple layers of features\nfrom tiny images. Tech Report , 2009."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc6a-6db4-11ee-96b0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-46", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "ICCV , 2015.\n[23] A. Krizhevsky. Learning multiple layers of features\nfrom tiny images. Tech Report , 2009.\n[24] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classi\ufb01cation with deep convolutional neural\nnetworks. In NIPS , 2012.\n[25] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.\nHoward, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition.\nNeural computation , 1989.\n[26] M. Lin, Q. Chen, and S. Yan. Network in network. In\nICLR , 2014.\n[27] T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona,\nD. Ramanan, P. Doll \u00b4ar, and C. L. Zitnick. Microsoft\nCOCO: Common objects in context. In ECCV . 2014.\n[28] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR ,\n2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc6b-6db4-11ee-a6dd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-47", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "[28] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR ,\n2015.\n[29] D. G. Lowe. Distinctive image features from scaleinvariant keypoints. IJCV , 2004.\n[30] A. Oord, S. Dieleman, H. Zen, K. Simonyan,\nO. Vinyals, A. Graves, N. Kalchbrenner, A. Senior,\nand K. Kavukcuoglu. Wavenet: A generative model\nfor raw audio. arXiv:1609.03499 , 2016.\n[31] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning\nto segment object candidates. In NIPS , 2015.\n[32] S. Ren, K. He, R. Girshick, and J. Sun. Faster RCNN: Towards real-time object detection with region\nproposal networks. In NIPS , 2015.\n[33] O. Russakovsky, J. Deng, H. Su, J. Krause,\nS. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla,\nM. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc6c-6db4-11ee-ad52-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-48", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet\nLarge Scale Visual Recognition Challenge. IJCV ,\n2015.\n[34] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y . LeCun. Overfeat: Integrated recognition, localization and detection using convolutional\nnetworks. In ICLR , 2014.\n[35] L. Sifre and S. Mallat. Rigid-motion scattering for\ntexture classi\ufb01cation. arXiv:1403.1687 , 2014.\n[36] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In\nICLR , 2015.\n[37] C. Szegedy, S. Ioffe, and V . Vanhoucke. Inceptionv4, inception-resnet and the impact of residual connections on learning. In ICLR Workshop , 2016.\n[38] C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed,\nD. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR ,\n2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc6d-6db4-11ee-aceb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-49", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR ,\n2015.\n[39] C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and\nZ. Wojna. Rethinking the inception architecture for\ncomputer vision. In CVPR , 2016.[40] A. Veit, M. Wilber, and S. Belongie. Residual networks behave like ensembles of relatively shallow network. In NIPS , 2016.\n[41] Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi,\nW. Macherey, M. Krikun, Y . Cao, Q. Gao,\nK. Macherey, et al. Google\u2019s neural machine translation system: Bridging the gap between human and\nmachine translation. arXiv:1609.08144 , 2016.\n[42] W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer,\nA. Stolcke, D. Yu, and G. Zweig. The Microsoft\n2016 Conversational Speech Recognition System.\narXiv:1609.03528 , 2016."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc6e-6db4-11ee-9265-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.05431-50", "title": "Aggregated Residual Transformations for Deep Neural Networks", "source": "http://arxiv.org/pdf/1611.05431", "start_index": 0, "chroma:document": "2016 Conversational Speech Recognition System.\narXiv:1609.03528 , 2016.\n[43] S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC , 2016.\n[44] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional neural networks. In ECCV ,\n2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc6f-6db4-11ee-bc27-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-0", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "Deep Watershed Transform for Instance Segmentation\nMin Bai Raquel Urtasun\nDepartment of Computer Science, University of Toronto\nfmbai, urtasun g@cs.toronto.edu\nAbstract\nMost contemporary approaches to instance segmentation use complex pipelines involving conditional random\n\ufb01elds, recurrent neural networks, object proposals, or template matching schemes. In this paper, we present a simple yet powerful end-to-end convolutional neural network to\ntackle this task. Our approach combines intuitions from the\nclassical watershed transform and modern deep learning to\nproduce an energy map of the image where object instances\nare unambiguously represented as energy basins. We then\nperform a cut at a single energy level to directly yield connected components corresponding to object instances. Our\nmodel achieves more than double the performance over\nthe state-of-the-art on the challenging Cityscapes Instance\nLevel Segmentation task.\n1. Introduction\nInstance segmentation seeks to identify the semantic\nclass of each pixel as well as associate each pixel with a\nphysical instance of an object. This is in contrast with\nsemantic segmentation, which is only concerned with the\n\ufb01rst task. Instance segmentation is particularly challenging in street scenes, where the scale of the objects can vary\ntremendously. Furthermore, the appearance of objects are"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc70-6db4-11ee-9a57-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-1", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "\ufb01rst task. Instance segmentation is particularly challenging in street scenes, where the scale of the objects can vary\ntremendously. Furthermore, the appearance of objects are\naffected by partial occlusions, specularities, intensity saturation, and motion blur. Solving this task will, however,\ntremendously bene\ufb01t applications such as object manipulation in robotics, or scene understanding and tracking for\nself-driving cars.\nCurrent approaches generally use complex pipelines to\nhandle instance extraction involving object proposals [20,\n22, 7], conditional random \ufb01elds (CRF) [32, 33], large recurrent neural networks (RNN) [24, 23, 2], or template\nmatching [28]. In contrast, we present an exceptionally\nsimple and intuitive method that signi\ufb01cantly outperforms\nthe state-of-the-art. In particular, we derive a novel approach which brings together classical grouping techniques\nand modern deep neural networks.\nThe watershed transform is a well studied method inmathematical morphology. Its application to image segmentation can be traced back to the 70\u2019s [4, 3]. The idea\nbehind this transform is fairly intuitive. Any greyscale image can be considered as a topographic surface. If we \ufb02ood\nthis surface from its minima and prevent the merging of the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc71-6db4-11ee-bf73-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-2", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "behind this transform is fairly intuitive. Any greyscale image can be considered as a topographic surface. If we \ufb02ood\nthis surface from its minima and prevent the merging of the\nwaters coming from different sources, we effectively partition the image into different components (i.e., regions).\nThis transformation is typically applied to the image gradient, thus the basins correspond to homogeneous regions in\nthe image. A signi\ufb01cant limitation of the watershed transform is its propensity to over-segment the image. One of\nthe possible solutions is to estimate the locations of object\ninstance markers, which guide the selection of a subset of\nthese basins [11, 21]. Heuristics on the relative depth of the\nbasins can be exploited in order to merge basins. However,\nextracting appropriate markers and creating good heuristics\nis dif\ufb01cult in practice. As a consequence, modern techniques for instance segmentation do not exploit the watershed transform.\nIn this paper, we propose a novel approach which combines the strengths of modern deep neural networks with the\npower of this classical bottom-up grouping technique. We\npropose to directly learn the energy of the watershed transform such that each basin corresponds to a single instance,\nwhile all dividing ridges are at the same height in the energy domain. As a consequence, the components can be"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc72-6db4-11ee-b9fd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-3", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "while all dividing ridges are at the same height in the energy domain. As a consequence, the components can be\nextracted by a cut at a single energy level without leading\nto over-segmentation. Our approach has several key advantages: it can be easily trained end-to-end, and produces very\nfast and accurate estimates. Our method does not rely on iterative strategies such as RNNs, thus has a constant runtime\nregardless of the number of object instances.\nWe demonstrate the effectiveness of our approach in the\nchallenging Cityscapes Instance Segmentation benchmark\n[6], and show that we more than double the performance\nof the current state-of-the-art. In the following sections, we\n\ufb01rst review related work. We then present the details behind\nour intuition and model design, followed by an analysis of\nour model\u2019s performance. Finally, we explore the impact of\nvarious parts of our model in ablation studies.\n1arXiv:1611.08303v2  [cs.CV]  4 May 2017\n(a) Input Image\n (b) Semantic Segmentation [34]\n (c) Our Instance Segmentation\n (d) GT Instance Segmentation\nFigure 1: Sample prediction: the input image is gated by sem. segmentation from [34] and passed through our model."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc73-6db4-11ee-abec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-4", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "(d) GT Instance Segmentation\nFigure 1: Sample prediction: the input image is gated by sem. segmentation from [34] and passed through our model.\n2. Related\nSeveral instance level segmentation approaches have\nbeen proposed in recent years. We now brie\ufb02y review them.\nProposal based: Many approaches are based on the re\ufb01nement of object proposals. For example, [1] generates\nobject segments proposals, and reasons about combining\nthem into object instances. In a similar spirit, [12] selects\nproposals using CNN features and non-maximum suppression. Based on this, [5] further reasons about multiple object proposals to handle occlusion scenarios where single\nobjects are split into multiple disconnected patches. [7] uses\na deep cascaded neural network to propose object instance\nbounding boxes, re\ufb01ne instance masks, and semantically\nlabel the masks in sequence. [20, 22] generate segmentation proposals using deep CNNs, which are then further re\ufb01ned to achieve better segmentation boundaries. Additionally, [31] uses a modi\ufb01ed R-CNN model to propose instance\nbounding boxes, which are then further re\ufb01ned to obtain instance level segmentation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1adc74-6db4-11ee-81b0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-5", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "bounding boxes, which are then further re\ufb01ned to obtain instance level segmentation.\nDeep structured models: [32, 33] \ufb01rst use CNNs to perform local instance disambiguation and labelling, followed\nby a global conditional random \ufb01eld (CRF) to achieve instance label consistency. Recent work by [2] uses object\ndetection proposals in conjunction with a deep high order\nCRF to reason about pixel assignment in overlapping object\nproposal boxes.\nTemplate matching: [28] extracts image features using\nCNNs to assign a sector label to each pixel in an object instance, which corresponds to one of eight discretized radial\nbins around the object\u2019s visible center. A template matching\nscheme is then used to associate instance center proposals\nand pixels with an object instance.\nRecurrent Networks: [24] uses CNNs for feature extraction, followed by a recurrent neural network (RNN) that\ngenerates instance labels for one object at a time. The recurrent structures (based on ConvLSTM [25]) keep track of\ninstances that have already been generated, and inhibit these\nregions from further instance generation. Additionally, [23]\nextracts image features similar to [28] and employs a fairly\ncomplex pipeline including a ConvLSTM structure to direct"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b0ffb-6db4-11ee-8a56-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-6", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "regions from further instance generation. Additionally, [23]\nextracts image features similar to [28] and employs a fairly\ncomplex pipeline including a ConvLSTM structure to direct\na bounding box generation network followed by a segmentation network that extracts individual instances.CNN: [15] leverages only a CNN trained to provide multiple outputs to simultaneously predict instance numbers,\nbounding box coordinates, and category con\ufb01dence scores\nfor each pixel. This is followed by generic clustering algorithms to group the resulting output into instance-wise labels. Additionally, [13] proposed deep convolutional neural\nnetwork that learns the underlying shapes of objects, and\nperforms multiple unconstrained inference steps to re\ufb01ne\nregions corresponding to an object instance while ignoring\nneighboring pixels that do not belong to the primary object.\nProposal + recursion: [14] proposed a novel method\nthat recursively re\ufb01nes proposals.\nIn contrast, in this paper we propose a novel approach\nwhich combines the strengths of modern deep neural networks with the power of the watershed transform. Our\nmodel is simple, fast, accurate, and inherently handles an\narbitrary number of instances per image with ease.\n3. A Review on the Watershed Transform\nWe start our discussion with a review of the watershed"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b0ffc-6db4-11ee-ae52-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-7", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "arbitrary number of instances per image with ease.\n3. A Review on the Watershed Transform\nWe start our discussion with a review of the watershed\ntransform, a well studied method in mathematical morphology. This technique is built on the fact that any greyscale\nimage can be considered as a topographic surface. If we\n\ufb02ood this surface from its minima while building barriers\nto prevent the merging of the waters coming from different sources, we effectively partition the image into different\ncomponents or regions. These components are called catchment basins. The barriers or watershed lines then represent\nthe boundaries between the different basins (i.e., boundaries\nbetween regions).\nThis process is illustrated in the \ufb01rst row of Fig. 2 for\na one dimensional energy function. In this case, the watershed transform results in seven components, which are\nillustrated in different colors. Note that the traditional watershed transform tends to produce an over-segmentation of\nthe image due to spurious small ridges which produce separate components. In this example, although there are 3\nmain components, the watershed transform over-segments\nthe image because of small perturbations in the energy.\nSeveral algorithms have been developed to estimate the\ncomponents. [3] proposed an algorithm that iteratively \ufb01lls\nthe watershed landscape from each local minimum, adding\ndams wherever two neighbouring bodies of water meet."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b0ffd-6db4-11ee-b5fe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-8", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "components. [3] proposed an algorithm that iteratively \ufb01lls\nthe watershed landscape from each local minimum, adding\ndams wherever two neighbouring bodies of water meet.\n(a) Traditional Watershed Energy\n(b) Our learned energy\nFigure 2: Conceptual 1-D comparison between the traditional watershed transform and our deep watershed transform. The resulting instances are represented as colors\nacross the top.\nThese dams de\ufb01ne the segmentation boundaries. Additionally, [18] details a number of alternative watershed transform algorithms, including topological distance, shortest\npath algorithms, spanning trees, and marker based methods.\nThe watershed transform is typically applied to the image gradient, while the catchment basins correspond to homogeneous grey level regions in the image. However, estimating sharp gradients that represent the boundaries between different instances is a very challenging process.\nIn the next section, we will show an alternative approach\nwhich directly learns to predict the energy landscape.\n4. Deep Watershed Tranform\nIn this section, we present our novel approach to instance\nlevel segmentation. In particular, we learn the energy of the\nwatershed transform with a feed-forward neural network.\nThe idea behind our approach is very intuitive. It consists\nof learning to predict an energy landscape such that each\nbasin corresponds to a single instance, while all ridges are\nat the same height in the energy domain. As a consequence,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b0ffe-6db4-11ee-9360-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-9", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "of learning to predict an energy landscape such that each\nbasin corresponds to a single instance, while all ridges are\nat the same height in the energy domain. As a consequence,\nthe watershed cut corresponds to a single threshold in the\nenergy, which does not lead to over segmentation. We refer\nthe reader to the lower half of Fig. 2 for an illustration of\nthe desired energy.\nUnfortunately, learning the energy landscape from\nscratch is a complex task. Therefore, we aid the network\nby de\ufb01ning an intermediate task, where we learn the direction of descent of the watershed energy. This is then passedthrough another set of network layers to learn the \ufb01nal energy. In principle, one can interpret this network as learning\nto perform the distance transform of each point within an\nobject instance to the instance\u2019s boundary. Fig. 4 shows an\nexample of the input, intermediate results, and \ufb01nal output\nof our method. We refer the reader to Fig. 3 for an illustration of our network architecture.\n4.1. Direction Network (DN)\nOur approach leverages semantic segmentation as input\nto focus only on relevant areas of the image. Note that our\nnetwork is agnostic to the choice of segmentation algorithm.\nIn our paper, we use the semantic segmentation results from\nPSPNet [34]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b0fff-6db4-11ee-bd96-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-10", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "network is agnostic to the choice of segmentation algorithm.\nIn our paper, we use the semantic segmentation results from\nPSPNet [34].\nThe network takes as input the original RGB image gated\nby a binarized semantic segmentation, where all pixels that\nare not part of one of the semantic classes of interest are set\nto zero. The input image is augmented by adding the semantic segmentation as a fourth channel. Because the RGB\nvalues range from 0 to 255 for each channel before mean\nsubtraction, we likewise scale the encoding of the semantic\nsegmentation image such that the class labels are equally\nspaced numerically, and the variance is approximately equal\nto that of values in the RGB channels.\nTo aid the model in producing an accurate energy landscape, we pre-train the overall network\u2019s \ufb01rst part (referenced here as the Direction Network, DN) to estimate the\ndirection of descent of the energy at each pixel. We parameterize it with a unit vector pointing away from the nearest\npoint on the object\u2019s instance boundary. This supervision\ngives a very strong training signal: the direction of the nearest boundary is directly encoded in the unit vector. Furthermore, a pair of pixels straddling an occlusion boundary\nbetween two objects will have target unit vectors pointing\nin opposite directions. Here, associating a pixel with the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1000-6db4-11ee-8cef-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-11", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "between two objects will have target unit vectors pointing\nin opposite directions. Here, associating a pixel with the\nwrong object instance incurs the maximum possible angular\nerror. This is true regardless of the shape of the objects, even\nwhen they are highly concave or elongated. This forces the\nnetwork to learn very accurate boundary localization at the\npixel level.\nLetDgt(p)be the ground truth distance transform from\na point to the boundary of the instance it belongs to. We\nde\ufb01ne our ground truth targets as the normalized gradient\n~ upof this distance transform. More formally,\n~ up;gt=rDgt(p)\njrDgt(p)j\nThus, the DN produces a two channel output at the input\nimage\u2019s resolution representing the directional unit vector.\nIt is important to note that the normalization layer at the output of the DN restricts the sum of each channel\u2019s squared\noutput to be 1. This greatly reduces the dif\ufb01culty of using the output of non-linear activation functions to model\nFigure 3: Complete network architecture. The network takes the original RGB image gated by semantic segmentation and\nconcatenated with the semantic segmentation as input, and produces the deep watershed transform energy map as the output.\n(a) Input Image\n (b) GT angle of ~ up\n (c) GT Watershed Energy"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1001-6db4-11ee-a711-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-12", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "(a) Input Image\n (b) GT angle of ~ up\n (c) GT Watershed Energy\n (d) GT Instances\n(e) Sem. Segmentation of [34]\n (f) Pred. angle of ~ up\n (g) Pred. Watershed Transform\n (h) Pred. Instances\nFigure 4: Our network takes the RGB image (a) and the semantic segmentation (e) as input, and predicts a unit vector at each\nforeground pixel pointing directly away from the nearest boundary (f). Based on this, we then predict a modi\ufb01ed watershed\ntransform energy (g), upon which we perform cut at a \ufb01xed threshold to yield the \ufb01nal predictions (h).\na scalar target. Additionally, phrasing the training target as\na unit vector instead of an angle bypasses the problem of\nthe equivalent angles 0\u000eand 360\u000ehaving a large numerical\ndifference.\nThe feature extraction portion of the DN\u2019s architecture\nis inspired by VGG16 [27]. However, there are several important modi\ufb01cations. Direction prediction is a very precise task, as the output can vary greatly between neighboring pixels. Thus, it is critical to avoid losing spatial\nresolution. We utilize a modi\ufb01ed version of the \ufb01rst 13"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1002-6db4-11ee-8ddc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-13", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "resolution. We utilize a modi\ufb01ed version of the \ufb01rst 13\nlayers of VGG, where the third and fourth pooling layers are changed to average pooling while the \ufb01fth pooling\nlayer is removed. To preserve spatial resolution, we exploit a high-capacity, multi-scale information aggregation\nscheme inspired by popular methods in semantic segmenta-tion [9, 30, 16]. In particular, the outputs of conv3 ,conv4 ,\nandconv5 individually undergo a 5\u00025convolution followed\nby two 1\u00021convolutions. After this, the outputs of the\nsecond and third paths are upsampled to the resolution of\nthe \ufb01rst path. The resulting feature volumes from the three\npaths are concatenated. This undergoes an additional set\nof three 1\u00021convolutions, before being upsampled to the\ninput resolution.\n4.2. Watershed Transform Network (WTN)\nThe second half of the overall network takes as input the\n2-channel unit vector map, and produces a discretized modi\ufb01ed watershed transform map with K= 16 possible energy\nvalues. In this case, bin 0 corresponds to background or regions within 2 pixels of an instance boundary and is referred\nto as having an energy level of 0. Meanwhile, higher numbered bins with higher energy levels correspond to regions"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1003-6db4-11ee-81d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-14", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "to as having an energy level of 0. Meanwhile, higher numbered bins with higher energy levels correspond to regions\nin the interior of object instances. The bins are chosen to\nmaximize the binning resolution of energy levels near zero\n(to facilitate accurate cutting), while achieving an approximate balance between the total numbers of pixels within\neach class.\nThe WTN is a fairly generic CNN, with an emphasis on\nreasoning with high spatial accuracy and resolution. The architecture is shown in the right half of Fig. 3. In particular,\nthe network consists of two 5\u00025convolutional \ufb01lter blocks\neach followed by a 2\u00022average pooling, before undergoing a set of 1\u00021convolutions and upsampling to the input\nimage resolution.\n4.3. Network Training\nWe \ufb01rst pre-train our DN and WTN networks. This process is followed by end-to-end training of the whole network.\nDirection Network pre-training: We pre-train the network using the mean squared error in the angular domain as\nloss:\nldirection =X\np2P objwpkcos\u00001< ~ up;GT; ~ up;pred>k2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1004-6db4-11ee-9a7e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-15", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "loss:\nldirection =X\np2P objwpkcos\u00001< ~ up;GT; ~ up;pred>k2\nwherePobjis the set of all pixels belonging to a semantic class with object instances, and wpis a weighting factor proportional to the inverse of the square root of the object instance\u2019s area. We use the mean squared angular error as opposed to the commonly used cosine distance, as\nlarge angular errors (occuring when the network associates\na pixel with the wrong object instance) should incur significant larger penalties than small angular errors (which have\nlittle impact).\nThis network is pre-trained using the original input images gated by the ground truth semantic segmentation and\nthe ground truth unit vectors ~ uGT. The lower layers (conv1\nto conv5) are initialized with VGG16 [27]. The weights of\nthe upper layers are initialized randomly according to the\nXavier initialization scheme [10]. However, the intialization variance of the weights of the \ufb01nal layer of each prediction branch is set manually such that the variance of each\nbranch\u2019s output is of the same order of magnitude before\nconcatenation. This encourages the network to consider the\noutput of each branch. We use the ADAM optimizer to train"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1005-6db4-11ee-86be-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-16", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "concatenation. This encourages the network to consider the\noutput of each branch. We use the ADAM optimizer to train\nthe network for 20 epochs with a batch size of 4, with a constant learning rate of 1e-5 and L2 weight penalty of 1e-5.\nWatershed Network pre-training: The network is\ntrained using a modi\ufb01ed cross-entropy loss\nlwatershed =X\np2P objKX\nk=1wpck(\u0016tp;klog \u0016yp;k+tp;klogyp;k)where tp;kis the k-th element of pixel p\u2019s one-hot target\nvector, yp;kis the k-th channel of the network output at p,\nwpis a coef\ufb01cient to adjust the importance of smaller objects (as de\ufb01ned in the DN pre-training section), and ckis\na scaling constant speci\ufb01c to each discretization class. Because the single energy level cut occurs at just above zero\nenergy (i.e., near object instance boundaries), an accurate\nestimation of pixels at low energy levels is of crucial importance. Thus, the set fckgis selected to be in increasing order. In this case, any errors of low energy levels are assigned\na greater level of penalty, which encourages the network to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1006-6db4-11ee-b3be-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-17", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "a greater level of penalty, which encourages the network to\nfocus on boundary regions. We pre-train the network using\nthe ground truth semantic segmentation and ground truth\ndirection predictions as input, and the discretized ground\ntruth modi\ufb01ed watershed transform as target. All weights\nare initialized using Xavier initialization [10]. We train the\nnetwork for 25 epochs using the ADAM optimizer. A batch\nsize of 6, constant learning rate of 5e-4, and a L2 weight\npenalty of 1e-6 are used.\nEnd-to-end \ufb01ne-tuning: We cascaded the pre-trained\nmodels for the DN and WTN and \ufb01ne-tuned the complete\nmodel for 20 epochs using the RGB image and semantic\nsegmentation output of PSPNet as input, and the ground\ntruth distance transforms as the training target. We use a\nbatch size of 3, constant learning rate of 5e-6, and a L2\nweight penalty of 1e-6.\n4.4. Energy Cut and Instance Extraction\nWe cut our watershed transform output at energy level 1\n(0 to 2 pixels from boundary) for classes with many smaller\nobjects ( person ,rider ,bicycle ,motorcycle ), and level 2 (3 to\n4 pixels from boundary) for classes with larger objects ( car,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1007-6db4-11ee-a978-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-18", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "objects ( person ,rider ,bicycle ,motorcycle ), and level 2 (3 to\n4 pixels from boundary) for classes with larger objects ( car,\nbus,truck ,train ). Following this, the instances are dilated\nusing a circular structuring element whose radius equals the\nmaximum erosion at the energy level of the cut. This offsets the boundary erosion from the non-zero threshold. The\nconnected components are identi\ufb01ed in the resulting image,\ndirectly yielding the proposed instances. The proposals are\nfurther re\ufb01ned by basic hole-\ufb01lling. Finally, we remove\nsmall, spurious instances.\n5. Experimental Evaluation\nIn this section, we evaluate our approach on the challenging Cityscapes Instance-Level Semantic Labelling Task [6].\nThe of\ufb01cial benchmark test and validation set results are\nfound in Tables 1 and 2. We then perform ablation studies with the validation set to examine the performance of\nvarious aspects of our model.\nDataset: The Cityscapes Instance Labelling Task contains 5000 \ufb01nely annotated street scene images taken by a\nvehicle-mounted camera. Each image has a resolution of\nMethodCityscapes Test Set Cityscapes Val Set\nAP AP 50% AP 100m AP 50m AP muCov"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1008-6db4-11ee-bf51-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-19", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "vehicle-mounted camera. Each image has a resolution of\nMethodCityscapes Test Set Cityscapes Val Set\nAP AP 50% AP 100m AP 50m AP muCov\nvan den Brand et al. [29] 2.3% 3.7% 3.9% 4.9% - Cordts et al. [6] 4.6% 12.9% 7.7% 10.3% - Uhrig et al. [28] 8.9% 21.1% 15.3% 16.7% 9.9% Ours 19.4% 35.3% 31.4% 36.8% 21.2% 68.0%\nTable 1: Cityscapes instance segmentation results using metrics de\ufb01ned in [6] for AP and [26] for muCov.\nMethod Person Rider Car Truck Bus Train Motorcycle Bicycle\nvan den Brand et al. [29] - - 18.2% - - - - Cordts et al. [6] 1.3% 0.6% 10.5% 6.1% 9.7% 5.9% 1.7% 0.5%\nUhrig et al. [28] 12.5% 11.7% 22.5% 3.3% 5.9% 3.2% 6.9% 5.1%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1009-6db4-11ee-9e41-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-20", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "Uhrig et al. [28] 12.5% 11.7% 22.5% 3.3% 5.9% 3.2% 6.9% 5.1%\nOurs 15.5% 14.1% 31.5% 22.5% 27.0% 22.9% 13.9% 8.0%\nTable 2: Cityscapes instance segmentation class speci\ufb01c test set AP scores using metrics de\ufb01ned in [6].\n2048x1024 pixels. Unlike other commonly used datasets\nfor instance segmentation (e.g., Pascal VOC [8], BSDS500\n[17], and CVPPP [19]) Cityscapes has a large number of\nscenes involving dozens of instances with large degrees of\nocclusions at vastly different scales. Eight of the semantically labelled categories have instance-level labelling. We\nrefer the reader to Table 3 for a summary of the statistics of\nthe object instances in this dataset. Note that while the car\nandpeople classes have signi\ufb01cant numbers of instances,\nthe other six classes are rather uncommon. As a result, the\nrare classes have far less training data, and are thus much\nmore challenging. We use the of\ufb01cial training, validation,\nand testing set splits, with 2975, 500, and 1525 images, respectively."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b100a-6db4-11ee-8b6f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-21", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "more challenging. We use the of\ufb01cial training, validation,\nand testing set splits, with 2975, 500, and 1525 images, respectively.\nMetric: We report the metrics employed by the\nCityscapes leaderboard. Several variants of the instancelevel average precision score are used. This is calculated\nby \ufb01nding the precision at various levels of intersectionover-union (IoU) between predicted instances and ground\ntruth instances, ranging from 50% to 95%. The main score\nis the Average Precision (AP). Additionally, there are three\nminor scores: AP at 50% overlap, AP of objects closer\nthan 50m, and AP of objects closer than 100m. Methods\nare ranked by the AP score over all classes (mean AP).\nNote that AP is a detection score, which does not penalize\noverlapping instances or a large number of predictions,\nas long as they are ranked in the proper order. Thus,\nit places approaches (like ours) which predict a single\ninstance label (or background) per pixel at a disadvantage,\nwhile favoring detection-based methods. Despite this, we\nuse these metrics to be consistent with the the evaluation\nof the state-of-the-art. For the fbus,truck ,traingclasses,\nwe order the instances by simply averaging the semantic"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b100b-6db4-11ee-9adb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-22", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "of the state-of-the-art. For the fbus,truck ,traingclasses,\nwe order the instances by simply averaging the semantic\nsegmentation\u2019s output con\ufb01dence within each instance to\nsomewhat counteract the errors by semantic segmentation.Category Total\nPixelsTotal\nInstancesAverage\nSizeInstances\n/ Image\nPerson 6.6e+7 17900 3691 6.0\nRider 7.4e+6 1754 4244 0.6\nCar 3.8e+8 26929 14233 9.1\nTruck 1.5e+7 482 30648 0.2\nBus 1.3e+7 379 34275 0.1\nTrain 3.2e+6 168 19143 0.1\nMotorcycle 5.4e+6 735 7298 0.2\nBicycle 1.6e+7 3655 4332 1.2\nTable 3: Statistics of the training set of the cityscapes instance level segmentation. Average size is in pixels.\nSem. Seg. Src. Sem. Seg. IoU Inst. Seg. AP\nLRR [9] 71.8% 20.2%\nPSPNet [34] 80.2% 21.2%\nTable 4: Comparison of instance segmentation performance\non the validation set with various semantic segmentation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b100c-6db4-11ee-a92a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-23", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "PSPNet [34] 80.2% 21.2%\nTable 4: Comparison of instance segmentation performance\non the validation set with various semantic segmentation\nsources. Semantic segmentation IoU Class scores [6] are\nalso provided.\nFor all other classes, we use a random ordering. Note that\nsophisticated ranking techniques can be used to further\nimprove the score. However, the segmentation quality\nremains this same. This again highlights a shortcoming of\nusing AP as the metric.\nIn addition to AP, we report the mean weighted coverage\nscore introduced in [26] on the validation set. This metric\nenforces a single instance label per pixel, and is therefore\nmore suitable for evaluating our approach. We hope that\nfuture work will likewise report this score when applicable.\nMethod Average Person Rider Car Truck Bus Train Motorcycle Bicycle\nOurs 20.50% 15.50% 13.80% 33.10% 23.60% 39.90% 17.40% 11.90% 8.80%\nOurs + Ordering 21.24% 15.50% 13.80% 33.10% 27.10% 45.20% 14.50% 11.90% 8.80%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b100d-6db4-11ee-ac16-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-24", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "Ours + Oracle Ordering 27.58% 20.60% 18.70% 40.10% 31.50% 50.60% 28.30% 17.40% 13.40%\nTable 5: Comparison of AP scores with various instance ordering techniques using the validation set.\nComparison to the state-of-the-art: We show the instance segmentation test set scores in Table 1. Additionally, we show the class-speci\ufb01c AP scores in Table 2. It is\nevident that we achieve a large improvement over the stateof-the-art in all semantic classes. Moreover, we do not use\ndepth information to train our model, unlike [28].\nAnalysis of the intermediate training target: Our \ufb01nal\nnetwork is the result of the end-to-end \ufb01ne-tuning of two\npre-trained sub-networks (DN and WTN). Fig. 4 (f) shows\nthe output of the DN after \ufb01netuning. It is evident that the\n\ufb01ne-tuned model retained the direction prediction as an intermediate task. This suggests that the intermediate training\ntarget is effective.\nIn\ufb02uence of semantic segmentation: While we elected\nto use PSPNet [34] as our semantic segmentation source, we\nadditionally demonstrate that our method is able to use other"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b100e-6db4-11ee-a09f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-25", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "to use PSPNet [34] as our semantic segmentation source, we\nadditionally demonstrate that our method is able to use other\nsources. Table 4 shows the use of LRR [9] for semantic segmentation. Using the same pre-trained DN and DTN models, we perform end-to-end \ufb01ne-tuning using LRR as semantic segmentation. We note that the performance of our\nmodel improves with better semantic segmentation. Thus,\nfuture advances in segmentic segmentation methods can\nfurther improve the performance of our approach.\nCon\ufb01dence score estimate: As mentioned, the AP score\ncalculated by the Cityscapes benchmark requires a con\ufb01dence score for each instance. For the fbus,truck ,traing\nset, we produce a weak ranking based on semantic segmentation softmax con\ufb01dence. Instances of all other classes are\nrandomly ranked. Table 5 explores the impact of various ordering schemes. We compare our ordering with random for\nall semantic classes, as well as optimal ordering using oracle IoU. We see that ordering using oracle IoU can increase\nour model\u2019s performance by 6.34%. Note, however, that\nthis has no impact on the actual quality of proposed object\ninstances, which remain the same. This shows the necessity"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b100f-6db4-11ee-bf4e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-26", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "this has no impact on the actual quality of proposed object\ninstances, which remain the same. This shows the necessity\nof a different metric such as muCov [26] that can evaluate\nsegmentation-based approaches fairly.\nQualitative Results: Fig. 5 depicts visualizations of sample results on the validation set, which is not used as part of\ntraining. It is evident that our model produces very high\nquality instance segmentation results. In these results, predicted and ground truth object instances only share the same\ncolor if they have greater than 50% IoU.Failure Modes: Our model has several weaknesses.\nSome images in Fig. 5 demonstrate these cases. The \ufb01rst\nissue is that the current formulation of our method does not\nhandle objects that have been separated into multiple pieces\nby occlusion. This is most obvious in the 3rd image from\nthe bottom in Fig. 5 as the far-right vehicle is bisected by the\nstreet sign pole, and the bicycle in the right part of the image just above. The resulting pieces are not merged into one\ncomponent. This is a drawback of most bottom-up grouping approaches. The second issue are cases where two objects sharing an occlusion boundary are mostly but not fully\nseparated by a line of low energy. This is seen in the rightmost vehicle in the 11th image. We anticipate that a combination of our method with top-down reasoning approaches"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1010-6db4-11ee-879c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-27", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "separated by a line of low energy. This is seen in the rightmost vehicle in the 11th image. We anticipate that a combination of our method with top-down reasoning approaches\nwill greatly alleviate these two issues.\nBecause we rely upon correct semantic segmentation, errors from this (such as identifying a train as a bus) cannot be\n\ufb01xed by our method. This is clearly shown by the truck in\nthe last example. A possible solution could be to use semantic segmentation as soft gating, or to reason about semantic\nand instance segmentation jointly. Finally, some very complex scenes such as some subgroups of people on the left in\nthe second to fourth example are incorrectly separated by\nour model, and are fused together.\n6. Conclusion\nIn this paper, we have proposed a simple instance segmentation technique inspired by the intuitive and classical watershed transform. Using a novel deep convolutional neural network and innovative loss functions for pretraining and \ufb01ne-tuning, we proposed a model that generates a modi\ufb01ed watershed energy landscape. From this energy landscape, we directly extract high quality object instances. Our experiments show that we can more than double the performance of the state-of-the-art in the challenging Cityscapes Instance Segmentation task. We will release\nthe network weights and code for training and testing our"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1011-6db4-11ee-ae6c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-28", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "the network weights and code for training and testing our\nmodel. In the future, we plan to augment the method to handle object instances bisected by occlusions. Additionally,\nwe wish to explore the possibility of extending our approach\nto perform joint semantic and instance level segmentation,\nin hopes of further re\ufb01ning both outputs simultaneously.\nAcknowledgements: This work was partially supported\nby ONR-N00014-14-1-0232, Samsung, NVIDIA, Google\nand NSERC.\n(a) Input Image\n (b) Sem. Segmentation [34]\n (c) Our Instance Segmentation\n (d) GT Instance Segmentation\nFigure 5: Sample output of our model on the validation set. Note that predicted object instances and ground truth object\ninstances are only given the same color if they have over 50% IoU.\nReferences\n[1] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale Combinatorial Grouping. In CVPR , 2014.\n2\n[2] A. Arnab and P. Torr. Bottom-up instance segmentation using deep higher-order crfs. In BMVC , 2016. 1, 2\n[3] S. Beucher. The watershed transformation applied to image segmentation. Scanning Microscopy International ,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1012-6db4-11ee-9684-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-29", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "[3] S. Beucher. The watershed transformation applied to image segmentation. Scanning Microscopy International ,\nSuppl:6(1):299\u2013314, 1991. 1, 2\n[4] S. Beucher and C. Lantuejoul. Use of watersheds in contour\ndetection. Proc. Int. Workshop Image Processing, Real-Time\nEdge and Motion Detection/Estimation , 1976. 1\n[5] Y . Chen, X. Liu, and M. Yang. Bottom-up instance segmentation using deep higher-order crfs. In CVPR , 2015. 2\n[6] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,\nR. Benenson, U. Franke, S. Roth, and B. Schiele. The\nCityscapes Dataset for Semantic Urban Scene Understanding. In CVPR , 2016. 1, 5, 6\n[7] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 1,\n2\n[8] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,\nand A. Zisserman. The PASCAL Visual Object Classes"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1013-6db4-11ee-acf6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-30", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "and A. Zisserman. The PASCAL Visual Object Classes\nChallenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html.\n6\n[9] C. F. G. Ghiasi. Laplacian Pyramid Reconstruction and Re\ufb01nement for Semantic Segmentation. In ECCV , 2016. 4, 6,\n7\n[10] X. Glorot and Y . Bengio. Understanding the dif\ufb01culty of\ntraining deep feedforward neural networks. In AISTATS ,\n2010. 5\n[11] V . Grau, A. Mewes, M. Alcaiz, R. Kikinis, and S. War\ufb01eld.\nImproved watershed transform for medical image segmentation using prior information. IEEE Transactions on Medical\nImaging , 23(4):447\u2013458, 2004. 1\n[12] B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV , 2014. 2\n[13] K. Li, B. Hariharan, and J. Malik. Iterative instance segmentation. In CVPR , 2016. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1014-6db4-11ee-9db0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-31", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "[13] K. Li, B. Hariharan, and J. Malik. Iterative instance segmentation. In CVPR , 2016. 2\n[14] X. Liang, Y . Wei, X. Shen, Z. Jie, J. Feng, L. Lin, and S. Yan.\nReversible recursive instance-level object segmentation. In\nCVPR , 2016. 2\n[15] X. Liang, Y . Wei, X. Shen, J. Yang, L. Lin, and S. Yan. Endto-end instance segmentation and counting with recurrent attention. arXiv preprint arXiv:1509.02636 , 2015. 2\n[16] J. Long, E. Shelhamer, and T. Darrell. Fully Convolutional\nNetworks for Semantic Segmentation. In CVPR , 2015. 4\n[17] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database\nof human segmented natural images and its application to\nevaluating segmentation algorithms and measuring ecological statistics. In ICCV , 2001. 6\n[18] F. Meyer. The watershed concept and its use in segmentation:\na brief history. arXiv preprint arXiv:1202.0216 , 2012. 3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1015-6db4-11ee-8edc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-32", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "[18] F. Meyer. The watershed concept and its use in segmentation:\na brief history. arXiv preprint arXiv:1202.0216 , 2012. 3\n[19] M. Minervini, A. Fischbach, H.Scharr, , and S. Tsaftaris.\nFinely-grained annotated datasets for image-based plant phenotyping. Pattern Recognition Letters , pages 1\u201310, 2015. 6[20] P. O. Pinheiro, R. Collobert, and P. Dollr. Learning to segment object candidates. In NIPS , 2015. 1, 2\n[21] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Dollr. Segmentation of 3d head mr images using morphological reconstruction under constraints and automatic selection of markers. In\nICIP , pages 1075\u20131078, 2001. 1\n[22] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Dollr. Learning\nto re\ufb01ne object segments. In ECCV , 2016. 1, 2\n[23] M. Ren and R. Zemel. End-to-end instance segmentation and counting with recurrent attention. arXiv preprint"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1016-6db4-11ee-9278-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-33", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "[23] M. Ren and R. Zemel. End-to-end instance segmentation and counting with recurrent attention. arXiv preprint\narXiv:1605.09410 , 2016. 1, 2\n[24] B. Romera-Paredes and P. H. S. Torr. Recurrent Instance\nSegmentation. In ECCV , 2016. 1, 2\n[25] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo.\nConvolutional LSTM network: A machine learning approach for precipitation nowcasting. In NIPS , 2015. 2\n[26] N. Silberman, D. Sontag, and R. Fergus. Instance segmentation of indoor scenes using a coverage loss. In ECCV , 2014.\n6, 7\n[27] K. Simonyan and A. Zisserman. Very Deep Convolutional\nNetworks for Large-Scale Image Recognition. In ICLR ,\n2015. 4, 5\n[28] J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level\nEncoding and Depth Layering for Instance-level Semantic\nLabeling. In GCPR , 2016. 1, 2, 6, 7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1017-6db4-11ee-a4d2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-34", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "Encoding and Depth Layering for Instance-level Semantic\nLabeling. In GCPR , 2016. 1, 2, 6, 7\n[29] J. van den Brand, M. Ochs, and R. Mester. Instance-level\nSegmentation of Vehicles by Deep Contours. In ACCV Workshop , 2016. 6\n[30] F. Yu and V . Koltun. Multi-Scale Context Aggregation by\nDilated Convolutions. In ICLR , 2016. 4\n[31] S. Zagoruyko, A. Lerer, T.-Y . Lin, P. O. Pinheiro, S. Gross,\nS. Chintala, and P. Doll \u00b4ar. A multipath network for object\ndetection. In BMVC , 2016. 2\n[32] Z. Zhang, S. Fidler, and R. Urtasun. Instance-Level Segmentation with Deep Densely Connected MRFs. In CVPR , 2016.\n1, 2\n[33] Z. Zhang, A. Schwing, S. Fidler, and R. Urtasun. Monocular\nobject instance segmentation and depth ordering with cnns.\nInICCV , 2015. 1, 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1018-6db4-11ee-8d39-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1611.08303-35", "title": "Deep Watershed Transform for Instance Segmentation", "source": "http://arxiv.org/pdf/1611.08303", "start_index": 0, "chroma:document": "object instance segmentation and depth ordering with cnns.\nInICCV , 2015. 1, 2\n[34] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid Scene\nParsing Network. arXiv preprint arXiv:1612.01105 , 2016. 2,\n3, 4, 6, 7, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1019-6db4-11ee-a18c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-0", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "Interaction Networks for Learning about Objects,\nRelations and Physics\nPeter W. Battaglia\nGoogle DeepMind\nLondon, UK N1C 4AG\npeterbattaglia@google.comRazvan Pascanu\nGoogle DeepMind\nLondon, UK N1C 4AG\nrazp@google.comMatthew Lai\nGoogle DeepMind\nLondon, UK N1C 4AG\nmatthewlai@google.com\nDanilo Rezende\nGoogle DeepMind\nLondon, UK N1C 4AG\ndanilor@google.comKoray Kavukcuoglu\nGoogle DeepMind\nLondon, UK N1C 4AG\nkorayk@google.com\nAbstract\nReasoning about objects, relations, and physics is central to human intelligence, and\na key goal of arti\ufb01cial intelligence. Here we introduce the interaction network , a\nmodel which can reason about how objects in complex systems interact, supporting\ndynamical predictions, as well as inferences about the abstract properties of the\nsystem. Our model takes graphs as input, performs object- and relation-centric\nreasoning in a way that is analogous to a simulation, and is implemented using\ndeep neural networks. We evaluate its ability to reason about several challenging\nphysical domains: n-body problems, rigid-body collision, and non-rigid dynamics."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b101a-6db4-11ee-b2f4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-1", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "deep neural networks. We evaluate its ability to reason about several challenging\nphysical domains: n-body problems, rigid-body collision, and non-rigid dynamics.\nOur results show it can be trained to accurately simulate the physical trajectories of\ndozens of objects over thousands of time steps, estimate abstract quantities such\nas energy, and generalize automatically to systems with different numbers and\ncon\ufb01gurations of objects and relations. Our interaction network implementation\nis the \ufb01rst general-purpose, learnable physics engine, and a powerful general\nframework for reasoning about object and relations in a wide variety of complex\nreal-world domains.\n1 Introduction\nRepresenting and reasoning about objects, relations and physics is a \u201ccore\u201d domain of human common\nsense knowledge [ 26], and among the most basic and important aspects of intelligence [ 28,15]. Many\neveryday problems, such as predicting what will happen next in physical environments or inferring\nunderlying properties of complex scenes, are challenging because their elements can be composed\nin combinatorially many possible arrangements. People can nevertheless solve such problems by\ndecomposing the scenario into distinct objects and relations, and reasoning about the consequences\nof their interactions and dynamics. Here we introduce the interaction network \u2013 a model that can\nperform an analogous form of reasoning about objects and relations in complex systems."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b101b-6db4-11ee-8c9f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-1", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 1170, "chroma:document": "decomposing the scenario into distinct objects and relations, and reasoning about the consequences\nof their interactions and dynamics. Here we introduce the interaction network \u2013 a model that can\nperform an analogous form of reasoning about objects and relations in complex systems.\nInteraction networks combine three powerful approaches: structured models, simulation, and deep"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b101c-6db4-11ee-b4df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-2", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "perform an analogous form of reasoning about objects and relations in complex systems.\nInteraction networks combine three powerful approaches: structured models, simulation, and deep\nlearning. Structured models [ 7] can exploit rich, explicit knowledge of relations among objects,\nindependent of the objects themselves, which supports general-purpose reasoning across diverse\ncontexts. Simulation is an effective method for approximating dynamical systems, predicting how the\nelements in a complex system are in\ufb02uenced by interactions with one another, and by the dynamics\nof the system. Deep learning [ 24,16] couples generic architectures with ef\ufb01cient optimization\nalgorithms to provide highly scalable learning and inference in challenging real-world settings.arXiv:1612.00222v1  [cs.AI]  1 Dec 2016\nObject reasoning Relational reasoning\nCompute interaction Apply object dynamicsEffectsObjects,\nrelationsPredictions, \ninferences\na.\nb.Figure 1: Schematic of an interaction network. a.For physical reasoning, the model takes objects and relations\nas input, reasons about their interactions, and applies the effects and physical dynamics to predict new states. b.\nFor more complex systems, the model takes as input a graph that represents a system of objects, oj, and relations,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b101d-6db4-11ee-ab9e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-3", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "For more complex systems, the model takes as input a graph that represents a system of objects, oj, and relations,\nhi; j; r kik, instantiates the pairwise interaction terms, bk, and computes their effects, ek, via a relational model,\nfR(\u0001). The ekare then aggregated and combined with the ojand external effects, xj, to generate input (as cj),\nfor an object model, fO(\u0001), which predicts how the interactions and dynamics in\ufb02uence the objects, p.\nInteraction networks explicitly separate how they reason about relations from how they reason about\nobjects, assigning each task to distinct models which are: fundamentally object- and relation-centric;\nand independent of the observation modality and task speci\ufb01cation (see Model section 2 below\nand Fig. 1a). This lets interaction networks automatically generalize their learning across variable\nnumbers of arbitrarily ordered objects and relations, and also recompose their knowledge of entities\nand interactions in novel and combinatorially many ways. They take relations as explicit input,\nallowing them to selectively process different potential interactions for different input data, rather\nthan being forced to consider every possible interaction or those imposed by a \ufb01xed architecture."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b101e-6db4-11ee-88a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-4", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "allowing them to selectively process different potential interactions for different input data, rather\nthan being forced to consider every possible interaction or those imposed by a \ufb01xed architecture.\nWe evaluate interaction networks by testing their ability to make predictions and inferences about various physical systems, including n-body problems, and rigid-body collision, and non-rigid dynamics.\nOur interaction networks learn to capture the complex interactions that can be used to predict future\nstates and abstract physical properties, such as energy. We show that they can roll out thousands of\nrealistic future state predictions, even when trained only on single-step predictions. We also explore\nhow they generalize to novel systems with different numbers and con\ufb01gurations of elements. Though\nthey are not restricted to physical reasoning, the interaction networks used here represent the \ufb01rst\ngeneral-purpose learnable physics engine, and even have the potential to learn novel physical systems\nfor which no physics engines currently exist.\nRelated work Our model draws inspiration from previous work that reasons about graphs and\nrelations using neural networks. The \u201cgraph neural network\u201d [ 23,19] is a framework that shares\nlearning across nodes and edges, the \u201crecursive autoencoder\u201d [ 25] adapts its processing architecture\nto exploit an input parse tree, the \u201cneural programmer-interpreter\u201d [ 22] is a composable network that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b101f-6db4-11ee-8f5f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-5", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "to exploit an input parse tree, the \u201cneural programmer-interpreter\u201d [ 22] is a composable network that\nmimics the execution trace of a program, and the \u201cspatial transformer\u201d [ 11] learns to dynamically\nmodify network connectivity to capture certain types of interactions. Others have explored deep\nlearning of logical and arithmetic relations [ 27], and relations suitable for visual question-answering\n[1].\nThe behavior of our model is similar in spirit to a physical simulation engine [ 2], which generates\nsequences of states by repeatedly applying rules that approximate the effects of physical interactions\nand dynamics on objects over time. The interaction rules are relation-centric, operating on two or\nmore objects that are interacting, and the dynamics rules are object-centric, operating on individual\nobjects and the aggregated effects of the interactions they participate in. A similar approach has been\ndeveloped independently by Chang et al. [4].\nPrevious AI work on physical reasoning explored commonsense knowledge, qualitative representations, and simulation techniques for approximating physical prediction and inference [ 29,9] . The\n\u201cNeuroAnimator\u201d [ 8] was perhaps the \ufb01rst quantitative approach to learning physical dynamics, by\ntraining neural networks to predict and control the state of articulated bodies. Ladick\u00fd et al. [ 14]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1020-6db4-11ee-81df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-6", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "training neural networks to predict and control the state of articulated bodies. Ladick\u00fd et al. [ 14]\nrecently used regression forests to learn \ufb02uid dynamics. Recent advances in convolutional neural\nnetworks (CNNs) have led to efforts that learn to predict coarse-grained physical dynamics from\n2\nimages [ 20,17,18]. Notably, Fragkiadaki et al. [ 6] used CNNs to predict and control a moving\nball from an image centered at its coordinates. Mottaghi et al. [ 21] trained CNNs to predict the 3D\ntrajectory of an object after an external impulse is applied. Wu et al. [ 30] used CNNs to parse objects\nfrom images, which were then input to a physics engine that supported prediction and inference.\n2 Model\nDe\ufb01nition To describe our model, we use physical reasoning as an example (Fig. 1a), and build\nfrom a simple model to the full interaction network (abbreviated IN). To predict the dynamics of a\nsingle object, one might use an object-centric function, fO, which inputs the object\u2019s state, ot, at\ntimet, and outputs a future state, ot+1. If two or more objects are governed by the same dynamics,\nfOcould be applied to each, independently, to predict their respective future states. But if the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1021-6db4-11ee-a57a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-7", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "fOcould be applied to each, independently, to predict their respective future states. But if the\nobjects interact with one another, then fOis insuf\ufb01cient because it does not capture their relationship.\nAssuming two objects and one directed relationship, e.g., a \ufb01xed object attached by a spring to a freely\nmoving mass, the \ufb01rst (the sender ,o1) in\ufb02uences the second (the receiver ,o2) via their interaction.\nThe effect of this interaction, et+1, can be predicted by a relation-centric function, fR. ThefRtakes\nas inputo1,o2, as well as attributes of their relationship, r, e.g., the spring constant. The fOis\nmodi\ufb01ed so it can input both et+1and the receiver\u2019s current state, o2;t, enabling the interaction to\nin\ufb02uence its future state, o2;t+1,\net+1=fR(o1;t;o2;t;r) o2;t+1=fO(o2;t;et+1)\nThe above formulation can be expanded to larger and more complex systems by representing them"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1022-6db4-11ee-abf0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-8", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "The above formulation can be expanded to larger and more complex systems by representing them\nas a graph,G=hO;Ri, where the nodes, O, correspond to the objects, and the edges, R, to the\nrelations (see Fig. 1b). We assume an attributed, directed multigraph because the relations have\nattributes, and there can be multiple distinct relations between two objects (e.g., rigid and magnetic\ninteractions). For a system with NOobjects andNRrelations, the inputs to the IN are,\nO=fojgj=1:::NO; R=fhi;j;r kikgk=1:::NRwherei6=j;1\u0014i;j\u0014NO; X=fxjgj=1:::NO\nTheOrepresents the states of each object. The triplet, hi;j;r kik, represents the k-th relation in the\nsystem, from sender, oi, to receiver, oj, with relation attribute, rk. TheXrepresents external effects,\nsuch as active control inputs or gravitational acceleration, which we de\ufb01ne as not being part of the\nsystem, and which are applied to each object separately.\nThe basic IN is de\ufb01ned as,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1023-6db4-11ee-a222-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-9", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "system, and which are applied to each object separately.\nThe basic IN is de\ufb01ned as,\nIN(G) =\u001eO(a(G; X; \u001e R(m(G) ) )) (1)\nm(G) =B=fbkgk=1:::NR\nfR(bk) =ek\n\u001eR(B) =E=fekgk=1:::NRa(G;X;E ) =C=fcjgj=1:::NO\nfO(cj) =pj\n\u001eO(C) =P=fpjgj=1:::NO(2)\nThe marshalling function, m, rearranges the objects and relations into interaction terms, bk=\nhoi;oj;rki2B, one per relation, which correspond to each interaction\u2019s receiver, sender, and\nrelation attributes. The relational model, \u001eR, predicts the effect of each interaction, ek2E, by\napplyingfRto eachbk. The aggregation function, a, collects all effects, ek2E, that apply to each\nreceiver object, merges them, and combines them with OandXto form a set of object model inputs,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1024-6db4-11ee-87c3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-10", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "receiver object, merges them, and combines them with OandXto form a set of object model inputs,\ncj2C, one per object. The object model, \u001eO, predicts how the interactions and dynamics in\ufb02uence\nthe objects by applying fOto eachcj, and returning the results, pj2P. This basic IN can predict\nthe evolution of states in a dynamical system \u2013 for physical simulation, Pmay equal the future states\nof the objects, Ot+1.\nThe IN can also be augmented with an additional component to make abstract inferences about the\nsystem. The pj2P, rather than serving as output, can be combined by another aggregation function,\ng, and input to an abstraction model, \u001eA, which returns a single output, q, for the whole system. We\nexplore this variant in our \ufb01nal experiments that use the IN to predict potential energy.\nAn IN applies the same fRandfOto everybkandcj, respectively, which makes their relational and\nobject reasoning able to handle variable numbers of arbitrarily ordered objects and relations. But\none additional constraint must be satis\ufb01ed to maintain this: the afunction must be commutative and\n3\nassociative over the objects and relations. Using summation within ato merge the elements of Einto"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1025-6db4-11ee-927f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-11", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "3\nassociative over the objects and relations. Using summation within ato merge the elements of Einto\nCsatis\ufb01es this, but division would not.\nHere we focus on binary relations, which means there is one interaction term per relation, but another\noption is to have the interactions correspond to n-th order relations by combining nsenders in each bk.\nThe interactions could even have variable order, where each bkincludes all sender objects that interact\nwith a receiver, but would require a fRthan can handle variable-length inputs. These possibilities are\nbeyond the scope of this work, but are interesting future directions.\nImplementation The general de\ufb01nition of the IN in the previous section is agnostic to the choice\nof functions and algorithms, but we now outline a learnable implementation capable of reasoning\nabout complex systems with nonlinear relations and dynamics. We use standard deep neural network\nbuilding blocks, multilayer perceptrons (MLP), matrix operations, etc., which can be trained ef\ufb01ciently\nfrom data using gradient-based optimization, such as stochastic gradient descent.\nWe de\ufb01neOas aDS\u0002NOmatrix, whose columns correspond to the objects\u2019 DS-length state vectors."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1026-6db4-11ee-9b86-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-12", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "We de\ufb01neOas aDS\u0002NOmatrix, whose columns correspond to the objects\u2019 DS-length state vectors.\nThe relations are a triplet, R=hRr;Rs;Rai, whereRrandRsareNO\u0002NRbinary matrices which\nindex the receiver and sender objects, respectively, and Rais aDR\u0002NRmatrix whose DR-length\ncolumns represent the NRrelations\u2019 attributes. The j-th column of Rris a one-hot vector which\nindicates the receiver object\u2019s index; Rsindicates the sender similarly. For the graph in Fig. 1b,\nRr=h0 0\n1 1\n0 0i\nandRs=h1 0\n0 0\n0 1i\n. TheXis aDX\u0002NOmatrix, whose columns are DX-length vectors\nthat represent the external effect applied each of the NOobjects.\nThe marshalling function, m, computes the matrix products, ORrandORs, and concatenates them\nwithRa:m(G) = [ORr;ORs;Ra] =B.\nThe resulting Bis a(2DS+DR)\u0002NRmatrix, whose columns represent the interaction terms, bk,\nfor theNRrelations (we denote vertical and horizontal matrix concatenation with a semicolon and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1027-6db4-11ee-827b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-13", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "for theNRrelations (we denote vertical and horizontal matrix concatenation with a semicolon and\ncomma, respectively). The way mconstructs interaction terms can be modi\ufb01ed, as described in our\nExperiments section (3).\nTheBis input to\u001eR, which applies fR, an MLP, to each column. The output of fRis aDE-length\nvector,ek, a distributed representation of the effects. The \u001eRconcatenates the NReffects to form the\nDE\u0002NReffect matrix, E.\nTheG,X, andEare input toa, which computes the DE\u0002NOmatrix product, \u0016E=ERT\nr, whose\nj-th column is equivalent to the elementwise sum across all ekwhose corresponding relation has\nreceiver object, j. The \u0016Eis concatenated with OandX:a(G;X;E ) = [O;X;\u0016E] =C.\nThe resulting Cis a(DS+DX+DE)\u0002NOmatrix, whose NOcolumns represent the object states,\nexternal effects, and per-object aggregate interaction effects.\nTheCis input to\u001eO, which applies fO, another MLP, to each of the NOcolumns. The output of fO"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1028-6db4-11ee-91c0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-14", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "TheCis input to\u001eO, which applies fO, another MLP, to each of the NOcolumns. The output of fO\nis aDP-length vector, pj, and\u001eOconcatenates them to form the output matrix, P.\nTo infer abstract properties of a system, an additional \u001eAis appended and takes Pas input. The g\naggregation function performs an elementwise sum across the columns of Pto return aDP-length\nvector, \u0016P. The \u0016Pis input to\u001eA, another MLP, which returns a DA-length vector, q, that represents\nan abstract, global property of the system.\nTraining an IN requires optimizing an objective function over the learnable parameters of \u001eRand\u001eO.\nNote,mandainvolve matrix operations that do not contain learnable parameters.\nBecause\u001eRand\u001eOare shared across all relations and objects, respectively, training them is statistically ef\ufb01cient. This is similar to CNNs, which are very ef\ufb01cient due to their weight-sharing scheme.\nA CNN treats a local neighborhood of pixels as related, interacting entities: each pixel is effectively\na receiver object and its neighboring pixels are senders. The convolution operator is analogous to\n\u001eR, wherefRis the local linear/nonlinear kernel applied to each neighborhood. Skip connections,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1029-6db4-11ee-9ea4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-15", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "R, wherefRis the local linear/nonlinear kernel applied to each neighborhood. Skip connections,\nrecently popularized by residual networks, are loosely analogous to how the IN inputs Oto both\n\u001eRand\u001eO, though in CNNs relation- and object-centric reasoning are not delineated. But because\nCNNs exploit local interactions in a \ufb01xed way which is well-suited to the speci\ufb01c topology of images,\ncapturing longer-range dependencies requires either broad, insensitive convolution kernels, or deep\nstacks of layers, in order to implement suf\ufb01ciently large receptive \ufb01elds. The IN avoids this restriction\nby being able to process arbitrary neighborhoods that are explicitly speci\ufb01ed by the Rinput.\n4\n3 Experiments\nPhysical reasoning tasks Our experiments explored two types of physical reasoning tasks: predicting future states of a system, and estimating their abstract properties, speci\ufb01cally potential energy.\nWe evaluated the IN\u2019s ability to learn to make these judgments in three complex physical domains:\nn-body systems; balls bouncing in a box; and strings composed of springs that collide with rigid\nobjects. We simulated the 2D trajectories of the elements of these systems with a physics engine, and\nrecorded their sequences of states. See the Supplementary Material for full details."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b102a-6db4-11ee-9fcf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-16", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "objects. We simulated the 2D trajectories of the elements of these systems with a physics engine, and\nrecorded their sequences of states. See the Supplementary Material for full details.\nIn the n-body domain, such as solar systems, all nbodies exert distance- and mass-dependent\ngravitational forces on each other, so there were n(n\u00001)relations input to our model. Across\nsimulations, the objects\u2019 masses varied, while all other \ufb01xed attributes were held constant. The\ntraining scenes always included 6 bodies, and for testing we used 3, 6, and 12 bodies. In half of\nthe systems, bodies were initialized with velocities that would cause stable orbits, if not for the\ninteractions with other objects; the other half had random velocities.\nIn the bouncing balls domain, moving balls could collide with each other and with static walls.\nThe walls were represented as objects whose shape attribute represented a rectangle, and whose\ninverse-mass was 0. The relations input to the model were between the nobjects (which included the\nwalls), for (n(n\u00001)relations). Collisions are more dif\ufb01cult to simulate than gravitational forces, and\nthe data distribution was much more challenging: each ball participated in a collision on less than 1%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b102b-6db4-11ee-bad7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-17", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "the data distribution was much more challenging: each ball participated in a collision on less than 1%\nof the steps, following straight-line motion at all other times. The model thus had to learn that despite\nthere being a rigid relation between two objects, they only had meaningful collision interactions when\nthey were in contact. We also varied more of the object attributes \u2013 shape, scale and mass (as before)\n\u2013 as well as the coef\ufb01cient of restitution, which was a relation attribute. Training scenes contained 6\nballs inside a box with 4 variably sized walls, and test scenes contained either 3, 6, or 9 balls.\nThe string domain used two types of relations (indicated in rk), relation structures that were more\nsparse and speci\ufb01c than all-to-all, as well as variable external effects. Each scene contained a string,\ncomprised of masses connected by springs, and a static, rigid circle positioned below the string. The\nnmasses had spring relations with their immediate neighbors ( 2(n\u00001)), and all masses had rigid\nrelations with the rigid object ( 2n). Gravitational acceleration, with a magnitude that was varied\nacross simulation runs, was applied so that the string always fell, usually colliding with the static\nobject. The gravitational acceleration was an external input (not to be confused with the gravitational"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b102c-6db4-11ee-8115-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-18", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "across simulation runs, was applied so that the string always fell, usually colliding with the static\nobject. The gravitational acceleration was an external input (not to be confused with the gravitational\nattraction relations in the n-body experiments). Each training scene contained a string with 15 point\nmasses, and test scenes contained either 5, 15, or 30 mass strings. In training, one of the point masses\nat the end of the string, chosen at random, was always held static, as if pinned to the wall, while the\nother masses were free to move. In the test conditions, we also included strings that had both ends\npinned, and no ends pinned, to evaluate generalization.\nOur model takes as input the state of each system, G, decomposed into the objects, O(e.g., n-body\nobjects, balls, walls, points masses that represented string elements), and their physical relations, R\n(e.g., gravitational attraction, collisions, springs), as well as the external effects, X(e.g., gravitational\nacceleration). Each object state, oj, could be further divided into a dynamic state component\n(e.g., position and velocity) and a static attribute component (e.g., mass, size, shape). The relation\nattributes,Ra, represented quantities such as the coef\ufb01cient of restitution, and spring constant. The"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b102d-6db4-11ee-9d76-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-19", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "attributes,Ra, represented quantities such as the coef\ufb01cient of restitution, and spring constant. The\ninput represented the system at the current time. The prediction experiment\u2019s target outputs were the\nvelocities of the objects on the subsequent time step, and the energy estimation experiment\u2019s targets\nwere the potential energies of the system on the current time step. We also generated multi-step\nrollouts for the prediction experiments (Fig. 2), to assess the model\u2019s effectiveness at creating visually\nrealistic simulations. The output velocity, vt, on time step tbecame the input velocity on t+ 1, and\nthe position at t+ 1was updated by the predicted velocity at t.\nData Each of the training, validation, test data sets were generated by simulating 2000 scenes\nover 1000 time steps, and randomly sampling 1 million, 200k, and 200k one-step input/target pairs,\nrespectively. The model was trained for 2000 epochs, randomly shuf\ufb02ing the data indices between\neach. We used mini-batches of 100, and balanced their data distributions so the targets had similar\nper-element statistics. The performance reported in the Results was measured on held-out test data.\nWe explored adding a small amount of Gaussian noise to 20% of the data\u2019s input positions and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b102e-6db4-11ee-a7ad-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-20", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "We explored adding a small amount of Gaussian noise to 20% of the data\u2019s input positions and\nvelocities during the initial phase of training, which was reduced to 0% from epochs 50 to 250. The\n5\nnoise std. dev. was 0:05\u0002the std. dev. of each element\u2019s values across the dataset. It allowed the\nmodel to experience physically impossible states which could not have been generated by the physics\nengine, and learn to project them back to nearby, possible states. Our error measure did not re\ufb02ect\nclear differences with or without noise, but rollouts from models trained with noise were slightly\nmore visually realistic, and static objects were less subject to drift over many steps.\nModel architecture ThefRandfOMLPs contained multiple hidden layers of linear transforms\nplus biases, followed by recti\ufb01ed linear units (ReLUs), and an output layer that was a linear transform\nplus bias. The best model architecture was selected by a grid search over layer sizes and depths. All\ninputs (except RrandRs) were normalized by centering at the median and rescaling the 5th and 95th\npercentiles to -1 and 1. All training objectives and test measures used mean squared error (MSE)\nbetween the model\u2019s prediction and the ground truth target."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b102f-6db4-11ee-9433-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-21", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "percentiles to -1 and 1. All training objectives and test measures used mean squared error (MSE)\nbetween the model\u2019s prediction and the ground truth target.\nAll prediction experiments used the same architecture, with parameters selected by a hyperparameter\nsearch. The fRMLP had four, 150-length hidden layers, and output length DE= 50 . ThefOMLP\nhad one, 100-length hidden layer, and output length DP= 2, which targeted the x;y-velocity. The\nmandawere customized so that the model was invariant to the absolute positions of objects in the\nscene. Themconcatenated three terms for each bk: the difference vector between the dynamic states\nof the receiver and sender, the concatenated receiver and sender attribute vectors, and the relation\nattribute vector. The aonly outputs the velocities, not the positions, for input to \u001eO.\nThe energy estimation experiments used the IN from the prediction experiments with an additional\n\u001eAMLP which had one, 25-length hidden layer. Its Pinputs\u2019 columns were length DP= 10 , and\nits output length was DA= 1.\nWe optimized the parameters using Adam [ 13], with a waterfall schedule that began with a learning\nrate of 0:001and down-scaled the learning rate by 0:8each time the validation error, estimated over"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1030-6db4-11ee-9f74-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-22", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "rate of 0:001and down-scaled the learning rate by 0:8each time the validation error, estimated over\na window of 40epochs, stopped decreasing.\nTwo forms of L2 regularization were explored: one applied to the effects, E, and another to the model\nparameters. Regularizing Eimproved generalization to different numbers of objects and reduced\ndrift over many rollout steps. It likely incentivizes sparser communication between the \u001eRand\u001eO,\nprompting them to operate more independently. Regularizing the parameters generally improved\nperformance and reduced over\ufb01tting. Both penalty factors were selected by a grid search.\nFew competing models are available in the literature to compare our model against, but we considered\nseveral alternatives: a constant velocity baseline which output the input velocity; an MLP baseline,\nwith two 300-length hidden layers, which took as input a \ufb02attened vector of all of the input data; and\na variant of the IN with the \u001eRcomponent removed (the interaction effects, E, was set to a 0-matrix).\n4 Results\nPrediction experiments Our results show that the IN can predict the next-step dynamics of our task\ndomains very accurately after training, with orders of magnitude lower test error than the alternative"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1031-6db4-11ee-9b24-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-23", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "4 Results\nPrediction experiments Our results show that the IN can predict the next-step dynamics of our task\ndomains very accurately after training, with orders of magnitude lower test error than the alternative\nmodels (Fig. 3a, d and g, and Table 2). Because the dynamics of each domain depended crucially on\ninteractions among objects, the IN was able to learn to exploit these relationships for its predictions.\nThe dynamics-only IN had no mechanism for processing interactions, and performed similarly to the\nconstant velocity model. The baseline MLP\u2019s connectivity makes it possible, in principle, for it to\nlearn the interactions, but that would require learning how to use the relation indices to selectively\nprocess the interactions. It would also not bene\ufb01t from sharing its learning across relations and\nobjects, instead being forced to approximate the interactive dynamics in parallel for each objects.\nThe IN also generalized well to systems with fewer and greater numbers of objects (Figs. 3b-c, e-f\nand h-k, and Table SM1 in Supp. Mat.). For each domain, we selected the best IN model from the\nsystem size on which it was trained, and evaluated its MSE on a different system size. When tested\non smaller n-body and spring systems from those on which it was trained, its performance actually"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1032-6db4-11ee-b781-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-24", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "on smaller n-body and spring systems from those on which it was trained, its performance actually\nexceeded a model trained on the smaller system. This may be due to the model\u2019s ability to exploit its\ngreater experience with how objects and relations behave, available in the more complex system.\nWe also found that the IN trained on single-step predictions can be used to simulate trajectories over\nthousands of steps very effectively, often tracking the ground truth closely, especially in the n-body\nand string domains. When rendered into images and videos, the model-generated trajectories are\n6\nTrue Model True Model True ModelTime Time TimeFigure 2: Prediction rollouts. Each column contains three panels of three video frames (with motion blur),\neach spanning 1000 rollout steps. Columns 1-2 are ground truth and model predictions for n-body systems, 3-4\nare bouncing balls, and 5-6 are strings. Each model column was generated by a single model, trained on the\nunderlying states of a system of the size in the top panel. The middle and bottom panels show its generalization\nto systems of different sizes and structure. For n-body, the training was on 6 bodies, and generalization was to 3\nand 12 bodies. For balls, the training was on 6 balls, and generalization was to 3 and 9 balls. For strings, the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1033-6db4-11ee-b119-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-25", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "and 12 bodies. For balls, the training was on 6 balls, and generalization was to 3 and 9 balls. For strings, the\ntraining was on 15 masses with 1 end pinned, and generalization was to 30 masses with 0 and 2 ends pinned.\nThe URLs to the full videos of each rollout are in Table 1.\n7\nTable 1: Rollout video URLs\nSystem True Model\nn-body 6 https://youtu.be/otIGNTFJwpU https://youtu.be/CXzubiwu4GM\nn-body 3 https://youtu.be/zOTVzziJz24 https://youtu.be/3bFrbjdTc6s\nn-body 12 https://youtu.be/JgSRaYmQyNY https://youtu.be/Rl44jdaIIiw\nBalls 6 https://youtu.be/9c-UPXSjOHI https://youtu.be/SV741g_qp8M\nBalls 3 https://youtu.be/op2d9IHh3Ak https://youtu.be/i0lOn_v8xVg\nBalls 9 https://youtu.be/oQoquoVJ8V0 https://youtu.be/41y_saKvi3s"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1034-6db4-11ee-b4d0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-26", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "Balls 9 https://youtu.be/oQoquoVJ8V0 https://youtu.be/41y_saKvi3s\nString 15, 1 pinned https://youtu.be/xiz49xOPX9c https://youtu.be/pbk6VM_XQb4\nString 30, 0 pinned https://youtu.be/TpixFr_ekAE https://youtu.be/tsE7cev0w08\nString 30, 2 pinned https://youtu.be/x6g1ogRG31o https://youtu.be/ZDCZyqyrv5o\n10-2\n10-3g. 15, 1h. 5, 1i. 30, 1j. 15, 0k. 15, 2String\n1\n10-110102MSE (log-scale)a. 6b. 3c. 12n-body\n10-210-1\n10-3d. 6e. 3f. 9Balls\nIN (15 obj, 1 pin) IN (5 obj, 1 pin)\nIN (15 obj, 0 pin)IN (30 obj, 1 pin)\nIN (15 obj, 2 pin)IN (3 obj) IN (12 obj) IN (6 obj)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1035-6db4-11ee-b51c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-27", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "IN (15 obj, 0 pin)IN (30 obj, 1 pin)\nIN (15 obj, 2 pin)IN (3 obj) IN (12 obj) IN (6 obj)\nConstant velocity Baseline MLP Dynamics-only ININ (3 obj) IN (9 obj) IN (6 obj)10-2\nFigure 3: Prediction experiment accuracy and generalization. Each colored bar represents the MSE between a\nmodel\u2019s predicted velocity and the ground truth physics engine\u2019s (the y-axes are log-scaled). Sublots ( a-c) show\nn-body performance, ( d-f) show balls, and ( g-k) show string. The leftmost subplots in each ( a, d, g ) for each\ndomain compare the constant velocity model (black), baseline MLP (grey), dynamics-only IN (red), and full IN\n(blue). The other panels show the IN\u2019s generalization performance to different numbers and con\ufb01gurations of\nobjects, as indicated by the subplot titles. For the string systems, the numbers correspond to: (the number of\nmasses, how many ends were pinned).\nusually visually indistinguishable from those of the ground truth physics engine (Fig. 2; see Table 1\nfor URLs to the full videos of each rollout). This is not to say that given the same initial conditions,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1036-6db4-11ee-a5a4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-28", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "for URLs to the full videos of each rollout). This is not to say that given the same initial conditions,\nthey cohere perfectly: the dynamics are highly nonlinear and imperceptible prediction errors by the\nmodel can rapidly lead to large differences in the systems\u2019 states. But the incoherent rollouts do\nnot violate people\u2019s expectations, and might be roughly on par with people\u2019s understanding of these\ndomains.\nEstimating abstract properties We trained an abstract-estimation variant of our model to predict\npotential energies in the n-body and string domains (the ball domain\u2019s potential energies were always\n0), and found it was much more accurate (n-body MSE 1:4, string MSE 1:1) than the MLP baseline\n(n-body MSE 19, string MSE 425). The IN presumably learns the gravitational and spring potential\nenergy functions, applies them to the relations in their respective domains, and combines the results.\n5 Discussion\nWe introduced interaction networks as a \ufb02exible and ef\ufb01cient model for explicit reasoning about\nobjects and relations in complex systems. Our results provide surprisingly strong evidence of their\nability to learn accurate physical simulations and generalize their training to novel systems with\ndifferent numbers and con\ufb01gurations of objects and relations. They could also learn to infer abstract"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1037-6db4-11ee-bdcc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-29", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "ability to learn accurate physical simulations and generalize their training to novel systems with\ndifferent numbers and con\ufb01gurations of objects and relations. They could also learn to infer abstract\nproperties of physical systems, such as potential energy. The alternative models we tested performed\nmuch more poorly, with orders of magnitude greater error. Simulation over rich mental models is\nthought to be a crucial mechanism of how humans reason about physics and other complex domains\n[5,12,10], and Battaglia et al. [ 3] recently posited a simulation-based \u201cintuitive physics engine\u201d\nmodel to explain human physical scene understanding. Our interaction network implementation is the\n\ufb01rst learnable physics engine that can scale up to real-world problems, and is a promising template for\nnew AI approaches to reasoning about other physical and mechanical systems, scene understanding,\nsocial perception, hierarchical planning, and analogical reasoning.\nIn the future, it will be important to develop techniques that allow interaction networks to handle\nvery large systems with many interactions, such as by culling interaction computations that will have\n8\nTable 2: Prediction experiment MSEs\nDomain Constant velocity Baseline Dynamics-only IN IN\nn-body 82 79 76 0.25\nBalls 0.074 0.072 0.074 0.0020\nString 0.018 0.016 0.017 0.0011"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1038-6db4-11ee-9128-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-30", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "Balls 0.074 0.072 0.074 0.0020\nString 0.018 0.016 0.017 0.0011\nnegligible effects. The interaction network may also serve as a powerful model for model-predictive\ncontrol inputting active control signals as external effects \u2013 because it is differentiable, it naturally\nsupports gradient-based planning. It will also be important to prepend a perceptual front-end that\ncan infer from objects and relations raw observations, which can then be provided as input to an\ninteraction network that can reason about the underlying structure of a scene. By adapting the\ninteraction network into a recurrent neural network, even more accurate long-term predictions might\nbe possible, though preliminary tests found little bene\ufb01t beyond its already-strong performance.\nBy modifying the interaction network to be a probabilistic generative model, it may also support\nprobabilistic inference over unknown object properties and relations.\nBy combining three powerful tools from the modern machine learning toolkit \u2013 relational reasoning\nover structured knowledge, simulation, and deep learning \u2013 interaction networks offer \ufb02exible,\naccurate, and ef\ufb01cient learning and inference in challenging domains. Decomposing complex\nsystems into objects and relations, and reasoning about them explicitly, provides for combinatorial"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1039-6db4-11ee-877f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-31", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "accurate, and ef\ufb01cient learning and inference in challenging domains. Decomposing complex\nsystems into objects and relations, and reasoning about them explicitly, provides for combinatorial\ngeneralization to novel contexts, one of the most important future challenges for AI, and a crucial\nstep toward closing the gap between how humans and machines think.\nReferences\n[1]J Andreas, M Rohrbach, T Darrell, and D Klein. Learning to compose neural networks for question\nanswering. NAACL , 2016.\n[2]D Baraff. Physically based modeling: Rigid body simulation. SIGGRAPH Course Notes, ACM SIGGRAPH ,\n2(1), 2001.\n[3]PW Battaglia, JB Hamrick, and JB Tenenbaum. Simulation as an engine of physical scene understanding.\nProceedings of the National Academy of Sciences , 110(45):18327\u201318332, 2013.\n[4]Michael B. Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum. A compositional\nobject-based approach to learning physical dynamics. Under Review , 2016.\n[5] K.J.W. Craik. The nature of explanation . Cambridge University Press, 1943."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b103a-6db4-11ee-8682-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-32", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "object-based approach to learning physical dynamics. Under Review , 2016.\n[5] K.J.W. Craik. The nature of explanation . Cambridge University Press, 1943.\n[6]K Fragkiadaki, P Agrawal, S Levine, and J Malik. Learning visual predictive models of physics for playing\nbilliards. ICLR , 2016.\n[7]Z. Ghahramani. Probabilistic machine learning and arti\ufb01cial intelligence. Nature , 521(7553):452\u2013459,\n2015.\n[8]R Grzeszczuk, D Terzopoulos, and G Hinton. Neuroanimator: Fast neural network emulation and control of\nphysics-based models. In Proceedings of the 25th annual conference on Computer graphics and interactive\ntechniques , pages 9\u201320. ACM, 1998.\n[9]P.J Hayes. The naive physics manifesto . Universit\u00e9 de Gen\u00e8ve, Institut pour les \u00e9tudes s \u00e9 mantiques et\ncognitives, 1978.\n[10] M. Hegarty. Mechanical reasoning by mental simulation. TICS , 8(6):280\u2013285, 2004.\n[11] M Jaderberg, K Simonyan, and A Zisserman. Spatial transformer networks. In NIPS , pages 2008\u20132016,\n2015."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b103b-6db4-11ee-ba51-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-33", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "[11] M Jaderberg, K Simonyan, and A Zisserman. Spatial transformer networks. In NIPS , pages 2008\u20132016,\n2015.\n[12] P.N. Johnson-Laird. Mental models: towards a cognitive science of language, inference, and consciousness ,\nvolume 6. Cambridge University Press, 1983.\n[13] D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR , 2015.\n[14] L Ladick\u00fd, S Jeong, B Solenthaler, M Pollefeys, and M Gross. Data-driven \ufb02uid simulations using\nregression forests. ACM Transactions on Graphics (TOG) , 34(6):199, 2015.\n[15] B Lake, T Ullman, J Tenenbaum, and S Gershman. Building machines that learn and think like people.\narXiv:1604.00289 , 2016.\n[16] Y LeCun, Y Bengio, and G Hinton. Deep learning. Nature , 521(7553):436\u2013444, 2015.\n9\n[17] A Lerer, S Gross, and R Fergus. Learning physical intuition of block towers by example. arXiv:1603.01312 ,\n2016."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b103c-6db4-11ee-9ebc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-34", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "9\n[17] A Lerer, S Gross, and R Fergus. Learning physical intuition of block towers by example. arXiv:1603.01312 ,\n2016.\n[18] W Li, S Azimi, A Leonardis, and M Fritz. To fall or not to fall: A visual approach to physical stability\nprediction. arXiv:1604.00066 , 2016.\n[19] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks.\narXiv preprint arXiv:1511.05493 , 2015.\n[20] R Mottaghi, H Bagherinezhad, M Rastegari, and A Farhadi. Newtonian image understanding: Unfolding\nthe dynamics of objects in static images. arXiv:1511.04048 , 2015.\n[21] R Mottaghi, M Rastegari, A Gupta, and A Farhadi. \" what happens if...\" learning to predict the effect of\nforces in images. arXiv:1603.05600 , 2016.\n[22] SE Reed and N de Freitas. Neural programmer-interpreters. ICLR , 2016."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b103d-6db4-11ee-9553-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-35", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "[22] SE Reed and N de Freitas. Neural programmer-interpreters. ICLR , 2016.\n[23] F. Scarselli, M. Gori, A.C. Tsoi, M. Hagenbuchner, and G. Monfardini. The graph neural network model.\nIEEE Trans. Neural Networks , 20(1):61\u201380, 2009.\n[24] J. Schmidhuber. Deep learning in neural networks: An overview. Neural Networks , 61:85\u2013117, 2015.\n[25] R Socher, E Huang, J Pennin, C Manning, and A Ng. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In in NIPS , pages 801\u2013809, 2011.\n[26] E Spelke, K Breinlinger, J Macomber, and K Jacobson. Origins of knowledge. Psychol. Rev. , 99(4):605\u2013\n632, 1992.\n[27] I Sutskever and GE Hinton. Using matrices to model symbolic relationship. In D. Koller, D. Schuurmans,\nY . Bengio, and L. Bottou, editors, NIPS , pages 1593\u20131600. 2009."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b103e-6db4-11ee-839b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-36", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "Y . Bengio, and L. Bottou, editors, NIPS , pages 1593\u20131600. 2009.\n[28] J.B. Tenenbaum, C. Kemp, T.L. Grif\ufb01ths, and N.D. Goodman. How to grow a mind: Statistics, structure,\nand abstraction. Science , 331(6022):1279, 2011.\n[29] P Winston and B Horn. The psychology of computer vision , volume 73. McGraw-Hill New York, 1975.\n[30] J Wu, I Yildirim, JJ Lim, B Freeman, and J Tenenbaum. Galileo: Perceiving physical object properties by\nintegrating a physics engine with deep learning. In NIPS , pages 127\u2013135, 2015.\n10\nA Supplementary material\nA.1 Experimental details\nA.1.1 Physics engine details\nEvery simulated trajectory contained the states of the objects in the system on each frame of a sequence of\n1000 one-millisecond time steps. The parameters of each system were chosen so that a diverse set of dynamics\ncould unfold within the trajectory. On each step, the physics engine took as input the current state of the\nsystem, calculated the forces associated with each inter-entity interaction, and applied them to the individual"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b103f-6db4-11ee-8210-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-37", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "system, calculated the forces associated with each inter-entity interaction, and applied them to the individual\nentities as accelerations by dividing by the entity\u2019s mass, parameterized as inverse-mass, a=Fm\u00001in both the\nengine and model input. This also allows static objects to be represented as having m\u00001= 0. The previous\npositions and velocities, and newly computed accelerations, were input to an Euler integrator to update the\ncurrent velocities and positions of the entities. By using a custom engine we were able to have perfect control\nover the details of the simulation, and use one engine for all physical domains. It produced trajectory rollouts\nthat were indistinguishable from those of an off-the-shelf simulation engine, but was highly ef\ufb01cient because it\ncould perform thousands of runs in parallel on a GPU, allowing millions of simulated steps to be generated each\nsecond.\nA.1.2 Physical domains\nn-body All objects in n-body systems exerted gravitational forces on each other, which were a function of\nthe objects\u2019 pairwise distances and masses, giving rise to very complex dynamics that are highly sensitive to\ninitial conditions. Across simulation runs, the objects\u2019 masses varied, while all other non-dynamic variables\nwere held constant. The gravitational constant was set to ensure that objects could move across several hundred"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1040-6db4-11ee-a168-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-38", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "were held constant. The gravitational constant was set to ensure that objects could move across several hundred\nmeters within the 1000 step rollout. The training scenes always included 6 objects, and for testing we used 3,\n6, and 12 objects. The masses were uniformly sampled from [0:02;9]kg, their shapes were points, and their\ninitial positions were randomly sampled from all angles, with a distance in [10;100] m. We included two classes\nof scenes. The \ufb01rst, orbit systems, had one object (the star), initialized at position (0;0), with zero velocity\nand a mass of 100kg. The planets\u2019 velocities were initialized such that they would have stable orbits around\nthe star, if not for their interactions with other planets. The second, non-orbit systems, sampled initial x- and\ny-velocity components from [\u00003;3]m/s. The objects would typically begin attracting, and gave rise to complex\nsemi-periodic behavior.\nAn n-body system is a highly nonlinear (chaotic) dynamical system, which means they are highly sensitive to\ninitial conditions and extended predictions of their states are not possible under even small perturbations. The\nrelations between the objects corresponded to gravitational attraction. Between simulation runs, the masses of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1041-6db4-11ee-b562-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-39", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "initial conditions and extended predictions of their states are not possible under even small perturbations. The\nrelations between the objects corresponded to gravitational attraction. Between simulation runs, the masses of\nthe objects were varied, while all other non-dynamic variables were held constant (e.g., gravitational constant) or\nwere not meaningful (e.g., object scales and shapes). The gravitational force from object itojwas computed as,\nFij=Gmimj(xi\u0000xj)\nkxi\u0000xjk3, where Gis the gravitational constant. The denominator was clipped so that forces could\nnot go too high as the distances between objects went to zero. All forces applied to each object were summed to\nyield the per-object total forces.\nBouncing balls The bouncing balls domain still had all-to-all object relations\u2013collisions\u2013and any object\ncould collide with any other, including the walls. But colliding objects are more dif\ufb01cult to simulate than the\ngravitational interactions in n-body systems, and our bouncing balls domain also included more variability in\nobject attributes, such as shape, scale, and mass (as before), as well as relation attributes, such as the coef\ufb01cient\nof restitution. The data distribution was much more challenging: for more than 99% of the time steps, a ball was"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1042-6db4-11ee-be3a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-40", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "of restitution. The data distribution was much more challenging: for more than 99% of the time steps, a ball was\nnot in contact with any others, and its next-step velocity equaled its current velocity. For the remaining steps,\nhowever, collisions caused next-step velocities that was a complex function of its state and the state of the object\nit collides with. The training scene contained 6 balls inside a box with 4 walls, and test scenes contained either 3,\n6, or 9 balls. The balls\u2019 radii were sampled uniformly from [0:1;0:3]m, and masses from [0:75;1:25]kg. The\nwalls were static rectangular boxes, positioned so that the horizontal and vertical lengths of area they enclosed\nvaried independently between [1;3]m. The balls\u2019 initial positions were randomly sampled so that all balls \ufb01t\nwithin the box and did not interpenetrate any other object, with initial x- andy-velocity components sampled\nuniformly from [\u00005;5]m/s. The restitutions, an attribute of their collision relations, was sampled uniformly\nfrom[0:4;1].\nRigid body collision systems are highly nonlinear (chaotic) dynamical systems, as well. Collision forces were"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1043-6db4-11ee-a2d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-41", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "from[0:4;1].\nRigid body collision systems are highly nonlinear (chaotic) dynamical systems, as well. Collision forces were\napplied between objects when they began to interpenetrate, using two-step process: collision detection between\nall objects, and resolution of detected collisions. A detected collision between two objects meant that their\nshapes overlapped and their velocities were causing them to approach each other. To resolve a partially inelastic\ncollision, the post-collision velocities of each object were computed, and forces appropriate to effect these\nvelocities on the subsequent time step were then calculated and applied the the objects. This resulted in realistic\nbouncing ball trajectories.\n11\nTable 3: Prediction experiments - Generalization MSEs\nn-body Balls String\nModel 3 12 3 9 5, 1 30, 1 15, 0 15, 2\nConst. vel. 45 166 0.054 0.084 0.0080 0.010 0.020 0.012\nIN (within) 0.21 5.2 0.0053 0.0019 0.0030 0.00066 0.0010 0.00097\nIN (transfer) 0.052 7.8 0.0053 0.004 0.00052 0.00074 0.0014 0.00097"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1044-6db4-11ee-b484-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-42", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "IN (transfer) 0.052 7.8 0.0053 0.004 0.00052 0.00074 0.0014 0.00097\nString Our string domain used multiple types of relations (springs and collisions), relation structures that were\nmore sparse and speci\ufb01c than all-to-all, and variable external effects. Each scene contained one string, comprised\nof point masses connected by springs, and one static, rigid circle positioned below the string. Gravitational\nacceleration, varied across simulation runs, was applied so that the string always fell, usually colliding with\nthe static object. Each training scene contained a string with 15 point masses, and test scenes contained either\n5, 15, or 30 mass strings. In training, one of the point masses at the end of the string, chosen at random, was\nalways held static, as if pinned to the wall, while the other masses were free to move. In the test conditions, we\nalso included strings that had both ends pinned, and no ends pinned, to evaluate generalization. The string\u2019s\nmasses were sampled from [0:05;0:15]kg. The springs\u2019 rest lengths were set to 0:2m, spring constants to 100,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1045-6db4-11ee-a78b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-43", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "masses were sampled from [0:05;0:15]kg. The springs\u2019 rest lengths were set to 0:2m, spring constants to 100,\nand damping factors to 0:001. The static object\u2019s x-position was sampled from [\u00000:5;0:5]m,y-position was\nsampled from [\u00001;\u00000:5]m, and radius from [0:2;0:4]m. The string mass-rigid object coef\ufb01cient of restitution\nwas sampled from [0;1]. The gravitational acceleration applied to all non-static objects was sampled from\n[\u000030;\u00005]m/s2.\nThe spring force from object itojwas computed using Hooke\u2019s law as, F=Cs(1\u0000L\nkxi\u0000xjk)(xi\u0000xj),\nwhere Csis the spring constant and Lis the spring\u2019s rest length. A damping factor, which was proportional to\nthe difference in the objects\u2019 velocities, was also applied. Collision forces between the string\u2019s masses and the\nstatic, rigid object were calculated as described in the bouncing balls section above.\nA.2 Results details\nThe prediction experiment\u2019s generalization performance MSEs are shown Table 3. The URL links for the videos"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1046-6db4-11ee-8f56-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1612.00222-44", "title": "Interaction Networks for Learning about Objects, Relations and Physics", "source": "http://arxiv.org/pdf/1612.00222", "start_index": 0, "chroma:document": "A.2 Results details\nThe prediction experiment\u2019s generalization performance MSEs are shown Table 3. The URL links for the videos\nassociated with all still frames in Figure 2 are in Table 1.\n12"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1047-6db4-11ee-9f55-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-0", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Mask R-CNN\nKaiming He Georgia Gkioxari Piotr Doll \u00b4ar Ross Girshick\nFacebook AI Research (FAIR)\nAbstract\nWe present a conceptually simple, \ufb02exible, and general\nframework for object instance segmentation. Our approach\nef\ufb01ciently detects objects in an image while simultaneously\ngenerating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster\nR-CNN by adding a branch for predicting an object mask in\nparallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small\noverhead to Faster R-CNN, running at 5 fps. Moreover,\nMask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework.\nWe show top results in all three tracks of the COCO suite\nof challenges, including instance segmentation, boundingbox object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the\nCOCO 2016 challenge winners. We hope our simple and\neffective approach will serve as a solid baseline and help\nease future research in instance-level recognition. Code\nhas been made available at: https://github.com/"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1048-6db4-11ee-ac07-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-1", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "effective approach will serve as a solid baseline and help\nease future research in instance-level recognition. Code\nhas been made available at: https://github.com/\nfacebookresearch/Detectron .\n1. Introduction\nThe vision community has rapidly improved object detection and semantic segmentation results over a short period of time. In large part, these advances have been driven\nby powerful baseline systems, such as the Fast/Faster RCNN [12, 36] and Fully Convolutional Network (FCN) [30]\nframeworks for object detection and semantic segmentation, respectively. These methods are conceptually intuitive\nand offer \ufb02exibility and robustness, together with fast training and inference time. Our goal in this work is to develop a\ncomparably enabling framework for instance segmentation .\nInstance segmentation is challenging because it requires\nthe correct detection of all objects in an image while also\nprecisely segmenting each instance. It therefore combines\nelements from the classical computer vision tasks of object detection , where the goal is to classify individual objects and localize each using a bounding box, and semantic\nRoIAlignRoIAlignclass\nbox\nconvconv\n convconvFigure 1. The Mask R-CNN framework for instance segmentation.\nsegmentation , where the goal is to classify each pixel into"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1049-6db4-11ee-9f8c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-2", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "box\nconvconv\n convconvFigure 1. The Mask R-CNN framework for instance segmentation.\nsegmentation , where the goal is to classify each pixel into\na \ufb01xed set of categories without differentiating object instances.1Given this, one might expect a complex method\nis required to achieve good results. However, we show that\na surprisingly simple, \ufb02exible, and fast system can surpass\nprior state-of-the-art instance segmentation results.\nOur method, called Mask R-CNN , extends Faster R-CNN\n[36] by adding a branch for predicting segmentation masks\non each Region of Interest (RoI), in parallel with the existing branch for classi\ufb01cation and bounding box regression (Figure 1). The mask branch is a small FCN applied\nto each RoI, predicting a segmentation mask in a pixel-topixel manner. Mask R-CNN is simple to implement and\ntrain given the Faster R-CNN framework, which facilitates\na wide range of \ufb02exible architecture designs. Additionally,\nthe mask branch only adds a small computational overhead,\nenabling a fast system and rapid experimentation.\nIn principle Mask R-CNN is an intuitive extension of\nFaster R-CNN, yet constructing the mask branch properly"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b104a-6db4-11ee-8318-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-3", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "enabling a fast system and rapid experimentation.\nIn principle Mask R-CNN is an intuitive extension of\nFaster R-CNN, yet constructing the mask branch properly\nis critical for good results. Most importantly, Faster RCNN was not designed for pixel-to-pixel alignment between network inputs and outputs. This is most evident in\nhow RoIPool [18, 12], the de facto core operation for attending to instances, performs coarse spatial quantization\nfor feature extraction. To \ufb01x the misalignment, we propose a simple, quantization-free layer, called RoIAlign , that\nfaithfully preserves exact spatial locations. Despite being\n1Following common terminology, we use object detection to denote\ndetection via bounding boxes , not masks, and semantic segmentation to\ndenote per-pixel classi\ufb01cation without differentiating instances. Yet we\nnote that instance segmentation is both semantic and a form of detection.\n1arXiv:1703.06870v3  [cs.CV]  24 Jan 2018\ndining table.96person1.00person1.00 person1.00 person1.00 person1.00person1.00\nperson1.00person.94\nbottle.99\nbottle.99\nbottle.99\nmotorcycle1.00motorcycle1.00person1.00\nperson1.00"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b104b-6db4-11ee-ae4d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-4", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "bottle.99\nbottle.99\nbottle.99\nmotorcycle1.00motorcycle1.00person1.00\nperson1.00\nperson.96person1.00person.83person.96\nperson.98 person.90person.92person.99person.91\nbus.99\nperson1.00\nperson1.00person1.00\nbackpack.93person1.00person.99\nperson1.00\nbackpack.99\nperson.99person.98person.89person.95\nperson1.00\nperson1.00\ncar1.00traffic light .96\nperson.96truck1.00person.99\ncar.99person.85\nmotorcycle.95car.99car.92person.99person1.00traffic light .92 traffic light .84traffic light .95\ncar.93person.87\nperson1.00\nperson1.00umbrella.98\numbrella.98\nbackpack1.00\nhandba g.96\nelephant1.00person1.00person1.00 person.99\nsheep1. 00person1.00\nsheep.99sheep.91 sheep1. 00\nsheep.99sheep.99sheep.95person.99"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b104c-6db4-11ee-a792-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-5", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "sheep1. 00person1.00\nsheep.99sheep.91 sheep1. 00\nsheep.99sheep.99sheep.95person.99\nsheep1. 00sheep.96sheep.99sheep.99\nsheep.96\nsheep.96sheep.96sheep.86\nsheep.82sheep.93\ndining table.99\nchair.99chair.90\nchair.99chair.98\nchair.96chair.86chair.99\nbowl.81chair.96tv.99\nbottle.99\nwine glass.99wine glass1.00bowl.85\nknife.83wine glass1.00 wine glass.93wine glass.97\nfork.95Figure 2. Mask R-CNN results on the COCO test set. These results are based on ResNet-101 [19], achieving a mask AP of 35.7 and\nrunning at 5 fps. Masks are shown in color, and bounding box, category, and con\ufb01dences are also shown.\na seemingly minor change, RoIAlign has a large impact: it\nimproves mask accuracy by relative 10% to 50%, showing\nbigger gains under stricter localization metrics. Second, we\nfound it essential to decouple mask and class prediction: we\npredict a binary mask for each class independently, without"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b104d-6db4-11ee-aff5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-6", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "bigger gains under stricter localization metrics. Second, we\nfound it essential to decouple mask and class prediction: we\npredict a binary mask for each class independently, without\ncompetition among classes, and rely on the network\u2019s RoI\nclassi\ufb01cation branch to predict the category. In contrast,\nFCNs usually perform per-pixel multi-class categorization,\nwhich couples segmentation and classi\ufb01cation, and based\non our experiments works poorly for instance segmentation.\nWithout bells and whistles, Mask R-CNN surpasses all\nprevious state-of-the-art single-model results on the COCO\ninstance segmentation task [28], including the heavilyengineered entries from the 2016 competition winner. As\na by-product, our method also excels on the COCO object\ndetection task. In ablation experiments, we evaluate multiple basic instantiations, which allows us to demonstrate its\nrobustness and analyze the effects of core factors.\nOur models can run at about 200ms per frame on a GPU,\nand training on COCO takes one to two days on a single\n8-GPU machine. We believe the fast train and test speeds,\ntogether with the framework\u2019s \ufb02exibility and accuracy, will\nbene\ufb01t and ease future research on instance segmentation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b104e-6db4-11ee-ada1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-7", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "together with the framework\u2019s \ufb02exibility and accuracy, will\nbene\ufb01t and ease future research on instance segmentation.\nFinally, we showcase the generality of our framework\nvia the task of human pose estimation on the COCO keypoint dataset [28]. By viewing each keypoint as a one-hot\nbinary mask, with minimal modi\ufb01cation Mask R-CNN can\nbe applied to detect instance-speci\ufb01c poses. Mask R-CNN\nsurpasses the winner of the 2016 COCO keypoint competition, and at the same time runs at 5 fps. Mask R-CNN,\ntherefore, can be seen more broadly as a \ufb02exible framework\nforinstance-level recognition and can be readily extended\nto more complex tasks.\nWe have released code to facilitate future research.2. Related Work\nR-CNN: The Region-based CNN (R-CNN) approach [13]\nto bounding-box object detection is to attend to a manageable number of candidate object regions [42, 20] and evaluate convolutional networks [25, 24] independently on each\nRoI. R-CNN was extended [18, 12] to allow attending to\nRoIs on feature maps using RoIPool, leading to fast speed\nand better accuracy. Faster R-CNN [36] advanced this"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b104f-6db4-11ee-ac75-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-8", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "RoIs on feature maps using RoIPool, leading to fast speed\nand better accuracy. Faster R-CNN [36] advanced this\nstream by learning the attention mechanism with a Region\nProposal Network (RPN). Faster R-CNN is \ufb02exible and robust to many follow-up improvements ( e.g., [38, 27, 21]),\nand is the current leading framework in several benchmarks.\nInstance Segmentation: Driven by the effectiveness of RCNN, many approaches to instance segmentation are based\nonsegment proposals . Earlier methods [13, 15, 16, 9] resorted to bottom-up segments [42, 2]. DeepMask [33] and\nfollowing works [34, 8] learn to propose segment candidates, which are then classi\ufb01ed by Fast R-CNN. In these\nmethods, segmentation precedes recognition, which is slow\nand less accurate. Likewise, Dai et al. [10] proposed a complex multiple-stage cascade that predicts segment proposals\nfrom bounding-box proposals, followed by classi\ufb01cation.\nInstead, our method is based on parallel prediction of masks\nand class labels, which is simpler and more \ufb02exible.\nMost recently, Li et al. [26] combined the segment proposal system in [8] and object detection system in [11] for"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1050-6db4-11ee-b8f5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-9", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Most recently, Li et al. [26] combined the segment proposal system in [8] and object detection system in [11] for\n\u201cfully convolutional instance segmentation\u201d (FCIS). The\ncommon idea in [8, 11, 26] is to predict a set of positionsensitive output channels fully convolutionally. These\nchannels simultaneously address object classes, boxes, and\nmasks, making the system fast. But FCIS exhibits systematic errors on overlapping instances and creates spurious\nedges (Figure 6), showing that it is challenged by the fundamental dif\ufb01culties of segmenting instances.\n2\nAnother family of solutions [23, 4, 3, 29] to instance segmentation are driven by the success of semantic segmentation. Starting from per-pixel classi\ufb01cation results ( e.g.,\nFCN outputs), these methods attempt to cut the pixels of\nthe same category into different instances. In contrast to the\nsegmentation-\ufb01rst strategy of these methods, Mask R-CNN\nis based on an instance-\ufb01rst strategy. We expect a deeper incorporation of both strategies will be studied in the future.\n3. Mask R-CNN\nMask R-CNN is conceptually simple: Faster R-CNN has\ntwo outputs for each candidate object, a class label and a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1051-6db4-11ee-a0bd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-10", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "3. Mask R-CNN\nMask R-CNN is conceptually simple: Faster R-CNN has\ntwo outputs for each candidate object, a class label and a\nbounding-box offset; to this we add a third branch that outputs the object mask. Mask R-CNN is thus a natural and intuitive idea. But the additional mask output is distinct from\nthe class and box outputs, requiring extraction of much \ufb01ner\nspatial layout of an object. Next, we introduce the key elements of Mask R-CNN, including pixel-to-pixel alignment,\nwhich is the main missing piece of Fast/Faster R-CNN.\nFaster R-CNN: We begin by brie\ufb02y reviewing the Faster\nR-CNN detector [36]. Faster R-CNN consists of two stages.\nThe \ufb01rst stage, called a Region Proposal Network (RPN),\nproposes candidate object bounding boxes. The second\nstage, which is in essence Fast R-CNN [12], extracts features using RoIPool from each candidate box and performs\nclassi\ufb01cation and bounding-box regression. The features\nused by both stages can be shared for faster inference. We\nrefer readers to [21] for latest, comprehensive comparisons\nbetween Faster R-CNN and other frameworks.\nMask R-CNN: Mask R-CNN adopts the same two-stage"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1052-6db4-11ee-a4ae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-11", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "refer readers to [21] for latest, comprehensive comparisons\nbetween Faster R-CNN and other frameworks.\nMask R-CNN: Mask R-CNN adopts the same two-stage\nprocedure, with an identical \ufb01rst stage (which is RPN). In\nthe second stage, in parallel to predicting the class and box\noffset, Mask R-CNN also outputs a binary mask for each\nRoI. This is in contrast to most recent systems, where classi\ufb01cation depends on mask predictions ( e.g. [33, 10, 26]).\nOur approach follows the spirit of Fast R-CNN [12] that\napplies bounding-box classi\ufb01cation and regression in parallel (which turned out to largely simplify the multi-stage\npipeline of original R-CNN [13]).\nFormally, during training, we de\ufb01ne a multi-task loss on\neach sampled RoI as L=Lcls+Lbox+Lmask . The classi\ufb01cation loss Lclsand bounding-box loss Lboxare identical as those de\ufb01ned in [12]. The mask branch has a Km2dimensional output for each RoI, which encodes Kbinary\nmasks of resolution m\u0002m, one for each of the Kclasses."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1053-6db4-11ee-956d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-12", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "masks of resolution m\u0002m, one for each of the Kclasses.\nTo this we apply a per-pixel sigmoid, and de\ufb01ne Lmask as\nthe average binary cross-entropy loss. For an RoI associated\nwith ground-truth class k,Lmask is only de\ufb01ned on the k-th\nmask (other mask outputs do not contribute to the loss).\nOur de\ufb01nition of Lmask allows the network to generate\nmasks for every class without competition among classes;\nwe rely on the dedicated classi\ufb01cation branch to predict the\nFigure 3. RoIAlign: The dashed grid represents a feature map, the solid lines an RoI\n(with 2\u00022 bins in this example), and the dots\nthe 4 sampling points in each bin. RoIAlign\ncomputes the value of each sampling point\nby bilinear interpolation from the nearby grid\npoints on the feature map. No quantization is\nperformed on any coordinates involved in the\nRoI, its bins, or the sampling points.\nclass label used to select the output mask. This decouples\nmask and class prediction. This is different from common\npractice when applying FCNs [30] to semantic segmentation, which typically uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1054-6db4-11ee-9a0d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-13", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "practice when applying FCNs [30] to semantic segmentation, which typically uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes\ncompete; in our case, with a per-pixel sigmoid and a binary\nloss, they do not. We show by experiments that this formulation is key for good instance segmentation results.\nMask Representation: A mask encodes an input object\u2019s\nspatial layout. Thus, unlike class labels or box offsets\nthat are inevitably collapsed into short output vectors by\nfully-connected ( fc) layers, extracting the spatial structure\nof masks can be addressed naturally by the pixel-to-pixel\ncorrespondence provided by convolutions.\nSpeci\ufb01cally, we predict an m\u0002mmask from each RoI\nusing an FCN [30]. This allows each layer in the mask\nbranch to maintain the explicit m\u0002mobject spatial layout without collapsing it into a vector representation that\nlacks spatial dimensions. Unlike previous methods that resort to fclayers for mask prediction [33, 34, 10], our fully\nconvolutional representation requires fewer parameters, and\nis more accurate as demonstrated by experiments.\nThis pixel-to-pixel behavior requires our RoI features,\nwhich themselves are small feature maps, to be well aligned"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1055-6db4-11ee-bb67-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-14", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "is more accurate as demonstrated by experiments.\nThis pixel-to-pixel behavior requires our RoI features,\nwhich themselves are small feature maps, to be well aligned\nto faithfully preserve the explicit per-pixel spatial correspondence. This motivated us to develop the following\nRoIAlign layer that plays a key role in mask prediction.\nRoIAlign: RoIPool [12] is a standard operation for extracting a small feature map ( e.g., 7\u00027) from each RoI. RoIPool\n\ufb01rstquantizes a \ufb02oating-number RoI to the discrete granularity of the feature map, this quantized RoI is then subdivided into spatial bins which are themselves quantized, and\n\ufb01nally feature values covered by each bin are aggregated\n(usually by max pooling). Quantization is performed, e.g.,\non a continuous coordinate xby computing [x=16], where\n16 is a feature map stride and [\u0001]is rounding; likewise, quantization is performed when dividing into bins ( e.g., 7\u00027).\nThese quantizations introduce misalignments between the\nRoI and the extracted features. While this may not impact\nclassi\ufb01cation, which is robust to small translations, it has a\nlarge negative effect on predicting pixel-accurate masks."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1056-6db4-11ee-be8b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-15", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "classi\ufb01cation, which is robust to small translations, it has a\nlarge negative effect on predicting pixel-accurate masks.\nTo address this, we propose an RoIAlign layer that removes the harsh quantization of RoIPool, properly aligning\nthe extracted features with the input. Our proposed change\nis simple: we avoid any quantization of the RoI boundaries\n3\nor bins ( i.e., we use x=16instead of [x=16]). We use bilinear interpolation [22] to compute the exact values of the\ninput features at four regularly sampled locations in each\nRoI bin, and aggregate the result (using max or average),\nsee Figure 3 for details. We note that the results are not sensitive to the exact sampling locations, or how many points\nare sampled, as long as no quantization is performed.\nRoIAlign leads to large improvements as we show in\nx4.2. We also compare to the RoIWarp operation proposed\nin [10]. Unlike RoIAlign, RoIWarp overlooked the alignment issue and was implemented in [10] as quantizing RoI\njust like RoIPool. So even though RoIWarp also adopts\nbilinear resampling motivated by [22], it performs on par"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1057-6db4-11ee-bd60-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-16", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "just like RoIPool. So even though RoIWarp also adopts\nbilinear resampling motivated by [22], it performs on par\nwith RoIPool as shown by experiments (more details in Table 2c), demonstrating the crucial role of alignment.\nNetwork Architecture: To demonstrate the generality of\nour approach, we instantiate Mask R-CNN with multiple\narchitectures. For clarity, we differentiate between: (i) the\nconvolutional backbone architecture used for feature extraction over an entire image, and (ii) the network head\nfor bounding-box recognition (classi\ufb01cation and regression)\nand mask prediction that is applied separately to each RoI.\nWe denote the backbone architecture using the nomenclature network-depth-features . We evaluate ResNet [19]\nand ResNeXt [45] networks of depth 50 or 101 layers. The\noriginal implementation of Faster R-CNN with ResNets\n[19] extracted features from the \ufb01nal convolutional layer\nof the 4-th stage, which we call C4. This backbone with\nResNet-50, for example, is denoted by ResNet-50-C4. This\nis a common choice used in [19, 10, 21, 39]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1058-6db4-11ee-82e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-17", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "ResNet-50, for example, is denoted by ResNet-50-C4. This\nis a common choice used in [19, 10, 21, 39].\nWe also explore another more effective backbone recently proposed by Lin et al. [27], called a Feature Pyramid Network (FPN). FPN uses a top-down architecture with\nlateral connections to build an in-network feature pyramid\nfrom a single-scale input. Faster R-CNN with an FPN backbone extracts RoI features from different levels of the feature pyramid according to their scale, but otherwise the\nrest of the approach is similar to vanilla ResNet. Using a\nResNet-FPN backbone for feature extraction with Mask RCNN gives excellent gains in both accuracy and speed. For\nfurther details on FPN, we refer readers to [27].\nFor the network head we closely follow architectures\npresented in previous work to which we add a fully convolutional mask prediction branch. Speci\ufb01cally, we extend the Faster R-CNN box heads from the ResNet [19]\nand FPN [27] papers. Details are shown in Figure 4. The\nhead on the ResNet-C4 backbone includes the 5-th stage of\nResNet (namely, the 9-layer \u2018res5\u2019 [19]), which is computeintensive. For FPN, the backbone already includes res5 and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1059-6db4-11ee-b51e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-18", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "ResNet (namely, the 9-layer \u2018res5\u2019 [19]), which is computeintensive. For FPN, the backbone already includes res5 and\nthus allows for a more ef\ufb01cient head that uses fewer \ufb01lters.\nWe note that our mask branches have a straightforward\nstructure. More complex designs have the potential to improve performance but are not the focus of this work.\nave\nRoI\nRoI14\u00d714\n\u00d72567\u00d77\n\u00d7256\n14\u00d714\n\u00d72561024\n28\u00d728\n\u00d72561024\nmask14\u00d714\n\u00d7256class\nbox2048RoI res57\u00d77\n\u00d710247\u00d77\n\u00d72048\n\u00d74class\nbox\n14\u00d714\n\u00d780\nmask28\u00d728\n\u00d780Faster R-CNN\nw/ ResNet [19]Faster R-CNN\nw/ FPN [27]\nFigure 4. Head Architecture : We extend two existing Faster RCNN heads [19, 27]. Left/Right panels show the heads for the\nResNet C4 and FPN backbones, from [19] and [27], respectively,\nto which a mask branch is added. Numbers denote spatial resolution and channels. Arrows denote either conv, deconv, or fclayers\nas can be inferred from context (conv preserves spatial dimension"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b105a-6db4-11ee-8db5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-19", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "to which a mask branch is added. Numbers denote spatial resolution and channels. Arrows denote either conv, deconv, or fclayers\nas can be inferred from context (conv preserves spatial dimension\nwhile deconv increases it). All convs are 3 \u00023, except the output\nconv which is 1 \u00021, deconvs are 2 \u00022 with stride 2, and we use\nReLU [31] in hidden layers. Left: \u2018res5\u2019 denotes ResNet\u2019s \ufb01fth\nstage, which for simplicity we altered so that the \ufb01rst conv operates on a 7 \u00027 RoI with stride 1 (instead of 14 \u000214 / stride 2 as in\n[19]). Right : \u2018\u00024\u2019 denotes a stack of four consecutive convs.\n3.1. Implementation Details\nWe set hyper-parameters following existing Fast/Faster\nR-CNN work [12, 36, 27]. Although these decisions were\nmade for object detection in original papers [12, 36, 27], we\nfound our instance segmentation system is robust to them.\nTraining: As in Fast R-CNN, an RoI is considered positive\nif it has IoU with a ground-truth box of at least 0.5 and\nnegative otherwise. The mask loss Lmask is de\ufb01ned only on\npositive RoIs. The mask target is the intersection between"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b105b-6db4-11ee-8a2f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-20", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "negative otherwise. The mask loss Lmask is de\ufb01ned only on\npositive RoIs. The mask target is the intersection between\nan RoI and its associated ground-truth mask.\nWe adopt image-centric training [12]. Images are resized\nsuch that their scale (shorter edge) is 800 pixels [27]. Each\nmini-batch has 2 images per GPU and each image has N\nsampled RoIs, with a ratio of 1:3 of positive to negatives\n[12]. Nis 64 for the C4 backbone (as in [12, 36]) and 512\nfor FPN (as in [27]). We train on 8 GPUs (so effective minibatch size is 16) for 160k iterations, with a learning rate of\n0.02 which is decreased by 10 at the 120k iteration. We\nuse a weight decay of 0.0001 and momentum of 0.9. With\nResNeXt [45], we train with 1 image per GPU and the same\nnumber of iterations, with a starting learning rate of 0.01.\nThe RPN anchors span 5 scales and 3 aspect ratios, following [27]. For convenient ablation, RPN is trained separately and does not share features with Mask R-CNN, unless speci\ufb01ed. For every entry in this paper, RPN and Mask\nR-CNN have the same backbones and so they are shareable."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b105c-6db4-11ee-8a59-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-21", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "R-CNN have the same backbones and so they are shareable.\nInference: At test time, the proposal number is 300 for the\nC4 backbone (as in [36]) and 1000 for FPN (as in [27]). We\nrun the box prediction branch on these proposals, followed\nby non-maximum suppression [14]. The mask branch is\nthen applied to the highest scoring 100 detection boxes. Although this differs from the parallel computation used in\ntraining, it speeds up inference and improves accuracy (due\nto the use of fewer, more accurate RoIs). The mask branch\n4\nhorse1.00horse1.00horse1.00\nbus1.00bus1.00\ncar.98truck.88\ncar.93\ncar.78car.98\ncar.91 car.96car.99\ncar.94car.99car.98 truck.86\ncar.99car.95car1.00car.93 car.98car.95\ncar.97car.87\ncar.99\ncar.82car.78car.93\ncar.95\ncar.97\nperson.99traffic light .73\nperson1.00\nperson.99person.95person.93\nperson.93\nperson1.00person.98\nskateboard.82\nsuitcase1.00\nsuitcase.99suitcase.96suitcase1.00"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b105d-6db4-11ee-9bb2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-22", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "person.93\nperson1.00person.98\nskateboard.82\nsuitcase1.00\nsuitcase.99suitcase.96suitcase1.00\nsuitcase.93suitcase.98\nsuitcase.88suitcase.72stop sign.88\nperson1.00 person1.00\nperson1.00person1.00\nperson.99person.99\nbench.76skateboard.91\nskateboard.83handba g.81\nsurfboard1.00person1.00person1.00 surfboard1.00person1.00person.98\nsurfboard1.00person1.00\nsurfboard.98surfboard1.00person.91\nperson.74\nperson1.00person1.00\nperson1.00person1.00person1.00 person1.00 person.98person.99\nperson1.00person.99 umbrella1.00\nperson.95umbrella.99umbrella.97umbrella.97\numbrella.96\numbrella1.00\nbackpack.96umbrella.98\nbackpack.95person.80\nbackpack.98\nbicycle.93umbrella.89person.89handba g.97\nhandba g.85"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b105e-6db4-11ee-a4ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-23", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "backpack.95person.80\nbackpack.98\nbicycle.93umbrella.89person.89handba g.97\nhandba g.85\nperson1.00person1.00 person1.00person1.00person1.00person1.00\nmotorcycle.72kite.89\nperson.99kite.95\nperson.99\nperson1.00person.81person.72kite.93\nperson.89kite1.00\nperson.98\nperson1.00kite.84kite.97\nperson.80\nhandba g.80person.99kite.82\nperson.98 person.96kite.98\nperson.99person.82kite.81\nperson.95 person.84kite.98kite.72\nkite.99kite.84kite.99\nperson.94person.72person.98kite.95\nperson.98 person.77kite.73\nperson.78 person.71 person.87kite.88kite.88\nperson.94kite.86kite.89\nzebra.99\nzebra1.00zebra1.00\nzebra.99zebra1.00zebra.96\nzebra.74\nzebra.96zebra.99zebra.90"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b105f-6db4-11ee-b663-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-24", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "zebra.99zebra1.00zebra.96\nzebra.74\nzebra.96zebra.99zebra.90\nzebra.88zebra.76\ndining table.91dining table.78\nchair.97person.99\nperson.86\nchair.94chair.98person.95\nchair.95person.97\nchair.92chair.99person.97\nperson.99person.94 person.99person.87\nperson.99\nchair.83person.94person.99person.98\nchair.87chair.95person.97\nperson.96\nchair.99person.86person.89\nchair.89\nwine glass.93person.98person.88person.97\nperson.88person.88\nperson.91 chair.96person.95\nperson.77person.92\nwine glass.94cup.83\nwine glass.94\nwine glass.83\ncup.91chair.85 dining table.96\nwine glass.91person.96\ncup.98person.83\ndining table.75\ncup.96person.72\nwine glass.80chair.98person.81person.82\ndining table.81\nchair.85chair.78\ncup.75person.77\ncup.71wine glass.80cup.79cup.93\ncup.71"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1060-6db4-11ee-8fa5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-25", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "dining table.81\nchair.85chair.78\ncup.75person.77\ncup.71wine glass.80cup.79cup.93\ncup.71\nperson.99person.99\nperson1.00person1.00\nfrisbee1.00\nperson.80person.82\nelephant1.00elephant1.00elephant1.00\nelephant.97\nelephant.99\nperson1.00person1.00\ndining table.95person1.00person.88\nwine glass1.00bottle.97\nwine glass1.00wine glass.99tv.98 tv.84\nperson1.00\nbench.97person.98person1.00person1.00\nhandba g.73person.86 potted plant.92\nbird.93person.76person.98person.78 person.78 backpack.88handba g.91\ncell phone.77 clock.73\nperson.99person1.00person.98\nperson1.00person1.00 person1.00person.99\nperson.99 person.99 person1.00 person1.00person.98 person.99\nhandba g.88person1.00 person.98person.92\nhandba g.99person.97\nperson.95\nhandba g.88traffic light .99\nperson.95"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1061-6db4-11ee-9c78-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-26", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "handba g.99person.97\nperson.95\nhandba g.88traffic light .99\nperson.95\nperson.87person.95traffic light .87\ntraffic light .71\nperson.80person.95 person.95 person.73person.74\ntie.85\ncar.99\ncar.86car.97car1.00 car.95car.97traffic light 1.00traffic light .99\ncar.99person.99car.95\ncar.97 car.98car.98\ncar.91\ncar1.00car.96car.96\nbicycle.86car.97car.97\ncar.97car.94car.95\ncar.94car.81\nperson.87\nparking meter.98car.89\ndonut1.00donut.90\ndonut.88donut.81\ndonut.95\ndonut.96donut1.00 donut.98\ndonut.99donut.94donut.97donut.99\ndonut.98donut1.00\ndonut.95donut1.00\ndonut.98donut.98donut.99\ndonut.96\ndonut.89donut.96donut.95donut.98donut.89\ndonut.93donut.95"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1062-6db4-11ee-87ea-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-27", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "donut.96\ndonut.89donut.96donut.95donut.98donut.89\ndonut.93donut.95\ndonut.90donut.89\ndonut.89donut.89\ndonut.86donut.86\nperson1.00person1.00 person1.00\nperson1.00person1.00person1.00\nperson1.00\ndog1.00baseball bat.99\nbaseball bat.85\nbaseball bat.98\ntruck.92\ntruck.99truck.96truck.99truck.97\nbus.99truck.93 bus.90\nperson1.00person1.00horse.77horse.99\ncow.93person.96person1.00\nperson.99horse.97\nperson.98 person.97person.98\nperson.96\nperson1.00\ntennis racket1.00chair.73person.90person.77\nperson.97\nperson.81person.87\nperson.71 person.96 person.99 person.98 person.94chair.97\nchair.80\nchair.71chair.94chair.92chair.99chair.93\nchair.99\nchair.91 chair.81 chair.98 chair.83chair.81chair.81\nchair.93\nsports ball.99"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1063-6db4-11ee-aa0c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-28", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "chair.99\nchair.91 chair.81 chair.98 chair.83chair.81chair.81\nchair.93\nsports ball.99\nperson1.00\ncouch.82person1.00\nperson.99person1.00person1.00person1.00 person.99skateboard.99\nperson.90person.98person.99person.91\nperson.99person1.00\nperson.80\nskateboard.98Figure 5. More results of Mask R-CNN on COCO test images, using ResNet-101-FPN and running at 5 fps, with 35.7 mask AP (Table 1).\nbackbone AP AP 50 AP75 APS APM APL\nMNC [10] ResNet-101-C4 24.6 44.3 24.8 4.7 25.9 43.6\nFCIS [26] +OHEM ResNet-101-C5-dilated 29.2 49.5 - 7.1 31.3 50.0\nFCIS+++ [26] +OHEM ResNet-101-C5-dilated 33.6 54.5 - - - Mask R-CNN ResNet-101-C4 33.1 54.9 34.8 12.1 35.6 51.1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1064-6db4-11ee-871e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-29", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Mask R-CNN ResNet-101-FPN 35.7 58.0 37.8 15.5 38.1 52.4\nMask R-CNN ResNeXt-101-FPN 37.1 60.0 39.4 16.9 39.9 53.5\nTable 1. Instance segmentation mask AP on COCO test-dev . MNC [10] and FCIS [26] are the winners of the COCO 2015 and 2016\nsegmentation challenges, respectively. Without bells and whistles, Mask R-CNN outperforms the more complex FCIS+++, which includes\nmulti-scale train/test, horizontal \ufb02ip test, and OHEM [38]. All entries are single-model results.\ncan predict Kmasks per RoI, but we only use the k-th mask,\nwhere kis the predicted class by the classi\ufb01cation branch.\nThem\u0002m\ufb02oating-number mask output is then resized to\nthe RoI size, and binarized at a threshold of 0.5.\nNote that since we only compute masks on the top 100\ndetection boxes, Mask R-CNN adds a small overhead to its\nFaster R-CNN counterpart ( e.g.,\u001820% on typical models).\n4. Experiments: Instance Segmentation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1065-6db4-11ee-9912-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-30", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Faster R-CNN counterpart ( e.g.,\u001820% on typical models).\n4. Experiments: Instance Segmentation\nWe perform a thorough comparison of Mask R-CNN to\nthe state of the art along with comprehensive ablations on\nthe COCO dataset [28]. We report the standard COCO metrics including AP (averaged over IoU thresholds), AP 50,\nAP75, and AP S, AP M, AP L(AP at different scales). Unless noted, AP is evaluating using mask IoU. As in previous\nwork [5, 27], we train using the union of 80k train images\nand a 35k subset of val images ( trainval35k ), and report ablations on the remaining 5k val images ( minival ).\nWe also report results on test-dev [28].4.1. Main Results\nWe compare Mask R-CNN to the state-of-the-art methods in instance segmentation in Table 1. All instantiations of our model outperform baseline variants of previous state-of-the-art models. This includes MNC [10]\nand FCIS [26], the winners of the COCO 2015 and 2016\nsegmentation challenges, respectively. Without bells and\nwhistles, Mask R-CNN with ResNet-101-FPN backbone\noutperforms FCIS+++ [26], which includes multi-scale"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1066-6db4-11ee-b15e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-31", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "whistles, Mask R-CNN with ResNet-101-FPN backbone\noutperforms FCIS+++ [26], which includes multi-scale\ntrain/test, horizontal \ufb02ip test, and online hard example mining (OHEM) [38]. While outside the scope of this work, we\nexpect many such improvements to be applicable to ours.\nMask R-CNN outputs are visualized in Figures 2 and 5.\nMask R-CNN achieves good results even under challenging conditions. In Figure 6 we compare our Mask R-CNN\nbaseline and FCIS+++ [26]. FCIS+++ exhibits systematic\nartifacts on overlapping instances, suggesting that it is challenged by the fundamental dif\ufb01culty of instance segmentation. Mask R-CNN shows no such artifacts.\n5\nperson1.00\nperson1.00\nperson1.00person1.00umbrella1.00 umbrella.99\ncar.99car.93\ngiraffe1.00 giraffe1.00\nperson1.00person1.00person1.00person1.00\nperson.95\nsports ball1.00sports ball.98person1.00\nperson1.00\nperson1.00\ntie.95\ntie1.00\nFCIS Mask R-CNN\ntrain1.00\ntrain.99\ntrain.80\nperson1.00 person1.00person1.00"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1067-6db4-11ee-9a36-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-32", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "tie1.00\nFCIS Mask R-CNN\ntrain1.00\ntrain.99\ntrain.80\nperson1.00 person1.00person1.00\nperson1.00person1.00 person1.00\nskateboard.98person.99 person.99\nskateboard.99handba g.93\nFigure 6. FCIS+++ [26] (top) vs. Mask R-CNN (bottom, ResNet-101-FPN). FCIS exhibits systematic artifacts on overlapping objects.\nnet-depth-features AP AP 50 AP75\nResNet-50-C4 30.3 51.2 31.5\nResNet-101-C4 32.7 54.2 34.3\nResNet-50-FPN 33.6 55.2 35.3\nResNet-101-FPN 35.4 57.3 37.5\nResNeXt-101-FPN 36.7 59.5 38.9\n(a)Backbone Architecture : Better backbones bring expected gains: deeper networks\ndo better, FPN outperforms C4 features, and\nResNeXt improves on ResNet.AP AP 50 AP75\nsoftmax 24.8 44.1 25.1\nsigmoid 30.3 51.2 31.5\n+5.5 +7.1 +6.4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1068-6db4-11ee-a678-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-33", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "softmax 24.8 44.1 25.1\nsigmoid 30.3 51.2 31.5\n+5.5 +7.1 +6.4\n(b)Multinomial vs. Independent Masks\n(ResNet-50-C4): Decoupling via perclass binary masks (sigmoid) gives large\ngains over multinomial masks (softmax).align? bilinear? agg. AP AP 50 AP75\nRoIPool [12] max 26.9 48.8 26.4\nRoIWarp [10]X max 27.2 49.2 27.1\nX ave 27.1 48.9 27.1\nRoIAlignX X max 30.2 51.0 31.8\nX X ave 30.3 51.2 31.5\n(c)RoIAlign (ResNet-50-C4): Mask results with various RoI\nlayers. Our RoIAlign layer improves AP by \u00183 points and\nAP75by\u00185 points. Using proper alignment is the only factor that contributes to the large gap between RoI layers.\nAP AP 50 AP75 APbbAPbb\n50APbb\n75\nRoIPool 23.6 46.5 21.6 28.2 52.7 26.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1069-6db4-11ee-989a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-34", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "AP AP 50 AP75 APbbAPbb\n50APbb\n75\nRoIPool 23.6 46.5 21.6 28.2 52.7 26.9\nRoIAlign 30.9 51.8 32.1 34.0 55.3 36.4\n+7.3 + 5.3 +10.5 +5.8 +2.6 +9.5\n(d)RoIAlign (ResNet-50- C5,stride 32 ): Mask-level and box-level\nAP using large-stride features. Misalignments are more severe than\nwith stride-16 features (Table 2c), resulting in big accuracy gaps.mask branch AP AP 50 AP75\nMLP fc: 1024!1024!80\u000128231.5 53.7 32.8\nMLP fc: 1024!1024!1024!80\u000128231.5 54.0 32.6\nFCN conv: 256!256!256!256!256!80 33.6 55.2 35.3\n(e)Mask Branch (ResNet-50-FPN): Fully convolutional networks (FCN) vs.\nmulti-layer perceptrons (MLP, fully-connected) for mask prediction. FCNs improve results as they take advantage of explicitly encoding spatial layout."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b106a-6db4-11ee-8351-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-35", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "multi-layer perceptrons (MLP, fully-connected) for mask prediction. FCNs improve results as they take advantage of explicitly encoding spatial layout.\nTable 2. Ablations . We train on trainval35k , test on minival , and report mask AP unless otherwise noted.\n4.2. Ablation Experiments\nWe run a number of ablations to analyze Mask R-CNN.\nResults are shown in Table 2 and discussed in detail next.\nArchitecture: Table 2a shows Mask R-CNN with various\nbackbones. It bene\ufb01ts from deeper networks (50 vs. 101)\nand advanced designs including FPN and ResNeXt. We\nnote that notall frameworks automatically bene\ufb01t from\ndeeper or advanced networks (see benchmarking in [21]).\nMultinomial vs. Independent Masks: Mask R-CNN decouples mask and class prediction: as the existing box\nbranch predicts the class label, we generate a mask for each\nclass without competition among classes (by a per-pixel sigmoid and a binary loss). In Table 2b, we compare this to\nusing a per-pixel softmax and a multinomial loss (as commonly used in FCN [30]). This alternative couples the tasks\nof mask and class prediction, and results in a severe loss"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b106b-6db4-11ee-a1c5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-36", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "of mask and class prediction, and results in a severe loss\nin mask AP (5.5 points). This suggests that once the instance has been classi\ufb01ed as a whole (by the box branch),\nit is suf\ufb01cient to predict a binary mask without concern for\nthe categories, which makes the model easier to train.\nClass-Speci\ufb01c vs. Class-Agnostic Masks: Our default instantiation predicts class-speci\ufb01c masks, i.e., one m\u0002mmask per class. Interestingly, Mask R-CNN with classagnostic masks ( i.e., predicting a single m\u0002moutput regardless of class) is nearly as effective: it has 29.7 mask AP\nvs. 30.3 for the class-speci\ufb01c counterpart on ResNet-50-C4.\nThis further highlights the division of labor in our approach\nwhich largely decouples classi\ufb01cation and segmentation.\nRoIAlign: An evaluation of our proposed RoIAlign layer is\nshown in Table 2c. For this experiment we use the ResNet50-C4 backbone, which has stride 16. RoIAlign improves\nAP by about 3 points over RoIPool, with much of the gain\ncoming at high IoU (AP 75). RoIAlign is insensitive to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b106c-6db4-11ee-bfed-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-37", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "AP by about 3 points over RoIPool, with much of the gain\ncoming at high IoU (AP 75). RoIAlign is insensitive to\nmax/average pool; we use average in the rest of the paper.\nAdditionally, we compare with RoIWarp proposed in\nMNC [10] that also adopt bilinear sampling. As discussed\ninx3, RoIWarp still quantizes the RoI, losing alignment\nwith the input. As can be seen in Table 2c, RoIWarp performs on par with RoIPool and much worse than RoIAlign.\nThis highlights that proper alignment is key.\nWe also evaluate RoIAlign with a ResNet-50-C5 backbone, which has an even larger stride of 32 pixels. We use\nthe same head as in Figure 4 (right), as the res5 head is not\napplicable. Table 2d shows that RoIAlign improves mask\nAP by a massive 7.3 points, and mask AP 75by 10.5 points\n6\nbackbone APbbAPbb\n50APbb\n75APbb\nSAPbb\nMAPbb\nL\nFaster R-CNN+++ [19] ResNet-101-C4 34.9 55.7 37.4 15.6 38.7 50.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b106d-6db4-11ee-af32-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-38", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "MAPbb\nL\nFaster R-CNN+++ [19] ResNet-101-C4 34.9 55.7 37.4 15.6 38.7 50.9\nFaster R-CNN w FPN [27] ResNet-101-FPN 36.2 59.1 39.0 18.2 39.0 48.2\nFaster R-CNN by G-RMI [21] Inception-ResNet-v2 [41] 34.7 55.5 36.7 13.5 38.1 52.0\nFaster R-CNN w TDM [39] Inception-ResNet-v2-TDM 36.8 57.7 39.2 16.2 39.8 52.1\nFaster R-CNN, RoIAlign ResNet-101-FPN 37.3 59.6 40.3 19.8 40.2 48.8\nMask R-CNN ResNet-101-FPN 38.2 60.3 41.7 20.1 41.1 50.2\nMask R-CNN ResNeXt-101-FPN 39.8 62.3 43.4 22.1 43.2 51.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b106e-6db4-11ee-a78b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-39", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Mask R-CNN ResNeXt-101-FPN 39.8 62.3 43.4 22.1 43.2 51.2\nTable 3. Object detection single-model results (bounding box AP), vs. state-of-the-art on test-dev . Mask R-CNN using ResNet-101FPN outperforms the base variants of all previous state-of-the-art models (the mask output is ignored in these experiments). The gains of\nMask R-CNN over [27] come from using RoIAlign (+1.1 APbb), multitask training (+0.9 APbb), and ResNeXt-101 (+1.6 APbb).\n(50% relative improvement ). Moreover, we note that with\nRoIAlign, using stride-32 C5 features (30.9 AP) is more accurate than using stride-16 C4 features (30.3 AP, Table 2c).\nRoIAlign largely resolves the long-standing challenge of\nusing large-stride features for detection and segmentation.\nFinally, RoIAlign shows a gain of 1.5 mask AP and 0.5\nbox AP when used with FPN, which has \ufb01ner multi-level\nstrides. For keypoint detection that requires \ufb01ner alignment,\nRoIAlign shows large gains even with FPN (Table 6)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b106f-6db4-11ee-baf7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-40", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "strides. For keypoint detection that requires \ufb01ner alignment,\nRoIAlign shows large gains even with FPN (Table 6).\nMask Branch: Segmentation is a pixel-to-pixel task and\nwe exploit the spatial layout of masks by using an FCN.\nIn Table 2e, we compare multi-layer perceptrons (MLP)\nand FCNs, using a ResNet-50-FPN backbone. Using FCNs\ngives a 2.1 mask AP gain over MLPs. We note that we\nchoose this backbone so that the conv layers of the FCN\nhead are not pre-trained, for a fair comparison with MLP.\n4.3. Bounding Box Detection Results\nWe compare Mask R-CNN to the state-of-the-art COCO\nbounding-box object detection in Table 3. For this result,\neven though the full Mask R-CNN model is trained, only\nthe classi\ufb01cation and box outputs are used at inference (the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1070-6db4-11ee-a560-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-41", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "even though the full Mask R-CNN model is trained, only\nthe classi\ufb01cation and box outputs are used at inference (the\nmask output is ignored). Mask R-CNN using ResNet-101FPN outperforms the base variants of all previous state-ofthe-art models, including the single-model variant of GRMI [21], the winner of the COCO 2016 Detection Challenge. Using ResNeXt-101-FPN, Mask R-CNN further improves results, with a margin of 3.0 points box AP over\nthe best previous single model entry from [39] (which used\nInception-ResNet-v2-TDM).\nAs a further comparison, we trained a version of Mask\nR-CNN but without the mask branch, denoted by \u201cFaster\nR-CNN, RoIAlign\u201d in Table 3. This model performs better\nthan the model presented in [27] due to RoIAlign. On the\nother hand, it is 0.9 points box AP lower than Mask R-CNN.\nThis gap of Mask R-CNN on box detection is therefore due\nsolely to the bene\ufb01ts of multi-task training.\nLastly, we note that Mask R-CNN attains a small gap\nbetween its mask and box AP: e.g., 2.7 points between 37.1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1071-6db4-11ee-aa40-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-42", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Lastly, we note that Mask R-CNN attains a small gap\nbetween its mask and box AP: e.g., 2.7 points between 37.1\n(mask, Table 1) and 39.8 (box, Table 3). This indicates that\nour approach largely closes the gap between object detection and the more challenging instance segmentation task.4.4. Timing\nInference: We train a ResNet-101-FPN model that shares\nfeatures between the RPN and Mask R-CNN stages, following the 4-step training of Faster R-CNN [36]. This model\nruns at 195ms per image on an Nvidia Tesla M40 GPU (plus\n15ms CPU time resizing the outputs to the original resolution), and achieves statistically the same mask AP as the\nunshared one. We also report that the ResNet-101-C4 variant takes \u0018400ms as it has a heavier box head (Figure 4), so\nwe do not recommend using the C4 variant in practice.\nAlthough Mask R-CNN is fast, we note that our design\nis not optimized for speed, and better speed/accuracy tradeoffs could be achieved [21], e.g., by varying image sizes and\nproposal numbers, which is beyond the scope of this paper.\nTraining: Mask R-CNN is also fast to train. Training with"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1072-6db4-11ee-8957-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-43", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "proposal numbers, which is beyond the scope of this paper.\nTraining: Mask R-CNN is also fast to train. Training with\nResNet-50-FPN on COCO trainval35k takes 32 hours\nin our synchronized 8-GPU implementation (0.72s per 16image mini-batch), and 44 hours with ResNet-101-FPN. In\nfact, fast prototyping can be completed in less than one day\nwhen training on the train set. We hope such rapid training will remove a major hurdle in this area and encourage\nmore people to perform research on this challenging topic.\n5. Mask R-CNN for Human Pose Estimation\nOur framework can easily be extended to human pose\nestimation. We model a keypoint\u2019s location as a one-hot\nmask, and adopt Mask R-CNN to predict Kmasks, one for\neach of Kkeypoint types ( e.g., left shoulder, right elbow).\nThis task helps demonstrate the \ufb02exibility of Mask R-CNN.\nWe note that minimal domain knowledge for human pose\nis exploited by our system, as the experiments are mainly to\ndemonstrate the generality of the Mask R-CNN framework.\nWe expect that domain knowledge ( e.g., modeling structures [6]) will be complementary to our simple approach.\nImplementation Details: We make minor modi\ufb01cations to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1073-6db4-11ee-b4cb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-44", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "We expect that domain knowledge ( e.g., modeling structures [6]) will be complementary to our simple approach.\nImplementation Details: We make minor modi\ufb01cations to\nthe segmentation system when adapting it for keypoints.\nFor each of the Kkeypoints of an instance, the training\ntarget is a one-hot m\u0002mbinary mask where only a single\npixel is labeled as foreground. During training, for each visible ground-truth keypoint, we minimize the cross-entropy\nloss over an m2-way softmax output (which encourages a\n7\nFigure 7. Keypoint detection results on COCO test using Mask R-CNN (ResNet-50-FPN), with person segmentation masks predicted\nfrom the same model. This model has a keypoint AP of 63.1 and runs at 5 fps.\nAPkpAPkp\n50APkp\n75APkp\nMAPkp\nL\nCMU-Pose+++ [6] 61.8 84.9 67.5 57.1 68.2\nG-RMI [32]y62.4 84.0 68.5 59.1 68.1\nMask R-CNN , keypoint-only 62.7 87.0 68.4 57.4 71.1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1074-6db4-11ee-b93e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-45", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Mask R-CNN , keypoint-only 62.7 87.0 68.4 57.4 71.1\nMask R-CNN , keypoint & mask 63.1 87.3 68.7 57.8 71.4\nTable 4. Keypoint detection AP on COCO test-dev . Ours is a\nsingle model (ResNet-50-FPN) that runs at 5 fps. CMU-Pose+++\n[6] is the 2016 competition winner that uses multi-scale testing,\npost-processing with CPM [44], and \ufb01ltering with an object detector, adding a cumulative \u00185 points (clari\ufb01ed in personal communication).y: G-RMI was trained on COCO plus MPII [1] (25k images), using two models (Inception-ResNet-v2 for bounding box\ndetection and ResNet-101 for keypoints).\nsingle point to be detected). We note that as in instance segmentation, the Kkeypoints are still treated independently.\nWe adopt the ResNet-FPN variant, and the keypoint head\narchitecture is similar to that in Figure 4 (right). The keypoint head consists of a stack of eight 3 \u00023 512-d conv layers, followed by a deconv layer and 2 \u0002bilinear upscaling,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1075-6db4-11ee-8b3b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-46", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "producing an output resolution of 56 \u000256. We found that\na relatively high resolution output (compared to masks) is\nrequired for keypoint-level localization accuracy.\nModels are trained on all COCO trainval35k images that contain annotated keypoints. To reduce over\ufb01tting, as this training set is smaller, we train using image\nscales randomly sampled from [640, 800] pixels; inference\nis on a single scale of 800 pixels. We train for 90k iterations,\nstarting from a learning rate of 0.02 and reducing it by 10 at\n60k and 80k iterations. We use bounding-box NMS with a\nthreshold of 0.5. Other details are identical as in x3.1.\nMain Results and Ablations: We evaluate the person keypoint AP (APkp) and experiment with a ResNet-50-FPN\nbackbone; more backbones will be studied in the appendix.\nTable 4 shows that our result (62.7 APkp) is 0.9 points higher\nthan the COCO 2016 keypoint detection winner [6] that\nuses a multi-stage processing pipeline (see caption of Table 4). Our method is considerably simpler and faster.\nMore importantly, we have a uni\ufb01ed model that can si-APbb\nperson APmask\nperson APkp"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1076-6db4-11ee-8073-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-47", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "More importantly, we have a uni\ufb01ed model that can si-APbb\nperson APmask\nperson APkp\nFaster R-CNN 52.5 - Mask R-CNN, mask-only 53.6 45.8 Mask R-CNN, keypoint-only 50.7 - 64.2\nMask R-CNN, keypoint & mask 52.0 45.1 64.7\nTable 5. Multi-task learning of box, mask, and keypoint about the\nperson category, evaluated on minival . All entries are trained\non the same data for fair comparisons. The backbone is ResNet50-FPN. The entries with 64.2 and 64.7 AP on minival have\ntest-dev AP of 62.7 and 63.1, respectively (see Table 4).\nAPkpAPkp\n50APkp\n75APkp\nMAPkp\nL\nRoIPool 59.8 86.2 66.7 55.1 67.4\nRoIAlign 64.2 86.6 69.7 58.7 73.0\nTable 6. RoIAlign vs. RoIPool for keypoint detection on\nminival . The backbone is ResNet-50-FPN.\nmultaneously predict boxes, segments, and keypoints while"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1077-6db4-11ee-a741-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-48", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "minival . The backbone is ResNet-50-FPN.\nmultaneously predict boxes, segments, and keypoints while\nrunning at 5 fps. Adding a segment branch (for the person category) improves the APkpto 63.1 (Table 4) on\ntest-dev . More ablations of multi-task learning on\nminival are in Table 5. Adding the mask branch to the\nbox-only ( i.e., Faster R-CNN) or keypoint-only versions\nconsistently improves these tasks. However, adding the\nkeypoint branch reduces the box/mask AP slightly, suggesting that while keypoint detection bene\ufb01ts from multitask\ntraining, it does not in turn help the other tasks. Nevertheless, learning all three tasks jointly enables a uni\ufb01ed system\nto ef\ufb01ciently predict all outputs simultaneously (Figure 7).\nWe also investigate the effect of RoIAlign on keypoint\ndetection (Table 6). Though this ResNet-50-FPN backbone\nhas \ufb01ner strides ( e.g., 4 pixels on the \ufb01nest level), RoIAlign\nstill shows signi\ufb01cant improvement over RoIPool and increases APkpby 4.4 points. This is because keypoint detections are more sensitive to localization accuracy. This again"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1078-6db4-11ee-9310-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-49", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "indicates that alignment is essential for pixel-level localization, including masks and keypoints.\nGiven the effectiveness of Mask R-CNN for extracting\nobject bounding boxes, masks, and keypoints, we expect it\nbe an effective framework for other instance-level tasks.\n8\ntraining data AP [val] AP AP 50 person rider car truck bus train mcycle bicycle\nInstanceCut [23] fine +coarse 15.8 13.0 27.9 10.0 8.0 23.7 14.0 19.5 15.2 9.3 4.7\nDWT [4] fine 19.8 15.6 30.0 15.1 11.7 32.9 17.1 20.4 15.0 7.9 4.9\nSAIS [17] fine - 17.4 36.7 14.6 12.9 35.7 16.0 23.2 19.0 10.3 7.8\nDIN [3] fine +coarse - 20.0 38.8 16.5 16.7 25.7 20.6 30.0 23.4 17.1 10.1\nSGN [29] fine +coarse 29.2 25.0 44.9 21.8 20.1 39.4 24.8 33.2 30.8 17.7 12.4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1079-6db4-11ee-9f52-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-50", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Mask R-CNN fine 31.5 26.2 49.9 30.5 23.7 46.9 22.8 32.2 18.6 19.1 16.0\nMask R-CNN fine + COCO 36.4 32.0 58.1 34.8 27.0 49.1 30.1 40.9 30.9 24.1 18.7\nTable 7. Results on Cityscapes val (\u2018AP [ val]\u2019 column) and test (remaining columns) sets. Our method uses ResNet-50-FPN.\nAppendix A: Experiments on Cityscapes\nWe further report instance segmentation results on the\nCityscapes [7] dataset. This dataset has fine annotations for 2975 train, 500 val, and 1525 test images. It has\n20kcoarse training images without instance annotations,\nwhich we do notuse. All images are 2048 \u00021024 pixels.\nThe instance segmentation task involves 8 object categories,\nwhose numbers of instances on the fine training set are:\nperson rider car truck bus train mcycle bicycle\n17.9k 1.8k 26.9k 0.5k 0.4k 0.2k 0.7k 3.7k\nInstance segmentation performance on this task is measured"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b107a-6db4-11ee-910a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-51", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "Instance segmentation performance on this task is measured\nby the COCO-style mask AP (averaged over IoU thresholds); AP 50(i.e., mask AP at an IoU of 0.5) is also reported.\nImplementation: We apply our Mask R-CNN models with\nthe ResNet-FPN-50 backbone; we found the 101-layer\ncounterpart performs similarly due to the small dataset size.\nWe train with image scale (shorter side) randomly sampled\nfrom [800, 1024], which reduces over\ufb01tting; inference is on\na single scale of 1024 pixels. We use a mini-batch size of\n1 image per GPU (so 8 on 8 GPUs) and train the model\nfor 24k iterations, starting from a learning rate of 0.01 and\nreducing it to 0.001 at 18k iterations. It takes \u00184 hours of\ntraining on a single 8-GPU machine under this setting.\nResults: Table 7 compares our results to the state of the\nart on the val andtest sets. Without using the coarse\ntraining set, our method achieves 26.2 AP on test , which\nis over 30% relative improvement over the previous best entry (DIN [3]), and is also better than the concurrent work of\nSGN\u2019s 25.0 [29]. Both DIN and SGN use fine +coarse"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b107b-6db4-11ee-b153-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-52", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "SGN\u2019s 25.0 [29]. Both DIN and SGN use fine +coarse\ndata. Compared to the best entry using fine data only\n(17.4 AP), we achieve a \u001850% improvement.\nFor the person andcarcategories, the Cityscapes dataset\nexhibits a large number of within -category overlapping instances (on average 6 people and 9 cars per image). We\nargue that within-category overlap is a core dif\ufb01culty of instance segmentation. Our method shows massive improvement on these two categories over the other best entries (relative\u001840% improvement on person from 21.8 to 30.5 and\n\u001820% improvement on carfrom 39.4 to 46.9), even though\nour method does not exploit the coarse data.\nA main challenge of the Cityscapes dataset is training\nmodels in a low-data regime, particularly for the categories\noftruck ,bus, and train , which have about 200-500 traincar:1.00car:0.98\ncar:0.98 car:0.95 car:0.81car:0.52person:1.00person:1.00person:1.00\nperson:1.00 person:1.00person:1.00person:1.00\nperson:1.00 person:1.00person:1.00"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b107c-6db4-11ee-bfa9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-53", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "person:1.00 person:1.00person:1.00person:1.00\nperson:1.00 person:1.00person:1.00\nperson:1.00person:1.00\nperson:1.00\nperson:1.00person:0.99\nperson:0.99person:0.99 person:0.99person:0.98\nperson:0.98person:0.98person:0.98person:0.94 person:0.94person:0.82\nperson:0.82person:0.79\nperson:0.73person:0.67person:0.66person:0.59\ntruck:0.66bus:1.00\nbus:0.95rider:0.59\nbicycle:0.83\nbicycle:0.56car:1.00car:1.00\ncar:1.00 car:1.00car:1.00car:1.00\ncar:1.00car:1.00 car:1.00car:0.99car:0.95\ncar:0.95car:0.95 car:0.69car:0.68 car:0.68car:0.64\ncar:0.57 car:0.52person:1.00"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b107d-6db4-11ee-af96-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-54", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "car:0.57 car:0.52person:1.00\nperson:0.99person:0.99person:0.99 person:0.99person:0.98person:0.98 person:0.98 person:0.97 person:0.93person:0.92\nperson:0.91person:0.86 person:0.84person:0.82 person:0.73person:0.72 person:0.72person:0.72 person:0.63 rider:0.68\ncar:1.00\ncar:1.00car:1.00car:1.00\ncar:1.00car:1.00car:1.00car:1.00\ncar:1.00 car:1.00\ncar:1.00car:1.00\ncar:1.00car:1.00car:1.00car:1.00car:1.00\ncar:0.98car:0.97\ncar:0.88car:0.76car:0.72car:0.72 car:0.65car:0.50person:1.00\nperson:1.00 person:0.98person:0.93person:0.85person:0.78person:0.73 person:0.58\nperson:1.00"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b107e-6db4-11ee-b372-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-55", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "person:1.00\nperson:1.00person:1.00\nperson:1.00\nperson:1.00person:1.00\nperson:1.00person:1.00person:1.00\nperson:1.00person:1.00person:1.00\nperson:1.00person:1.00\nperson:1.00person:1.00person:1.00\nperson:1.00person:1.00\nperson:1.00person:1.00\nperson:1.00person:0.99\nperson:0.99 person:0.98person:0.97\nperson:0.96person:0.92\nperson:0.91person:0.70 person:0.59\nbicycle:0.99bicycle:0.97car:1.00\ncar:1.00 car:0.99\ncar:0.89person:1.00\nperson:1.00person:1.00\nperson:1.00\nperson:1.00person:0.96person:0.93\nperson:0.89person:0.88person:0.75\nrider:0.94\ncar:1.00\ncar:1.00car:1.00\ncar:1.00car:1.00"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b107f-6db4-11ee-8e06-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-56", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "rider:0.94\ncar:1.00\ncar:1.00car:1.00\ncar:1.00car:1.00\ncar:1.00car:1.00 car:0.99car:0.89 car:0.67person:1.00\nperson:1.00\nperson:1.00 person:1.00 person:0.82\nbus:0.75Figure 8. Mask R-CNN results on Cityscapes test (32.0 AP).\nThe bottom-right image shows a failure prediction.\ning samples each. To partially remedy this issue, we further\nreport a result using COCO pre-training. To do this, we initialize the corresponding 7 categories in Cityscapes from a\npre-trained COCO Mask R-CNN model ( rider being randomly initialized). We \ufb01ne-tune this model for 4k iterations\nin which the learning rate is reduced at 3k iterations, which\ntakes\u00181 hour for training given the COCO model.\nThe COCO pre-trained Mask R-CNN model achieves\n32.0 AP on test , almost a 6 point improvement over the\nfine -only counterpart. This indicates the important role\nthe amount of training data plays. It also suggests that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1080-6db4-11ee-a2df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-57", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "32.0 AP on test , almost a 6 point improvement over the\nfine -only counterpart. This indicates the important role\nthe amount of training data plays. It also suggests that\nmethods on Cityscapes might be in\ufb02uenced by their lowshot learning performance. We show that using COCO pretraining is an effective strategy on this dataset.\nFinally, we observed a bias between the val andtest\nAP, as is also observed from the results of [23, 4, 29]. We\nfound that this bias is mainly caused by the truck ,bus,\nand train categories, with the fine -only model having\nval/test AP of 28.8/22.8, 53.5/32.2, and 33.0/18.6, respectively. This suggests that there is a domain shift on\nthese categories, which also have little training data. COCO\npre-training helps to improve results the most on these categories; however, the domain shift persists with 38.0/30.1,\n57.5/40.9, and 41.2/30.9 val/test AP, respectively. Note\nthat for the person andcarcategories we do not see any\nsuch bias ( val/test AP are within\u00061point).\nExample results on Cityscapes are shown in Figure 8.\n9\ndescription backbone AP AP 50 AP75 APbbAPbb"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1081-6db4-11ee-8829-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-58", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "such bias ( val/test AP are within\u00061point).\nExample results on Cityscapes are shown in Figure 8.\n9\ndescription backbone AP AP 50 AP75 APbbAPbb\n50APbb\n75\noriginal baseline X-101-FPN 36.7 59.5 38.9 39.6 61.5 43.2\n+updated baseline X-101-FPN 37.0 59.7 39.0 40.5 63.0 43.7\n+e2e training X-101-FPN 37.6 60.4 39.9 41.7 64.1 45.2\n+ImageNet-5k X-101-FPN 38.6 61.7 40.9 42.7 65.1 46.6\n+train-time augm. X-101-FPN 39.2 62.5 41.6 43.5 65.9 47.2\n+deeper X-152-FPN 39.7 63.2 42.2 44.1 66.4 48.4\n+Non-local [43] X-152-FPN-NL 40.3 64.4 42.8 45.0 67.8 48.9\n+test-time augm. X-152-FPN-NL 41.8 66.0 44.8 47.3 69.3 51.5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1082-6db4-11ee-853e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-59", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "+test-time augm. X-152-FPN-NL 41.8 66.0 44.8 47.3 69.3 51.5\nTable 8. Enhanced detection results of Mask R-CNN on COCO\nminival . Each row adds an extra component to the above row.\nWe denote ResNeXt model by \u2018X\u2019 for notational brevity.\nAppendix B: Enhanced Results on COCO\nAs a general framework, Mask R-CNN is compatible with complementary techniques developed for detection/segmentation, including improvements made to\nFast/Faster R-CNN and FCNs. In this appendix we describe some techniques that improve over our original results. Thanks to its generality and \ufb02exibility, Mask R-CNN\nwas used as the framework by the three winning teams in\nthe COCO 2017 instance segmentation competition, which\nall signi\ufb01cantly outperformed the previous state of the art.\nInstance Segmentation and Object Detection\nWe report some enhanced results of Mask R-CNN in Table 8. Overall, the improvements increase mask AP 5.1\npoints (from 36.7 to 41.8) and box AP 7.7 points (from 39.6\nto 47.3). Each model improvement increases both mask AP\nand box AP consistently, showing good generalization of\nthe Mask R-CNN framework. We detail the improvements"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1083-6db4-11ee-9973-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-60", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "to 47.3). Each model improvement increases both mask AP\nand box AP consistently, showing good generalization of\nthe Mask R-CNN framework. We detail the improvements\nnext. These results, along with future updates, can be reproduced by our released code at https://github.com/\nfacebookresearch/Detectron , and can serve as\nhigher baselines for future research.\nUpdated baseline: We start with an updated baseline\nwith a different set of hyper-parameters. We lengthen the\ntraining to 180k iterations, in which the learning rate is reduced by 10 at 120k and 160k iterations. We also change\nthe NMS threshold to 0.5 (from a default value of 0.3). The\nupdated baseline has 37.0 mask AP and 40.5 box AP.\nEnd-to-end training: All previous results used stagewise training, i.e., training RPN as the \ufb01rst stage and Mask\nR-CNN as the second. Following [37], we evaluate endto-end (\u2018e2e\u2019) training that jointly trains RPN and Mask RCNN. We adopt the \u2018approximate\u2019 version in [37] that only\ncomputes partial gradients in the RoIAlign layer by ignoring the gradient w.r.t. RoI coordinates. Table 8 shows that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1084-6db4-11ee-a1b2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-61", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "computes partial gradients in the RoIAlign layer by ignoring the gradient w.r.t. RoI coordinates. Table 8 shows that\ne2e training improves mask AP by 0.6 and box AP by 1.2.\nImageNet-5k pre-training: Following [45], we experiment with models pre-trained on a 5k-class subset of ImageNet (in contrast to the standard 1k-class subset). This 5 \u0002\nincrease in pre-training data improves both mask and box 1\nAP. As a reference, [40] used \u0018250\u0002more images (300M)\nand reported a 2-3 box AP improvement on their baselines.description backbone APkpAPkp\n50APkp\n75APkp\nMAPkp\nL\noriginal baseline R-50-FPN 64.2 86.6 69.7 58.7 73.0\n+updated baseline R-50-FPN 65.1 86.6 70.9 59.9 73.6\n+deeper R-101-FPN 66.1 87.7 71.7 60.5 75.0\n+ResNeXt X-101-FPN 67.3 88.0 73.3 62.2 75.6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1085-6db4-11ee-9c8c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-62", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "+ResNeXt X-101-FPN 67.3 88.0 73.3 62.2 75.6\n+data distillation [35] X-101-FPN 69.1 88.9 75.3 64.1 77.1\n+test-time augm. X-101-FPN 70.4 89.3 76.8 65.8 78.1\nTable 9. Enhanced keypoint results of Mask R-CNN on COCO\nminival . Each row adds an extra component to the above row.\nHere we use only keypoint annotations but no mask annotations.\nWe denote ResNet by \u2018R\u2019 and ResNeXt by \u2018X\u2019 for brevity.\nTrain-time augmentation: Scale augmentation at train\ntime further improves results. During training, we randomly\nsample a scale from [640, 800] pixels and we increase the\nnumber of iterations to 260k (with the learning rate reduced\nby 10 at 200k and 240k iterations). Train-time augmentation improves mask AP by 0.6 and box AP by 0.8.\nModel architecture: By upgrading the 101-layer\nResNeXt to its 152-layer counterpart [19], we observe an\nincrease of 0.5 mask AP and 0.6 box AP. This shows a\ndeeper model can still improve results on COCO."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1086-6db4-11ee-aaac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-63", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "increase of 0.5 mask AP and 0.6 box AP. This shows a\ndeeper model can still improve results on COCO.\nUsing the recently proposed non-local (NL) model [43],\nwe achieve 40.3 mask AP and 45.0 box AP. This result is\nwithout test-time augmentation, and the method runs at 3fps\non an Nvidia Tesla P100 GPU at test time.\nTest-time augmentation: We combine the model results\nevaluated using scales of [400, 1200] pixels with a step of\n100 and on their horizontal \ufb02ips. This gives us a singlemodel result of 41.8 mask AP and 47.3 box AP.\nThe above result is the foundation of our submission to\nthe COCO 2017 competition (which also used an ensemble,\nnot discussed here). The \ufb01rst three winning teams for the\ninstance segmentation task were all reportedly based on an\nextension of the Mask R-CNN framework.\nKeypoint Detection\nWe report enhanced results of keypoint detection in Table 9. As an updated baseline, we extend the training schedule to 130k iterations in which the learning rate is reduced\nby 10 at 100k and 120k iterations. This improves APkpby\nabout 1 point. Replacing ResNet-50 with ResNet-101 and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1087-6db4-11ee-8c4a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-64", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "by 10 at 100k and 120k iterations. This improves APkpby\nabout 1 point. Replacing ResNet-50 with ResNet-101 and\nResNeXt-101 increases APkpto 66.1 and 67.3, respectively.\nWith a recent method called data distillation [35], we are\nable to exploit the additional 120k unlabeled images provided by COCO. In brief, data distillation is a self-training\nstrategy that uses a model trained on labeled data to predict annotations on unlabeled images, and in turn updates\nthe model with these new annotations. Mask R-CNN provides an effective framework for such a self-training strategy. With data distillation, Mask R-CNN APkpimprove by\n1.8 points to 69.1. We observe that Mask R-CNN can bene\ufb01t from extra data, even if that data is unlabeled .\nBy using the same test-time augmentation as used for\ninstance segmentation, we further boost APkpto 70.4.\n10\nAcknowledgements: We would like to acknowledge Ilija\nRadosavovic for contributions to code release and enhanced\nresults, and the Caffe2 team for engineering support.\nReferences\n[1] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2D"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1088-6db4-11ee-8735-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-65", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "References\n[1] M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2D\nhuman pose estimation: New benchmark and state of the art\nanalysis. In CVPR , 2014. 8\n[2] P. Arbel \u00b4aez, J. Pont-Tuset, J. T. Barron, F. Marques, and\nJ. Malik. Multiscale combinatorial grouping. In CVPR ,\n2014. 2\n[3] A. Arnab and P. H. Torr. Pixelwise instance segmentation\nwith a dynamically instantiated network. In CVPR , 2017. 3,\n9\n[4] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In CVPR , 2017. 3, 9\n[5] S. Bell, C. L. Zitnick, K. Bala, and R. Girshick. Insideoutside net: Detecting objects in context with skip pooling\nand recurrent neural networks. In CVPR , 2016. 5\n[6] Z. Cao, T. Simon, S.-E. Wei, and Y . Sheikh. Realtime multiperson 2d pose estimation using part af\ufb01nity \ufb01elds. In CVPR ,\n2017. 7, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1089-6db4-11ee-b2e9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-66", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "2017. 7, 8\n[7] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,\nR. Benenson, U. Franke, S. Roth, and B. Schiele. The\nCityscapes dataset for semantic urban scene understanding.\nInCVPR , 2016. 9\n[8] J. Dai, K. He, Y . Li, S. Ren, and J. Sun. Instance-sensitive\nfully convolutional networks. In ECCV , 2016. 2\n[9] J. Dai, K. He, and J. Sun. Convolutional feature masking for\njoint object and stuff segmentation. In CVPR , 2015. 2\n[10] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 2, 3,\n4, 5, 6\n[11] J. Dai, Y . Li, K. He, and J. Sun. R-FCN: Object detection via\nregion-based fully convolutional networks. In NIPS , 2016. 2\n[12] R. Girshick. Fast R-CNN. In ICCV , 2015. 1, 2, 3, 4, 6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b108a-6db4-11ee-a517-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-67", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "[12] R. Girshick. Fast R-CNN. In ICCV , 2015. 1, 2, 3, 4, 6\n[13] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic\nsegmentation. In CVPR , 2014. 2, 3\n[14] R. Girshick, F. Iandola, T. Darrell, and J. Malik. Deformable\npart models are convolutional neural networks. In CVPR ,\n2015. 4\n[15] B. Hariharan, P. Arbel \u00b4aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV . 2014. 2\n[16] B. Hariharan, P. Arbel \u00b4aez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and \ufb01ne-grained localization. In CVPR , 2015. 2\n[17] Z. Hayder, X. He, and M. Salzmann. Shape-aware instance\nsegmentation. In CVPR , 2017. 9\n[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling\nin deep convolutional networks for visual recognition. In"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b108b-6db4-11ee-a1f8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-68", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "[18] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling\nin deep convolutional networks for visual recognition. In\nECCV . 2014. 1, 2\n[19] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning\nfor image recognition. In CVPR , 2016. 2, 4, 7, 10\n[20] J. Hosang, R. Benenson, P. Doll \u00b4ar, and B. Schiele. What\nmakes for effective detection proposals? PAMI , 2015. 2[21] J. Huang, V . Rathod, C. Sun, M. Zhu, A. Korattikara,\nA. Fathi, I. Fischer, Z. Wojna, Y . Song, S. Guadarrama, et al.\nSpeed/accuracy trade-offs for modern convolutional object\ndetectors. In CVPR , 2017. 2, 3, 4, 6, 7\n[22] M. Jaderberg, K. Simonyan, A. Zisserman, and\nK. Kavukcuoglu. Spatial transformer networks. In\nNIPS , 2015. 4\n[23] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b108c-6db4-11ee-8be0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-69", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "NIPS , 2015. 4\n[23] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and\nC. Rother. Instancecut: from edges to instances with multicut. In CVPR , 2017. 3, 9\n[24] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classi\ufb01cation with deep convolutional neural networks. In NIPS ,\n2012. 2\n[25] Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E.\nHoward, W. Hubbard, and L. D. Jackel. Backpropagation\napplied to handwritten zip code recognition. Neural computation , 1989. 2\n[26] Y . Li, H. Qi, J. Dai, X. Ji, and Y . Wei. Fully convolutional\ninstance-aware semantic segmentation. In CVPR , 2017. 2,\n3, 5, 6\n[27] T.-Y . Lin, P. Doll \u00b4ar, R. Girshick, K. He, B. Hariharan, and\nS. Belongie. Feature pyramid networks for object detection.\nInCVPR , 2017. 2, 4, 5, 7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b108d-6db4-11ee-a324-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-70", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "S. Belongie. Feature pyramid networks for object detection.\nInCVPR , 2017. 2, 4, 5, 7\n[28] T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll \u00b4ar, and C. L. Zitnick. Microsoft COCO: Common objects in context. In ECCV , 2014. 2, 5\n[29] S. Liu, J. Jia, S. Fidler, and R. Urtasun. SGN: Sequential grouping networks for instance segmentation. In ICCV ,\n2017. 3, 9\n[30] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional\nnetworks for semantic segmentation. In CVPR , 2015. 1, 3, 6\n[31] V . Nair and G. E. Hinton. Recti\ufb01ed linear units improve restricted boltzmann machines. In ICML , 2010. 4\n[32] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy. Towards accurate multiperson pose estimation in the wild. In CVPR , 2017. 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b108e-6db4-11ee-a008-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-71", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "[33] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015. 2, 3\n[34] P. O. Pinheiro, T.-Y . Lin, R. Collobert, and P. Doll \u00b4ar. Learning to re\ufb01ne object segments. In ECCV , 2016. 2, 3\n[35] I. Radosavovic, P. Doll \u00b4ar, R. Girshick, G. Gkioxari, and\nK. He. Data distillation: Towards omni-supervised learning.\narXiv:1712.04440 , 2017. 10\n[36] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 1, 2, 3, 4, 7\n[37] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In TPAMI , 2017. 10\n[38] A. Shrivastava, A. Gupta, and R. Girshick. Training regionbased object detectors with online hard example mining. In\nCVPR , 2016. 2, 5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b108f-6db4-11ee-8738-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-72", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "CVPR , 2016. 2, 5\n[39] A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta. Beyond skip connections: Top-down modulation for object detection. arXiv:1612.06851 , 2016. 4, 7\n[40] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting\nunreasonable effectiveness of data in deep learning era. In\nICCV , 2017. 10\n11\n[41] C. Szegedy, S. Ioffe, and V . Vanhoucke. Inception-v4,\ninception-resnet and the impact of residual connections on\nlearning. In ICLR Workshop , 2016. 7\n[42] J. R. Uijlings, K. E. van de Sande, T. Gevers, and A. W.\nSmeulders. Selective search for object recognition. IJCV ,\n2013. 2\n[43] X. Wang, R. Girshick, A. Gupta, and K. He. Non-local neural\nnetworks. arXiv:1711.07971 , 2017. 10"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1090-6db4-11ee-8607-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1703.06870-73", "title": "Mask R-CNN", "source": "http://arxiv.org/pdf/1703.06870", "start_index": 0, "chroma:document": "networks. arXiv:1711.07971 , 2017. 10\n[44] S.-E. Wei, V . Ramakrishna, T. Kanade, and Y . Sheikh. Convolutional pose machines. In CVPR , 2016. 8\n[45] S. Xie, R. Girshick, P. Doll \u00b4ar, Z. Tu, and K. He. Aggregated\nresidual transformations for deep neural networks. In CVPR ,\n2017. 4, 10\n12"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1091-6db4-11ee-b45a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-0", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network\nAnurag Arnab and Philip H.S Torr\nUniversity of Oxford\n{anurag.arnab, philip.torr }@eng.ox.ac.uk\nAbstract\nSemantic segmentation and object detection research\nhave recently achieved rapid progress. However, the former\ntask has no notion of different instances of the same object,\nand the latter operates at a coarse, bounding-box level. We\npropose an Instance Segmentation system that produces a\nsegmentation map where each pixel is assigned an object\nclass and instance identity label. Most approaches adapt\nobject detectors to produce segments instead of boxes. In\ncontrast, our method is based on an initial semantic segmentation module, which feeds into an instance subnetwork.\nThis subnetwork uses the initial category-level segmentation, along with cues from the output of an object detector,\nwithin an end-to-end CRF to predict instances. This part\nof our model is dynamically instantiated to produce a variable number of instances per image. Our end-to-end approach requires no post-processing and considers the image holistically, instead of processing independent proposals. Therefore, unlike some related work, a pixel cannot belong to multiple instances. Furthermore, far more precise\nsegmentations are achieved, as shown by our substantial"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1092-6db4-11ee-a4e4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-1", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "segmentations are achieved, as shown by our substantial\nimprovements at high APrthresholds.\n1. Introduction\nSemantic segmentation and object detection are wellstudied scene understanding problems, and have recently\nwitnessed great progress due to deep learning [ 22,13,7].\nHowever, semantic segmentation \u2013 which labels every pixel\nin an image with its object class \u2013 has no notion of different\ninstances of an object (Fig. 1). Object detection does localise different object instances, but does so at a very coarse,\nbounding-box level. Instance segmentation localises objects at a pixel level, as shown in Fig. 1, and can be thought\nof being at the intersection of these two scene understanding\ntasks. Unlike the former, it knows about different instances\nof the same object, and unlike the latter, it operates at a pixel\nlevel. Accurate recognition and localisation of objects enables many applications, such as autonomous driving [ 9],\nimage-editing [ 53] and robotics [ 17].\nMany recent approaches to instance segmentation arebased on object detection pipelines where objects are \ufb01rst\nlocalised with bounding boxes. Thereafter, each bounding"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1093-6db4-11ee-93fd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-2", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "Many recent approaches to instance segmentation arebased on object detection pipelines where objects are \ufb01rst\nlocalised with bounding boxes. Thereafter, each bounding\nbox is re\ufb01ned into a segmentation [ 19,20,32,37,30]. Another related approach [ 12,56] is to use segment-based region proposals [ 10,41,42] instead of box-based proposals.\nHowever, these methods do not consider the entire image,\nbut rather independent proposals. As a result, occlusions\nbetween different objects are not handled. Furthermore,\nmany of these methods cannot easily produce segmentation\nmaps of the image, as shown in Fig. 1, since they process\nnumerous proposals independently. There are typically far\nmore proposals than actual objects in the image, and these\nproposals can overlap and be assigned different class labels.\nFinally, as these methods are based on an initial detection\nstep, they cannot recover from false detections.\nOur proposed method is inspired by the fact that instance\nsegmentation can be viewed as a more complex form of semantic segmentation, since we are not only required to label the object class of each pixel, but also its instance identity. We produce a pixelwise segmentation of the image,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1094-6db4-11ee-98b3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-3", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "where each pixel is assigned both a semantic class and instance label. Our end-to-end trained network, which outputs a variable number of instances per input image, begins with an initial semantic segmentation module. The\nfollowing, dynamic part of the network, then uses information from an object detector and a Conditional Random\nField (CRF) model to distinguish different instances. This\napproach is robust to false-positive detections, as well as\npoorly localised bounding boxes which do not cover the\nentire object, in contrast to detection-based methods to instance segmentation. Moreover, as it considers the entire\nimage when making predictions, it attempts to resolve occlusions between different objects and can produce segmentation maps as in Fig. 1without any post-processing.\nFurthermore, we note that the Average Precision (AP)\nmetric [ 14] used in evaluating object detection systems, and\nitsAPrvariant [ 19] used for instance segmentation, considers individual, potentially overlapping, object predictions in\nisolation, as opposed to the entire image. To evaluate methods such as ours, which produce complete segmentation\nmaps and reason about occlusions, we also evaluate using\n1\narXiv:1704.02386v1  [cs.CV]  7 Apr 2017\n(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1095-6db4-11ee-b523-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-4", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "(a) Object Detection (b) Semantic Segmentation (c) Instance Segmentation\nFigure 1: Object detection (a) localises the different people, but at a coarse, bounding-box level. Semantic segmentation\n(b) labels every pixel, but has no notion of instances. Instance segmentation (c) labels each pixel of each person uniquely.\nOur proposed method jointly produces both semantic and instance segmentations. Our method uses the output of an object\ndetector as a cue to identify instances, but is robust to false positive detections, poor bounding box localisation and occlusions.\nBest viewed in colour.\nthe \u201cMatching Intersection over Union\u201d metric.\nOur system, which is based on an initial semantic segmentation subnetwork, produces sharp and accurate instance segmentations. This is re\ufb02ected by the substantial improvements we achieve over state-of-the-art methods\nat highAPrthresholds on the Pascal VOC and Semantic\nBoundaries datasets. Furthermore, our network improves\non the semantic segmentation task while being trained for\nthe related task of instance segmentation.\n2. Related Work\nAn early work on instance segmentation was by Winn\nand Shotton [ 51]. A per-pixel unary classi\ufb01er was trained"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1096-6db4-11ee-ba7f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-5", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "2. Related Work\nAn early work on instance segmentation was by Winn\nand Shotton [ 51]. A per-pixel unary classi\ufb01er was trained\nto predict parts of an object. These parts were then encouraged to maintain a spatial ordering, that is characteristic of\nan instance, using asymmetric pairwise potentials in a Conditional Random Field (CRF). Subsequent work [ 54], presented another approach where detection outputs of DPM\n[15], with associated foreground masks, were assigned a\ndepth ordering using a generative, probabilistic model. This\ndepth ordering resolved occlusions.\nHowever, instance segmentation has become more common after the \u201cSimultaneous Detection and Segmentation\u201d\n(SDS) work of Hariharan et al. [19]. This system was based\non the R-CNN pipeline [ 16]: Region proposals, generated\nby the method of [ 1], were classi\ufb01ed into object categories\nwith a Convolutional Neural Network (CNN) before applying bounding-box regression as post-processing. A classspeci\ufb01c segmentation was then performed in this bounding"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1097-6db4-11ee-ad59-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-6", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "with a Convolutional Neural Network (CNN) before applying bounding-box regression as post-processing. A classspeci\ufb01c segmentation was then performed in this bounding\nbox to simultaneously detect and segment the object. Numerous works [ 20,8,30] have extended this pipeline. However, approaches that segment instances by re\ufb01ning detections [ 19,20,8,11,30] are inherently limited by the quality of the initial proposals. This problem is exacerbated by\nthe fact that this pipeline consists of several different modules trained with different objective functions. Furthermore,numerous post-processing steps such as \u201csuperpixel projection\u201d and rescoring are performed. Dai et al. [12] addressed\nsome of these issues by designing one end-to-end trained\nnetwork that generates box-proposals, creates foreground\nmasks from these proposals and then classi\ufb01es these masks.\nThis network can be seen as an extension of the end-to-end\nFaster-RCNN [ 44] detection framework, which generates\nbox-proposals and classi\ufb01es them. Additionally, Liu et al.\n[37] formulated an end-to-end version of the SDS network\n[19], whilst [ 32] iteratively re\ufb01ned object proposals.\nOn a separate track, algorithms have also been developed"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1098-6db4-11ee-9714-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-7", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "[19], whilst [ 32] iteratively re\ufb01ned object proposals.\nOn a separate track, algorithms have also been developed\nthat do not require object detectors. Zhang et al. [57,58]\nsegmented car instances by predicting the depth ordering\nof each pixel in the image. Unlike the previous detectionbased approaches, this method reasoned globally about all\ninstances in the image simultaneously (rather than individual proposals) with an MRF-based formulation. However,\ninference of this graphical model was not performed end-toend as shown to be possible in [ 60,2,5,34]. Furthermore,\nalthough this method does not use object detections, it is\ntrained with ground truth depth and assumes a maximum\nof nine cars in an image. Predicting all the instances in\nan image simultaneously (rather than classifying individual\nproposals) requires a model to be able to handle a variable\nnumber of output instances per image. As a result, [ 45]\nproposed a Recurrent Neural Network (RNN) for this task.\nHowever, this model was only for a single object category.\nOur proposed method not only outputs a variable number of\ninstances, but can also handle multiple object classes.\nLiang et al. [33] developed another proposal-free method\nbased on the semantic segmentation network of [ 6]. The"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b1099-6db4-11ee-b40a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-8", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "instances, but can also handle multiple object classes.\nLiang et al. [33] developed another proposal-free method\nbased on the semantic segmentation network of [ 6]. The\ncategory-level segmentation, along with CNN features, was\nused to predict instance-level bounding boxes. The number\nof instances of each class was also predicted to enable a \ufb01nal\nspectral clustering step. However, this additional information predicted by Liang\u2019s network could have been obtained\n Semantic Segmentation subnetwork  Instance Segmentation subnetwork Instance Unary Potentials Pixelwise \nCNN Semantic CRF Detector Box \nGlobal \nShape Instance CRF \nInput Image Instance Segmentation \nSemantic Segmentation End-to-end Network \nK+1 \nD+1 \nFigure 2: Network overview: Our end-to-end trained network consists of semantic- and instance-segmentation modules. The\nintermediate category-level segmentation, along with the outputs of an object detector, are used to reason about instances.\nThis is done by instance unary terms which use information from the detector\u2019s bounding boxes, the initial semantic segmentation and also the object\u2019s shape. A \ufb01nal CRF is used to combine all this information together to obtain an instance"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b109a-6db4-11ee-a96f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-9", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "segmentation. The output of the semantic segmentation module is a \ufb01xed size W\u00d7H\u00d7(K+ 1) tensor where Kis the\nnumber of object classes, excluding background, in the dataset. The \ufb01nal output, however, is of a variable W\u00d7H\u00d7(D+ 1)\ndimensions where Dis the number of detected objects (and one background label).\nfrom an object detector. Arnab et al. [3] also started with an\ninitial semantic segmentation network [ 2], and combined\nthis with the outputs of an object detector using a CRF to\nreason about instances. This method was not trained endto-end though, and could not really recover from errors in\nbounding-box localisation or occlusion.\nOur method also has an initial semantic segmentation\nsubnetwork, and uses the outputs of an object detector.\nHowever, in contrast to [ 3] it is trained end-to-end to improve on both semantic- and instance-segmentation performance (to our knowledge, this is the \ufb01rst work to achieve\nthis). Furthermore, it can handle detector localisation errors and occlusions better due to the energy terms in our\nend-to-end CRF. In contrast to detection-based approaches\n[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b109b-6db4-11ee-af18-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-10", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in\nFig. 1(c) and reasons about the entire image, rather than\nindependent proposals. This global reasoning allows our\nmethod to produce more accurate segmentations. Our proposed system also handles a variable number of instances\nper image, and thus does not assume a maximum number\nof instances like [ 57,58].\n3. Proposed Approach\nOur network (Fig. 2) contains an initial semantic segmentation module. We use the semantic segmentation result, along with the outputs of an object detector, to compute\nthe unary potentials of a Conditional Random Field (CRF)\nde\ufb01ned over object instances. We perform mean \ufb01eld inference in this random \ufb01eld to obtain the Maximum a Posteriori (MAP) estimate, which is our labelling. Although ournetwork consists of two conceptually different parts \u2013 a semantic segmentation module, and an instance segmentation\nnetwork \u2013 the entire pipeline is fully differentiable, given\nobject detections, and trained end-to-end.\n3.1. Semantic Segmentation subnetwork\nSemantic Segmentation assigns each pixel in an image a\nsemantic class label from a given set, L. In our case, this\nmodule uses the FCN8s architecture [ 38] which is based"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b109c-6db4-11ee-8de9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-11", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "semantic class label from a given set, L. In our case, this\nmodule uses the FCN8s architecture [ 38] which is based\non the VGG [ 47] ImageNet model. For better segmentation results, we include mean \ufb01eld inference of a Conditional Random Field as the last layer of this module. This\nCRF contains the densely-connected pairwise potentials described in [ 26] and is formulated as a recurrent neural network as in [ 60]. Additionally, we include the Higher Order\ndetection potential described in [ 2]. This detection potential has two primary bene\ufb01ts: Firstly, it improves semantic\nsegmentation quality by encouraging consistency between\nobject detections and segmentations. Secondly, it also recalibrates detection scores. This detection potential is similar to the one previously proposed by [ 28], [48], [52] and\n[55], but formulated for the differentiable mean \ufb01eld inference algorithm. We employ this potential as we are already\nusing object detection information for identifying object instances in the next stage. We denote the output at the semantic segmentation module of our network as the tensor\nQ, whereQi(l)denotes the probability (obtained by applying the softmax function on the network\u2019s activations) of\npixelitaking on the label l\u2208 L."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b109d-6db4-11ee-8fa5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-12", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "pixelitaking on the label l\u2208 L.\n(a) Semantic Segmentation (b) Instance Segmentation\nFigure 3: Instance segmentation using only the \u201cBox\u201d unary\npotential. This potential is effective when we have a good\ninitial semantic segmentation (a). Occlusions between objects of the same class can be resolved by the pairwise term\nbased on appearance differences. Note that we can ignore\nthe con\ufb01dent, false-positive \u201cbottle\u201d detections (b). This is\nin contrast to methods such as [ 8,19,20,30] which cannot\nrecover from detection errors.\n3.2. Instance Segmentation subnetwork\nAt the input to our instance segmentation subnetwork,\nwe assume that we have two inputs available: The semantic\nsegmentation predictions, Q, for each pixel and label, and a\nset of object detections. For each input image, we assume\nthat there are Dobject detections, and that the ithdetection\nis of the form (li,si,Bi)whereli\u2208 Lis the detected class\nlabel,si\u2208[0,1]is the con\ufb01dence score and Biis the set of\nindices of the pixels falling within the detector\u2019s bounding"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b109e-6db4-11ee-b5af-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-13", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "label,si\u2208[0,1]is the con\ufb01dence score and Biis the set of\nindices of the pixels falling within the detector\u2019s bounding\nbox. Note that the number Dvaries for every input image.\nThe problem of instance segmentation can then be\nthought of as assigning every pixel to either a particular object detection, or the background label. This is based on\nthe assumption that every object detection speci\ufb01es a potential object instance. We de\ufb01ne a multinomial random\nvariable,V, at each of the Npixels in the image, and\nV= [V1V2...V N]T. Each variable at pixel i,Vi, is assigned a label corresponding to its instance. This label set,\n{0,1,2,...,D }changes for each image since D, the number\nof detections, varies for every image (0 is the background\nlabel). In the case of instance segmentation of images, the\nquality of a prediction is invariant to the permutations of the\ninstance labelling. For example, labelling the \u201cblue person\u201d\nin Fig. 1(c) as \u201c1\u201d and the \u201cpurple person\u201d as \u201c2\u201d is no different to labelling them as \u201c2\u201d and \u201c1\u201d respectively. This"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b109f-6db4-11ee-a88b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-14", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "condition is handled by our loss function in Sec. 3.4.\nNote that unlike works such as [ 57] and [ 58] we do not\nassume a maximum number of possible instances and keep\na \ufb01xed label set. Furthermore, since we are considering object detection outputs jointly with semantic segmentation\npredictions, we have some robustness to high-scoring false\npositive detections unlike methods such as [ 8,20,37] which\nre\ufb01ne object detections into segmentations.\nWe formulate a Conditional Random Field over our instance variables, V, which consists of unary and pairwise\n(a) Only Box term (b) Box and Global terms\nFigure 4: The \u201cGlobal\u201d unary potential (b) is particularly\neffective in cases where the input detection bounding box\ndoes not cover the entire extent of the object. Methods\nwhich are based on re\ufb01ning bounding-box detections such\nas [19,20,8,12] cannot cope with poorly localised detections. Note, the overlaid detection boxes are an additional\ninput to our system.\nenergies. The energy of the assignment vto all the variables,V, is\nE(V=v) =/summationdisplay\niU(vi) +/summationdisplay\ni<jP(vi,vj). (1)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a0-6db4-11ee-bcea-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-15", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "E(V=v) =/summationdisplay\niU(vi) +/summationdisplay\ni<jP(vi,vj). (1)\nThe unary energy is a sum of three terms, which take into\naccount the object detection bounding boxes, the initial semantic segmentation and shape information,\nU(vi) =\u2212ln[w1\u03c8Box(vi) +w2\u03c8Global (vi)+\nw3\u03c8Shape (vi)], (2)\nand are described further in Sections 3.2.1 through 3.2.3 .\nw1,w2andw3are all weighting co-ef\ufb01cients learned via\nbackpropagation.\n3.2.1 Box Term\nThis potential encourages a pixel to be assigned to the instance corresponding to the kthdetection if it falls within\nthe detection\u2019s bounding box. This potential is proportional\nto the probability of the pixel\u2019s semantic class being equal\nto the detected class Qi(lk)and the detection score, sk.\n\u03c8Box(Vi=k) =/braceleftBigg\nQi(lk)skifi\u2208Bk\n0 otherwise(3)\nAs shown in Fig. 3, this potential performs well when the\ninitial semantic segmentation is good. It is robust to false"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a1-6db4-11ee-bd99-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-16", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "0 otherwise(3)\nAs shown in Fig. 3, this potential performs well when the\ninitial semantic segmentation is good. It is robust to false\npositive detections, unlike methods which re\ufb01ne bounding\nboxes [ 8,19,20] since the detections are considered in light\nof our initial semantic segmentation, Q. Together with the\npairwise term (Sec. 3.2.4 ), occlusions between objects of\nthe same class can be resolved if there are appearance differences in the different instances.\n3.2.2 Global Term\nThis term does not rely on bounding boxes, but only the segmentation prediction at a particular pixel, Qi. It encodes the\nintuition that if we only know there are dpossible instances\nof a particular object class, and have no further localisation information, each instance is equally probable, and this\npotential is proportional to the semantic segmentation con\ufb01dence for the detected object class at that pixel:\n\u03c8Global (Vi=k) =Qi(lk). (4)\nAs shown in Fig. 4, this potential overcomes cases where\nthe bounding box does not cover the entire extent of the object, as it assigns probability mass to a particular instance"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a2-6db4-11ee-87af-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-17", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "As shown in Fig. 4, this potential overcomes cases where\nthe bounding box does not cover the entire extent of the object, as it assigns probability mass to a particular instance\nlabel throughout all pixels in the image. This is also bene\ufb01cial during training, as it ensures that the \ufb01nal output is\ndependent on the segmentation prediction at all pixels in the\nimage, leading to error gradients that are more stable across\nbatches and thus more amenable to backpropagation.\n3.2.3 Shape Term\nWe also incorporate shape priors to help us reason about occlusions involving multiple objects of the same class, which\nmay have minimal appearance variation between them, as\nshown in Fig. 5. In such cases, a prior on the expected shape\nof an object category can help us to identify the foreground\ninstance within a bounding box. Previous approaches to incorporating shape priors in segmentation [ 23,8,50] have\ninvolved generating \u201cshape exemplars\u201d from the training\ndataset and, at inference time, matching these exemplars to\nobject proposals using the Chamfer distance [ 46,36].\nWe propose a fully differentiable method: Given a set of\nshape templates, T, we warp each shape template using bilinear interpolation into \u02dcTso that it matches the dimensions"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a3-6db4-11ee-9850-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-18", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "We propose a fully differentiable method: Given a set of\nshape templates, T, we warp each shape template using bilinear interpolation into \u02dcTso that it matches the dimensions\nof thekthbounding box, Bk. We then select the shape prior\nwhich matches the segmentation prediction for the detected\nclass within the bounding box, QBk(lk), the best according to the normalised cross correlation. Our shape prior is\nthen the Hadamard (elementwise) product ( \u2299) between the\nsegmentation unaries and the matched shape prior:\nt\u2217= arg max\nt\u2208\u02dcT/summationtextQBk(lk)\u2299t/vextenddouble/vextenddoubleQBk(lk)/vextenddouble/vextenddouble/bardblt/bardbl(5)\n\u03c8(VBk=k) =QBk(lk)\u2299t\u2217. (6)\nEquations 5and6can be seen as a special case of maxpooling, and the numerator of Eq. 5is simply a convolution\nthat produces a scalar output since the two arguments are of\nequal dimension. Additionally, during training, we can consider the shape priors Tas parameters of our \u201cshape term\u201d\nlayer and backpropagate through to the matched exemplar"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a4-6db4-11ee-a712-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-19", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "equal dimension. Additionally, during training, we can consider the shape priors Tas parameters of our \u201cshape term\u201d\nlayer and backpropagate through to the matched exemplar\nt\u2217to update it. In practice, we initialised these parameters\n(a) Without shape term (b) With Shape term\nFigure 5: The \u201cShape\u201d unary potential (b) helps us to distinguish between the green and purple sheep, which the other\ntwo unary potentials cannot. Input detections are overlaid\non the images.\nwith the shape priors described in [ 50]. This consists of\nroughly 250 shape templates for each of \ufb01ve different aspect ratios. These were obtained by clustering foreground\nmasks of object instances from the training set.\nHere, we have only matched a single shape template to\na proposed instance. This method could be extended in future to matching multiple templates to an instance, in which\ncase each shape exemplar would correspond to a part of the\nobject such as in DPM [ 15].\n3.2.4 Pairwise term\nThe pairwise term consists of densely-connected Gaussian\npotentials [ 26] and encourages appearance and spatial consistency. The weights governing the importance of these\nterms are also learnt via backpropagation, as in [ 60]. We"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a5-6db4-11ee-af9e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-20", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "potentials [ 26] and encourages appearance and spatial consistency. The weights governing the importance of these\nterms are also learnt via backpropagation, as in [ 60]. We\n\ufb01nd that these priors are useful in the case of instance segmentation as well, since nearby pixels that have similar appearance often belong to the same object instance. They are\noften able to resolve occlusions based on appearance differences between objects of the same class (Fig. 3).\n3.3. Inference of our Dynamic Instance CRF\nWe use mean \ufb01eld inference to approximately minimise\nthe Gibbs Energy in Eq. 1which corresponds to \ufb01nding the\nMaximum a Posteriori (MAP) labelling of the corresponding probability distribution, P(V=v) =1\nZexp (\u2212E(v))\nwhereZis the normalisation factor. Mean \ufb01eld inference is\ndifferentiable, and this iterative algorithm can be unrolled\nand seen as a recurrent neural network [ 60]. Following this\napproach, we can incorporate mean \ufb01eld inference of a CRF\nas a layer of our neural network. This enables us to train our\nentire instance segmentation network end-to-end.\nBecause we deal with a variable number of instances for\nevery image, our CRF needs to be dynamically instantiated"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a6-6db4-11ee-90fe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-21", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "entire instance segmentation network end-to-end.\nBecause we deal with a variable number of instances for\nevery image, our CRF needs to be dynamically instantiated\nto have a different number of labels for every image, as observed in [ 3]. Therefore, unlike [ 60], none of our weights\nare class-speci\ufb01c. This weight-sharing not only allows us to\ndeal with variable length inputs, but class-speci\ufb01c weights\n(a) Original\nground truth, G(b) Prediction, P (c) \u201cMatched\u201d\nground truth, G\u2217\nFigure 6: Due to the problem of label permutations, we\n\u201cmatch\u201d the ground truth with our prediction before computing the loss when training.\nalso do not make sense in the case of instance segmentation\nsince a class label has no particular semantic meaning.\n3.4. Loss Function\nWhen training for instance segmentation, we have a\nsingle loss function which we backpropagate through our\ninstance- and semantic-segmentation modules to update all\nthe parameters. As discussed previously, we need to deal\nwith different permutations of our \ufb01nal labelling which\ncould have the same \ufb01nal result. The works of [ 57] and [ 58]\norder instances by depth to break this symmetry. However,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a7-6db4-11ee-8993-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-22", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "could have the same \ufb01nal result. The works of [ 57] and [ 58]\norder instances by depth to break this symmetry. However,\nthis requires ground-truth depth maps during training which\nwe do not assume that we have. Proposal-based methods\n[12,19,20,37] do not have this issue since they consider a\nsingle proposal at a time, rather than the entire image. Our\napproach is similar to [ 45] in that we match the original\nground truth to our instance segmentation prediction based\non the Intersection over Union (IoU) [ 14] of each instance\nprediction and ground truth, as shown in Fig. 6.\nMore formally, we denote the ground-truth labelling of\nan image, G, to be a set of rsegments, {g1,g2,...,g r},\nwhere each segment (set of pixels) is an object instance\nand has an associated semantic class label. Our prediction, which is the output of our network, P, is a set of s\nsegments, {p1,p2,...,p s}, also where each segment corresponds to an instance label and also has an associated class\nlabel. Note that randsmay be different since we may predict greater or fewer instances than actually present. Let\nMdenote the set of all permutations of the ground-truth,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a8-6db4-11ee-a798-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-23", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "label. Note that randsmay be different since we may predict greater or fewer instances than actually present. Let\nMdenote the set of all permutations of the ground-truth,\nG. As can be seen in Fig. 6, different permutations of the\nground-truth correspond to the same qualitative result. We\nde\ufb01ne the \u201cmatched\u201d ground-truth, G\u2217, as the permutation\nof the original ground-truth labelling which maximises the\nIoU between the prediction, P, and ground truth:\nG\u2217= arg max\nm\u2208MIoU(m,P). (7)\nOnce we have the \u201cmatched\u201d ground truth, G\u2217, (Fig. 6) foran image, we can apply any loss function to train our network for segmentation. In our case, we use the common\ncross-entropy loss function. We found that this performed\nbetter than the approximate IoU loss proposed in [ 27,45].\nCrucially, we do not need to evaluate all permutations\nof the ground truth to compute Eq. 7, since it can be formulated as a maximum-weight bipartite matching problem.\nThe edges in our bipartite graph connect ground-truth and\npredicted segments. The edge weights are given by the\nIoU between the ground truth and predicted segments if"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10a9-6db4-11ee-b762-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-24", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "The edges in our bipartite graph connect ground-truth and\npredicted segments. The edge weights are given by the\nIoU between the ground truth and predicted segments if\nthey share the same semantic class label, and zero otherwise. Leftover segments are matched to \u201cdummy\u201d nodes\nwith zero overlap.\nAdditionally, the ordering of the instances in our network\nare actually determined by the object detector, which remains static during training. As a result, the ordering of\nour predictions does not \ufb02uctuate much during training \u2013 it\nonly changes in cases where there are multiple detections\noverlapping an object.\n3.5. Network Training\nWe \ufb01rst train a network for semantic segmentation with\nthe standard cross-entropy loss. In our case, this network is\nFCN8s [ 38] with a CRF whose inference is unrolled as an\nRNN and trained end-to-end, as described in [ 60] and [ 2].\nTo this pretrained network, we append our instance segmentation subnetwork, and \ufb01netune with instance segmentation\nannotations and only the loss detailed in Sec. 3.4. For the\nsemantic segmentation subnetwork, we train with an initial\nlearning rate of 10\u22128, momentum of 0.9 and batch size of\n20. The learning rate is low since we do not normalise the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10aa-6db4-11ee-a4dc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-25", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "learning rate of 10\u22128, momentum of 0.9 and batch size of\n20. The learning rate is low since we do not normalise the\nloss by the number of pixels. This is so that images with\nmore pixels contribute a higher loss. The normalised learning rate is approximately 2\u00d710\u22123. When training our instance segmentation network as well, we lower the learning\nrate to 10\u221212and use a batch size of 1 instead. Decreasing the batch size gave empirically better results. We also\nclipped gradients (a technique common in training RNNs\n[40]) with\u21132norms above 109. This threshold was set by\nobserving \u201cnormal\u201d gradient magnitudes during training.\nThe relatively high magnitude is due to the fact that our\nloss is not normalised. In our complete network, we have\ntwo CRF inference modules which are RNNs (one each in\nthe semantic- and instance-segmentation subnetworks), and\ngradient clipping facilitated successful training.\n3.6. Discussion\nOur network is able to compute a semantic and instance\nsegmentation of the input image in a single forward pass.\nWe do not require any post-processing, such as the patch\naggregation of [ 37], \u201cmask-voting\u201d of [ 12], \u201csuperpixel"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10ab-6db4-11ee-b57b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-26", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "We do not require any post-processing, such as the patch\naggregation of [ 37], \u201cmask-voting\u201d of [ 12], \u201csuperpixel\nprojection\u201d of [ 19,20,30] or spectral clustering of [ 33].\nThe fact that we compute an initial semantic segmentation\nmeans that we have some robustness to errors in the object detector (Fig. 3). Furthermore, we are not necessarily\nlimited by poorly localised object detections either (Fig. 4).\nOur CRF model allows us to reason about the entire image\nat a time, rather than consider independent object proposals,\nas done in [ 19,20,12,37,30]. Although we do not train our\nobject detector jointly with the network, it also means that\nour segmentation network and object detector do not succumb to the same failure cases. Moreover, it ensures that\nour instance labelling does not \u201cswitch\u201d often during training, which makes learning more stable. Finally, note that\nalthough we perform mean \ufb01eld inference of a CRF within\nour network, we do not optimise the CRF\u2019s likelihood, but\nrather a cross-entropy loss (Sec 3.4).\n4. Experimental Evaluation\nSections 4.1to4.6describe our evaluation on the Pascal VOC Validation Set [ 14] and the Semantic Boundaries"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10ac-6db4-11ee-be42-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-27", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "4. Experimental Evaluation\nSections 4.1to4.6describe our evaluation on the Pascal VOC Validation Set [ 14] and the Semantic Boundaries\nDataset (SBD) [ 18] (which provides per-pixel annotations\nto 11355 previously unlaballed images from Pascal VOC).\nSection 4.7details results on Cityscapes [ 9].\n4.1. Experimental Details\nWe \ufb01rst train a network for semantic segmentation, therafter we \ufb01netune it to the task of instance segmentation,\nas described in Sec. 3.5. Our training data for the semantic segmentation pretraining consists of images from Pascal VOC [ 14], SBD [ 18] and Microsoft COCO [ 35]. Finally, when \ufb01netuning for instance segmentation, we use\nonly training data from either the VOC dataset, or from the\nSBD dataset. We train separate models for evaluating on\nthe VOC Validation Set, and the SBD Validation Set. In\neach case, we remove validation set images from the initial\nsemantic segmentation pretraining set. We use the publicly\navailable R-FCN object detection framework [ 13], and ensure that the images used to train the detector do not fall\ninto our test sets for instance segmentation.\n4.2. Evaluation Metrics\nWe report the mean Average Precision over regions"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10ad-6db4-11ee-bf3c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-28", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "into our test sets for instance segmentation.\n4.2. Evaluation Metrics\nWe report the mean Average Precision over regions\n(APr) as de\ufb01ned by [ 19]. The difference between APrand\nthe AP metric used in object detection [ 14] is that the Intersection over Union (IoU) is computed over predicted and\nground-truth regions instead of bounding boxes. Furthermore, the standard AP metric uses an IoU threshold of 0.5\nto determine whether a prediction is correct or not. Here,\nwe use a variety of IoU thresholds since larger thresholds\nrequire more precise segmentations. Additionally, we report theAPr\nvolwhich is the average of the APrfor 9 IoU\nthresholds ranging from 0.1 to 0.9 in increments of 0.1.\nHowever, we also observe that the APrmetric requires\nan algorithm to produce a ranked list of segments and their\nobject class. It does not require, nor evaluate, the ability ofTable 1: The effect of the different CRF unary potentials,\nand end-to-end training with them, on the VOC 2012 Validation Set.\nAPr\nAPr\nvolmatch\nIoU 0.5 0.7 0.9\nBox Term\n(piecewise)60.0 47.3 21.2 54.9 42.6\nBox+Global"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10ae-6db4-11ee-a952-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-29", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "IoU 0.5 0.7 0.9\nBox Term\n(piecewise)60.0 47.3 21.2 54.9 42.6\nBox+Global\n(piecewise)59.1 46.1 23.4 54.6 43.0\nBox+Global+Shape\n(piecewise)59.5 46.4 23.3 55.2 44.8\nBox Term\n(end-to-end)60.7 47.4 24.6 56.2 46.9\nBox+Global\n(end-to-end)60.9 48.1 25.5 56.7 47.1\nBox+Global+Shape\n(end-to-end)61.7 48.6 25.1 57.5 48.3\nan algorithm to produce a globally coherent segmentation\nmap of the image, for example Fig. 1c. To measure this, we\npropose the \u201cMatching IoU\u201d which matches the predicted\nimage and ground truth, and then calculates the corresponding IoU as de\ufb01ned in [ 14]. This matching procedure is the\nsame as described in Sec. 3.4. This measure was originally\nproposed in [ 54], but has not been used since in evaluating\ninstance segmentation systems.\n4.3. Effect of Instance Potentials and End-to-End\ntraining"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10af-6db4-11ee-84bf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-30", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "proposed in [ 54], but has not been used since in evaluating\ninstance segmentation systems.\n4.3. Effect of Instance Potentials and End-to-End\ntraining\nWe \ufb01rst perform ablation studies on the VOC 2012 Validation set. This dataset, consisting of 1464 training and\n1449 validation images has very high-quality annotations\nwith detailed object delineations which makes it the most\nsuited for evaluating pixel-level segmentations.\nIn Tab. 1, we examine the effect of each of our unary potentials in our Instance subnetwork on overall performance.\nFurthermore, we examine the effect of end-to-end training\nthe entire network as opposed to piecewise training. Piecewise training refers to freezing the pretrained semantic segmentation subnetwork\u2019s weights and only optimising the instance segmentation subnetwork\u2019s parameters. Note that\nwhen training with only the \u201cBox\u201d (Eq. 3) unary potential and pairwise term, we also have to add in an additional\n\u201cBackground\u201d detection which encompasses the entire image. Otherwise, we cannot classify the background label.\nWe can see that each unary potential improves overall instance segmentation results, both in terms of APr\nvoland the\nMatching IoU. The \u201cGlobal\u201d term (Eq. 4) shows particular"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b0-6db4-11ee-9840-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-31", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "voland the\nMatching IoU. The \u201cGlobal\u201d term (Eq. 4) shows particular\nimprovement over the \u201cBox\u201d term at the high APrthreshold of 0.9. This is because it can overcome errors in bounding box localisation (Fig. 4) and leverage our semantic segmentation network\u2019s accurate predictions to produce precise\nTable 2: Comparison of Instance Segmentation performance to recent methods on the VOC 2012 Validation Set\nMethodAPr\nAPr\nvol0.5 0.6 0.7 0.8 0.9\nSDS [ 19] 43.8 34.5 21.3 8.7 0.9 \u2013\nChen et al. [8] 46.3 38.2 27.0 13.5 2.6 \u2013\nPFN [ 33] 58.7 51.3 42.5 31.2 15.7 52.3\nArnab et al. [3] 58.3 52.4 45.4 34.9 20.1 53.1\nMPA 1-scale [ 37] 60.3 54.6 45.9 34.3 17.3 54.5\nMPA 3-scale [ 37]62.1 56.6 47.4 36.1 18.5 56.5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b1-6db4-11ee-9017-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-32", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "MPA 3-scale [ 37]62.1 56.6 47.4 36.1 18.5 56.5\nOurs 61.7 55.5 48.6 39.5 25.1 57.5\nlabellings. The \u201cShape\u201d term\u2019s improvement in the APr\nvolis\nprimarily due to an improvement in the APrat low thresholds. By using shape priors, we are able to recover instances\nwhich were occluded and missed out. End-to-end training\nalso improves results at all APrthresholds. Training with\njust the \u201cBox\u201d term shows a modest improvement in the\nAPr\nvolof 1.3%. Training with the \u201cGlobal\u201d and \u201cShape\u201d\nterms shows larger improvements of 2.1% and 2.3% respectively. This may be because the \u201cBox\u201d term only considers\nthe semantic segmentation at parts of the image covered by\nobject detections. Once we include the \u201cGlobal\u201d term, we\nconsider the semantic segmentation over the entire image\nfor the detected class. Training makes more ef\ufb01cient use of\nimages, and error gradients are more stable in this case.\n4.4. Results on VOC Validation Set\nWe then compare our best instance segmentation model"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b2-6db4-11ee-a31f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-33", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "images, and error gradients are more stable in this case.\n4.4. Results on VOC Validation Set\nWe then compare our best instance segmentation model\nto recent methods on the VOC Validation Set in Tab. 2. The\nfact that our algorithm achieves the highest APrat thresholds above 0.7 indicates that our method produces more detailed and accurate segmentations.\nAt an IoU threshold of 0.9, our improvement over the\nprevious state-of-the-art (MPA [ 37]) is 6.6%, which is a\nrelative improvement of 36%. Unlike [ 37,19,8], our\nnetwork performs an initial semantic segmentation which\nmay explain our more accurate segmentations. Other\nsegmentation-based approaches, [ 3,33] are not fully endto-end trained. We also achieve the best APr\nvolof 57.5%.\nThe relatively small difference in APr\nvolto MPA [ 37] despite large improvements at high IoU thresholds indicates\nthat MPA performs better at low IoU thresholds. Proposalbased methods, such as [ 37,19] are more likely to perform\nbetter at low IoU thresholds since they output more proposals than actual instances in an image (SDS evaluates 2000\nproposals per image). Furthermore, note that whilst MPA"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b3-6db4-11ee-8cd9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-34", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "better at low IoU thresholds since they output more proposals than actual instances in an image (SDS evaluates 2000\nproposals per image). Furthermore, note that whilst MPA\ntakes 8.7s to process an image [ 37], our method requires approximately 1.5s on the same Titan X GPU. More detailed\nqualitative and quantitative results, including success and\nfailure cases, are included in the supplementary material.Table 3: Comparison of Instance Segmentation performance on the SBD Dataset\nMethodAPr\nAPr\nvolmatch\n0.5 0.7 IoU\nSDS [ 19] 49.7 25.3 41.4 \u2013\nMPA 1-scale [ 37] 55.5 \u2013 48.3 \u2013\nHypercolumn [ 20] 56.5 37.0 \u2013 \u2013\nIIS [30] 60.1 38.7 \u2013 \u2013\nCFM [ 11] 60.7 39.6 \u2013 \u2013\nHypercolumn rescore [ 20] 60.0 40.4 \u2013 \u2013\nMPA 3-scale rescore [ 37] 61.8 \u2013 52.0 \u2013\nMNC [ 12] 63.5 41.5 \u2013 39.0\nMNC, Instance FCN [ 10] 61.5 43.0 \u2013 \u2013\nIIS sp. projection, rescore [ 30]63.6 43.3 \u2013 \u2013"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b4-6db4-11ee-baa9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-35", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "MNC, Instance FCN [ 10] 61.5 43.0 \u2013 \u2013\nIIS sp. projection, rescore [ 30]63.6 43.3 \u2013 \u2013\nOurs (piecewise) 59.1 42.1 52.3 41.8\nOurs (end-to-end) 62.0 44.8 55.4 47.3\n4.5. Results on SBD Dataset\nWe also evaluate our model on the SBD dataset, which\nconsists of 5623 training and 5732 validation images, as\nshown in Tab. 3. Following other works, we only report\nAPrresults at IoU thresholds of 0.5 and 0.7. However, we\nprovide more detailed results in our supplementary material. Once again, we show signi\ufb01cant improvements over\nother work at high APrthresholds. Here, our APrat 0.7\nimproves by 1.5% over the previous state-of-the-art [ 30].\nNote that [ 30,37,20] perform additional post-processing\nwhere their results are rescored using an additional object\ndetector. In contrast, our results are obtained by a single\nforward pass through our network. We have also improved\nsubstantially on the APr\nvolmeasure (3.4%) compared to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b5-6db4-11ee-8195-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-36", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "forward pass through our network. We have also improved\nsubstantially on the APr\nvolmeasure (3.4%) compared to\nother works which have reported it. We also used the publicly available source code1, model and default parameters\nof MNC [ 12] to evaluate the \u201cMatching IoU\u201d. Our method\nimproves this by 8.3%. This metric is a stricter measure of\nsegmentation performance, and our method, which is based\non an initial semantic segmentation and includes a CRF as\npart of training therefore performs better.\n4.6. Improvement in Semantic Segmentation\nFinetuning our network for instance segmentation, with\nthe loss described in Sec. 3.4improves semantic segmentation performance on both the VOC and SBD dataset, as\nshown in Tab. 4. The improvement is 0.9% on VOC, and\n1% on SBD. The tasks of instance segmentation and semantic segmentation are highly related \u2013 in fact, instance\nsegmentation can be thought of as a more speci\ufb01c case of\nsemantic segmentation. As a result, \ufb01netuning for one task\nimproves the other.\n1https://github.com/daijifeng001/MNC\nTable 4: Semantic Segmentation performance before and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b6-6db4-11ee-8052-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-37", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "improves the other.\n1https://github.com/daijifeng001/MNC\nTable 4: Semantic Segmentation performance before and\nafter \ufb01netuning for Instance Segmentation on the VOC and\nSBD Validation Sets\nDatasetMean IoU [%] before\nInstance \ufb01netuningMean IoU [%] after\nInstance \ufb01netuning\nVOC 74.2 75.1\nSBD 71.5 72.5\n4.7. Results on Cityscapes\nFinally, we evaluate our algorithm on the Cityscapes\nroad-scene understanding dataset [ 9]. This dataset consists\nof 2975 training images, and the held-out test set consisting\nof 1525 images are evaluated on an online server. None of\nthe 500 validation images were used for training. We use\nan initial semantic segmentation subnetwork that is based\non the ResNet-101 architecture [ 59], and all of the instance\nunary potentials described in Sec. 3.2.\nAs shown in Tab. 5, our method sets a new state-of-theart on Cityscapes, surpassing concurrent work [ 21] and the\nbest previous published work [ 49] by signi\ufb01cant margins.\nTable 5: Results on Cityscapes Test Set. Evaluation metrics and results of competing methods obtained from the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b7-6db4-11ee-ba8c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-38", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "best previous published work [ 49] by signi\ufb01cant margins.\nTable 5: Results on Cityscapes Test Set. Evaluation metrics and results of competing methods obtained from the\nonline server. The \u201cAP\u201d metric of Cityscapes is similar to\nourAPr\nvolmetric.\nMethod AP AP at 0.5 AP 100m AP 50m\nOurs 20.0 38.8 32.6 37.6\nSAIS [ 21] 17.4 36.7 29.3 34.0\nDWT [ 4] 15.6 30.0 26.2 31.8\nInstanceCut [ 24] 13.0 27.9 22.1 26.1\nGraph Decomp. [ 29] 9.8 23.2 16.8 20.3\nRecAttend [ 43] 9.5 18.9 16.8 20.9\nPixel Encoding [ 49] 8.9 21.1 15.3 16.7\nR-CNN [ 9] 4.6 12.9 7.7 10.3\n5. Conclusion and Future Work\nWe have presented an end-to-end instance segmentation\napproach that produces intermediate semantic segmentations, and shown that \ufb01netuning for instance segmentation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b8-6db4-11ee-8997-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-39", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "We have presented an end-to-end instance segmentation\napproach that produces intermediate semantic segmentations, and shown that \ufb01netuning for instance segmentation\nimproves our network\u2019s semantic segmentations. Our approach differs from other methods which derive their architectures from object detection networks [ 12,37,20] in that\nour approach is more similar to a semantic segmentation\nnetwork. As a result, our system produces more accurate\nand detailed segmentations as shown by our substantial improvements at high APrthresholds. Moreover, our system\nproduces segmentation maps naturally, and in contrast to\nother published work, does not require any post-processing.\nFinally, our network produces a variable number of outputs,depending on the number of instances in the image. Our future work is to incorporate an object detector into the endto-end training of our system to create a network that performs semantic segmentation, object detection and instance\nsegmentation jointly. Possible techniques for doing this are\nsuggested by [ 25] and [ 39].\nAcknowledgements We thank Bernardino RomeraParedes and Stuart Golodetz for insightful discussions\nand feedback. This work was supported by the EPSRC,\nClarendon Fund, ERC grant ERC-2012-AdG 321162HELIOS, EPRSRC grant Seebibyte EP/M013774/1 and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10b9-6db4-11ee-92f5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-40", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "EPSRC/MURI grant EP/N019474/1.\nReferences\n[1] P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik. Multiscale combinatorial grouping. In CVPR , pages\n328\u2013335. IEEE, 2014. 2\n[2] A. Arnab, S. Jayasumana, S. Zheng, and P. H. S. Torr. Higher\norder conditional random \ufb01elds in deep neural networks. In\nECCV , 2016. 2,3,6\n[3] A. Arnab and P. H. S. Torr. Bottom-up instance segmentation\nwith deep higher order crfs. In BMVC , 2016. 3,5,8,20\n[4] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In arXiv preprint arXiv:1611.08303 ,\n2016. 9\n[5] L. Chen, A. Schwing, A. Yuille, and R. Urtasun. Learning\ndeep structured models. In ICML , Lille, France, 2015. 2\n[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10ba-6db4-11ee-b968-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-41", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "[6] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and\nA. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR , 2015. 2\n[7] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and\nA. L. Yuille. Deeplab: Semantic image segmentation with\ndeep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915 , 2016. 1\n[8] Y.-T. Chen, X. Liu, and M.-H. Yang. Multi-instance object segmentation with occlusion handling. In CVPR , pages\n3470\u20133478, 2015. 2,4,5,8,20\n[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler,\nR. Benenson, U. Franke, S. Roth, and B. Schiele. The\ncityscapes dataset for semantic urban scene understanding.\nInCVPR , 2016. 1,7,9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10bb-6db4-11ee-8f8f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-42", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "cityscapes dataset for semantic urban scene understanding.\nInCVPR , 2016. 1,7,9\n[10] J. Dai, K. He, Y. Li, S. Ren, and J. Sun. Instance-sensitive\nfully convolutional networks. In ECCV , 2016. 1,8\n[11] J. Dai, K. He, and J. Sun. Convolutional feature masking for\njoint object and stuff segmentation. In CVPR , 2015. 2,8\n[12] J. Dai, K. He, and J. Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR , 2016. 1,2,\n3,4,6,7,8,9,11,14,15\n[13] J. Dai, Y. Li, K. He, and J. Sun. R-fcn: Object detection via\nregion-based fully convolutional networks. In NIPS , 2016.\n1,7\n[14] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and\nA. Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 1,6,7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10bc-6db4-11ee-93a3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-43", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "A. Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2010. 1,6,7\n[15] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained partbased models. PAMI , 32(9):1627\u20131645, 2010. 2,5\n[16] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic\nsegmentation. In CVPR , 2014. 2\n[17] S. Gupta, R. Girshick, P. Arbelaez, and J. Malik. Learning\nrich features from RGB-D images for object detection and\nsegmentation. In ECCV . 2014. 1\n[18] B. Hariharan, P. Arbel \u00b4aez, L. Bourdev, S. Maji, and J. Malik.\nSemantic contours from inverse detectors. In ICCV , pages\n991\u2013998. IEEE, 2011. 7\n[19] B. Hariharan, P. Arbel \u00b4aez, R. Girshick, and J. Malik. Simultaneous detection and segmentation. In ECCV , pages 297\u2013"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10bd-6db4-11ee-9592-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-44", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "312. Springer, 2014. 1,2,3,4,6,7,8,20\n[20] B. Hariharan, P. Arbel \u00b4aez, R. Girshick, and J. Malik. Hypercolumns for object segmentation and \ufb01ne-grained localization. CVPR , 2015. 1,2,3,4,6,7,8,9\n[21] Z. Hayder, X. He, and M. Salzmann. Shape-aware instance\nsegmentation. In arXiv preprint arXiv:1612.03129 , 2016. 9\n[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning\nfor image recognition. In CVPR , 2016. 1\n[23] X. He and S. Gould. An Exemplar-based CRF for Multiinstance Object Segmentation. In CVPR , 2014. 5\n[24] A. Kirillov, E. Levinkov, B. Andres, B. Savchynskyy, and\nC. Rother. Instancecut: from edges to instances with multicut. In arXiv preprint arXiv:1611.08272 , 2016. 9\n[25] I. Kokkinos. Ubernet: Training a universal convolutional"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10be-6db4-11ee-9a28-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-45", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "[25] I. Kokkinos. Ubernet: Training a universal convolutional\nneural network for low-, mid-, and high-level vision using\ndiverse datasets and limited memory. In CVPR , 2017. 9\n[26] P. Kr \u00a8ahenb \u00a8uhl and V. Koltun. Ef\ufb01cient inference in fully\nconnected CRFs with Gaussian edge potentials. In NIPS ,\n2011. 3,5\n[27] P. Kr \u00a8ahenb \u00a8uhl and V. Koltun. Parameter learning and convergent inference for dense random \ufb01elds. In ICML , 2013.\n6\n[28] L. Ladick \u00b4y, P. Sturgess, K. Alahari, C. Russell, and P. H. S.\nTorr. What, where and how many? combining object detectors and crfs. In ECCV , pages 424\u2013437, 2010. 3\n[29] E. Levinkov, S. Tang, E. Insafutdinov, and B. Andres. Joint\ngraph decomposition and node labeling by local search.\narXiv preprint arXiv:1611.04399 , 2016. 9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10bf-6db4-11ee-91b5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-46", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "graph decomposition and node labeling by local search.\narXiv preprint arXiv:1611.04399 , 2016. 9\n[30] K. Li, B. Hariharan, and J. Malik. Iterative Instance Segmentation. In CVPR , 2016. 1,2,4,6,7,8,21\n[31] Y. Li, H. Qi, J. Dai, X. Ji, and Y. Wei. Fully convolutional\ninstance-aware semantic segmentation. In CVPR , 2017. 11,\n16\n[32] X. Liang, Y. Wei, X. Shen, Z. Jie, J. Feng, L. Lin, and S. Yan.\nReversible recursive instance-level object segmentation. In\nCVPR , 2016. 1,2\n[33] X. Liang, Y. Wei, X. Shen, J. Yang, L. Lin, and S. Yan.\nProposal-free network for instance-level object segmentation. arXiv preprint arXiv:1509.02636 , 2015. 2,6,8,18,\n20\n[34] G. Lin, C. Shen, and I. Reid. Ef\ufb01cient piecewise training of\ndeep structured models for semantic segmentation. In CVPR ,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c0-6db4-11ee-b540-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-47", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "[34] G. Lin, C. Shen, and I. Reid. Ef\ufb01cient piecewise training of\ndeep structured models for semantic segmentation. In CVPR ,\n2016. 2[35] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll \u00b4ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740\u2013755. Springer,\n2014. 7,16\n[36] M.-Y. Liu, O. Tuzel, A. Veeraraghavan, and R. Chellappa.\nFast directional chamfer matching. In CVPR , pages 1696\u2013\n1703. IEEE, 2010. 5\n[37] S. Liu, X. Qi, J. Shi, H. Zhang, and J. Jia. Multi-scale patch\naggregation (mpa) for simultaneous detection and segmentation. In CVPR , 2016. 1,2,3,4,6,7,8,9,18,20\n[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional\nnetworks for semantic segmentation. In CVPR , 2015. 3,6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c1-6db4-11ee-b255-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-48", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "[38] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional\nnetworks for semantic segmentation. In CVPR , 2015. 3,6\n[39] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Crossstitch Networks for Multi-task Learning. In CVPR , 2016. 9\n[40] R. Pascanu, T. Mikolov, and Y. Bengio. On the dif\ufb01culty of\ntraining recurrent neural networks. In ICML , pages 1310\u2013\n1318, 2013. 6\n[41] P. O. Pinheiro, R. Collobert, and P. Dollar. Learning to segment object candidates. In NIPS , 2015. 1\n[42] P. O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollar. Learning to re\ufb01ne object segments. In ECCV , 2016. 1\n[43] M. Ren and R. S. Zemel. End-to-end instance segmentation and counting with recurrent attention. In arXiv preprint\narXiv:1605.09410 , 2016. 9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c2-6db4-11ee-a8e9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-49", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "arXiv:1605.09410 , 2016. 9\n[44] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS , 2015. 2\n[45] B. Romera-Paredes and P. H. Torr. Recurrent instance segmentation. In ECCV , 2016. 2,6\n[46] J. Shotton, A. Blake, and R. Cipolla. Contour-based learning\nfor object detection. In ICCV , volume 1, pages 503\u2013510.\nIEEE, 2005. 5\n[47] K. Simonyan and A. Zisserman. Very deep convolutional\nnetworks for large-scale image recognition. In ICLR , 2015.\n3\n[48] M. Sun, B.-s. Kim, P. Kohli, and S. Savarese. Relating things\nand stuff via object property interactions. PAMI , 36(7):1370\u2013\n1383, 2014. 3\n[49] J. Uhrig, M. Cordts, U. Franke, and T. Brox. Pixel-level encoding and depth layering for instance-level semantic labeling. In German Conference on Pattern Recognition (GCPR) ,\n2016. 9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c3-6db4-11ee-80f8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-50", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "2016. 9\n[50] D. Weiss and B. Taskar. Scalpel: Segmentation cascades\nwith localized priors and ef\ufb01cient learning. In CVPR , pages\n2035\u20132042, 2013. 5\n[51] J. Winn and J. Shotton. The layout consistent random \ufb01eld\nfor recognizing and segmenting partially occluded objects.\nInCVPR , 2006. 2\n[52] C. Wojek and B. Schiele. A dynamic conditional random\n\ufb01eld model for joint labeling of object and scene classes. In\nECCV , pages 733\u2013747. Springer, 2008. 3\n[53] N. Xu, B. Price, S. Cohen, J. Yang, and T. Huang. Deep\ninteractive object selection. In CVPR , 2016. 1\n[54] Y. Yang, S. Hallman, D. Ramanan, and C. C. Fowlkes. Layered object models for image segmentation. PAMI , 2012. 2,\n7\n[55] J. Yao, S. Fidler, and R. Urtasun. Describing the Scene as\na Whole: Joint Object Detection, Scene Classi\ufb01cation and\nSemantic Segmentation. In CVPR , pages 702\u2013709, 2012. 3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c4-6db4-11ee-b830-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-51", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "a Whole: Joint Object Detection, Scene Classi\ufb01cation and\nSemantic Segmentation. In CVPR , pages 702\u2013709, 2012. 3\n[56] S. Zagoruyko, A. Lerer, T.-Y. Lin, P. O. Pinheiro, S. Gross,\nS. Chintala, and P. Doll \u00b4ar. A multipath network for object\ndetection. In BMVC , 2016. 1\n[57] Z. Zhang, S. Fidler, and R. Urtasun. Instance-level segmentation for autonomous driving with deep densely connected\nmrfs. In CVPR , 2016. 2,3,4,6\n[58] Z. Zhang, A. G. Schwing, S. Fidler, and R. Urtasun. Monocular object instance segmentation and depth ordering with\ncnns. In ICCV , pages 2614\u20132622, 2015. 2,3,4,6\n[59] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene\nparsing network. In arXiv preprint arXiv:1612.01105 , 2017.\n9\n[60] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c5-6db4-11ee-9c64-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-52", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "9\n[60] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,\nZ. Su, D. Du, C. Huang, and P. Torr. Conditional random\n\ufb01elds as recurrent neural networks. In ICCV , 2015. 2,3,5,6\nAppendix\nIn this supplementary material, we include more detailed\nqualitative and quantitative results on the VOC and SBD\ndatasets. Furthermore, we also show the runtime of our algorithm.\nFigures 7and8show success and failure cases of our\nalgorithm. Figure 9compares the results of our algorithm\nto the publicly available model for MNC [ 12]. Figure 10\ncompares our results to those of FCIS [ 31], concurrent work\nwhich won the COCO 2016 challenge. Figure 11presents\nsome qualitative results on the Cityscapes dataset.\nSection Ashows more detailed results on the VOC\ndataset. Figure 12shows a visualisation of our results at\ndifferentAPrthresholds, and Tables 7to9show per-class\nAPrresults at thresholds of 0.5, 0.7 and 0.9.\nSection Bshows more detailed results on the SBD"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c6-6db4-11ee-bc2f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-53", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "APrresults at thresholds of 0.5, 0.7 and 0.9.\nSection Bshows more detailed results on the SBD\ndataset. Table 6shows our mean APrresults at thresholds from 0.5 to 0.9, whilst Tables 10and11show per-class\nAPrresults at thresholds of 0.7 and 0.5 respectively.\nInput image Semantic Segmentation Instance Segmentation Ground truth\nFigure 7: Success cases of our method. First and second row: Our algorithm can leverage good initial semantic segmentations, and detections, to produce an instance segmentation. Third row: Notice that we have ignored three false-positive\ndetections. Additionally, the red bounding box does not completely encompass the person, but our algorithm is still able to\nassociate pixels \u201coutside-the-box\u201d with the correct detection (also applies to row 2). Fourth row: Our system is able to deal\nwith the heavily occluded sheep, and ignore the false-positive detection. Fifth row: We have not been able to identify one\nbicycle on the left since it was not detected, but otherwise have performed well. Sixth row: Although subjective, the train has\nnot been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identi\ufb01ed it."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c7-6db4-11ee-a8e6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-54", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "not been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identi\ufb01ed it.\nNote that the \ufb01rst three images are from the VOC dataset, and the last three from SBD. Annotations in the VOC dataset are\nmore detailed, and also make more use of the grey \u201cignore\u201d label to indicate uncertain areas in the image. The \ufb01rst column\nshows the input image, and the results of our object detector which are another input to our network. Best viewed in colour.\nInput image Semantic Segmentation Instance Segmentation Ground truth\nFigure 8: Failure cases of our method. First row: Both our initial detector, and semantic segmentation system did not\nidentify a car in the background. Additionally, the \u201cbrown\u201d person prediction actually consists of two people that have been\nmerged together. This is because the detector did not \ufb01nd the background person. Second row: Our initial semantic segmentation identi\ufb01ed the table, but it is not there in the Instance Segmentation. This is because there was no \u201ctable detection\u201d to\nassociate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c8-6db4-11ee-bb1c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-55", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "associate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have\nnot done this in our work. Third row: A dif\ufb01cult case where we have segmented most of the people. However, sometimes\ntwo people instances are joined together as one person instance. This problem is because we do not have a detection for\neach person in the image. Fourth row: Due to our initial semantic segmentation, we have not been able to segment the green\nperson and table correctly. Fifth row: We have failed to segment a bird although it was detected. Sixth row: The occluding\ncows, which all appear similar, pose a challenge, even with our shape priors. The \ufb01rst column shows the input image, and\nthe results of our object detector which are another input to our network. Best viewed in colour.\nInput image MNC [ 12] Ours Ground truth\nFigure 9: Comparison to MNC [ 12]The above examples emphasise the advantages in our method over MNC [ 12]. Unlike\nproposal-based approaches such as MNC, our method can handle false-positive detections, poor bounding box localisation,\nreasons globally about the image and also produces more precise segmentations due to the initial semantic segmentation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10c9-6db4-11ee-ae60-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-56", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "reasons globally about the image and also produces more precise segmentations due to the initial semantic segmentation\nmodule which includes a differentiable CRF. Row 1 shows a case where MNC, which scores segment-based proposals, is\nfooled by a false-positive detection and segments an imaginary human (yellow segment). Our method is robust to falsepositive detections due to the initial semantic segmentation module which does not have the same failure modes as the\ndetector. Rows 2, 3 and 4 show how MNC [ 12] cannot deal with poorly localised bounding boxes. The horizontal boundaries\nof the red person in Row 2, and light-blue person in Row 4 correspond to the limits of the proposal processed by MNC. Our\nmethod, in contrast, can segment \u201coutside the detection bounding box\u201d due to the global instance unary potential (Eq. 4). As\nMNC does not reason globally about the image, it cannot handle cases of overlapping bounding boxes well, and produces\nmore instances than there actually are. The \ufb01rst column shows the input image, and the results of our object detector which\nare another input to our network. MNC does not use these detections, but does internally produce box-based proposals which\nare not shown. Best viewed in colour.\nInput image MNC [ 12] Ours Ground truth"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10ca-6db4-11ee-a2e0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-57", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "are not shown. Best viewed in colour.\nInput image MNC [ 12] Ours Ground truth\nFigure 9 continued: Comparison to MNC [ 12]The above examples show that our method produces more precise segmentations than MNC, that adhere to the boundaries of the objects. However, in Rows 3, 4 and 5, we see that MNC is able\nto segment instances that our method misses out. In Row 3 , our algorithm does not segment the baby, although there is a\ndetection for it. This suggests that our shape prior which was formulated to overcome such occlusions could be better. As\nMNC processes individual instances, it does not have a problem with dealing with small, occluding instances. In Row 4 ,\nMNC has again identi\ufb01ed a person that our algorithm could not. However, this is because we did not have a detection for\nthis person. In Row 5 , MNC has segmented the horses on the right better than our method. The \ufb01rst column shows the input\nimage, and the results of our object detector which are another input to our network. MNC does not use these detections, but\ndoes internally produce box-based proposals which are not shown. We used the publicly available code, models and default\nparameters of MNC to produce this \ufb01gure. Best viewed in colour.\nInput image FCIS [ 31] Ours"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10cb-6db4-11ee-80aa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-58", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "parameters of MNC to produce this \ufb01gure. Best viewed in colour.\nInput image FCIS [ 31] Ours\nFigure 10: Comparison to FCIS [ 31]The above images compare our method to the concurrent work, FCIS [ 31], which was\ntrained on COCO [ 35] and won the COCO 2016 challenge. Unlike proposal-based methods such as FCIS, our method can\nhandle false-positive detections and poor bounding-box localisation. Furthermore, as our method reasons globally about the\nimage, one pixel can only be assigned to a single instance, which is not the case with FCIS. Our method also produces more\nprecise segmentations, as it includes a differentiable CRF, and it is based off a semantic segmentation network. The results of\nFCIS are obtained from their publicly available results on the COCO test set ( https://github.com/daijifeng001/\nTA-FCN ). Note that FCIS is trained on COCO, and our model is trained on Pascal VOC which does not have as many classes\nas COCO, such as \u201cumbrella\u201d and \u201csuitcase\u201d among others. As a result, we are not able to detect these objects. The \ufb01rst"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10cc-6db4-11ee-b5d5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-59", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "column shows the input image, and the results of our object detector which are another input to our network. FCIS does not\nuse these detections, but does internally produce proposals which are not shown. Best viewed in colour.\nInput image Semantic Segmentation Instance Segmentation\nFigure 11: Sample results on the Cityscapes dataset The above images show how our method can handle the large numbers\nof instances present in the Cityscapes dataset. Unlike other recent approaches, our algorithm can deal with objects that are\nnot continuous \u2013 such as the car in the \ufb01rst row which is occluded by a pole. Best viewed in colour.\nA. Detailed results on the VOC dataset\nFigure 12shows a visualisation of the AProbtained by\nour method for each class across nine different thresholds.\nEach \u201ccolumn\u201d of Fig. 12corresponds to the APrfor each\nclass at a given IoU threshold. It is therefore an alternate\nrepresentation for the results tables (Tables 7to9). We can\nsee that our method struggles with classes such as \u201cbicycle\u201d, \u201cchair\u201d, \u201cdining table\u201d and \u201cpotted plant\u201d. This may\nbe explained by the fact that current semantic segmentation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1b10cd-6db4-11ee-aeb5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1704.02386-60", "title": "Pixelwise Instance Segmentation with a Dynamically Instantiated Network", "source": "http://arxiv.org/pdf/1704.02386", "start_index": 0, "chroma:document": "be explained by the fact that current semantic segmentation\nsystems (including ours) struggle with these classes. All recent methods on the Pascal VOC leaderboard2obtain an\nIoU for these classes which is lower than the mean IoU for\nall classes. In fact the semantic segmentation IoU for the\n\u201cchair\u201d class is less than half of the mean IoU for all the\nclasses for 16 out of the 20 most recent submissions on the\nVOC leaderboard at the time of writing.\nTables 7to9show per-class instance segmentation results on the VOC dataset, at IoU thresholds of 0.9, 0.7 and\n0.5 respectively. At an IoU threshold of 0.9, our method\nachieves the highest APrfor 16 of the 20 object classes.\nAt the threshold of 0.7, we achieve the highest APrin 15\nclasses. Finally, at an IoU threshold of 0.5, our method,\nMPA 3-scale [ 37] and PFN [ 33] each achieve the highest\nAPrfor 6 categories.\nB. Detailed results on the SBD dataset\nOnce again, we show a visualisation of the AProbtained\nby our method for each class across nine different thresholds (Fig. 13). The trend is quite similar to the VOC dataset\nin that our algorithm struggles on the same object classes"}


