# embeddings_queue

created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c226b-6db4-11ee-a828-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-17", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Downsampling is performed by applying 3x3 average pooling with stride 2 while the following upsampling (required\nfor the concatenation) is obtained via bilinear interpolation.\n4. Experiments\nIn the subsequent experiments, we test Attention Augmentation on standard computer vision architectures such\nas ResNets [14, 47, 13], and MnasNet [42] on the CIFAR100 [22], ImageNet [9] and COCO [25] datasets. Our experiments show that Attention Augmentation leads to systematic improvements on both image classi\ufb01cation and object detection tasks across a broad array of architectures and\ncomputational demands. We validate the utility of the proposed two-dimensional relative attention mechanism in ablation experiments. In all experiments, we substitute convolutional feature maps with self-attention feature maps as\nit makes for an easier comparison against the baseline models. Unless speci\ufb01ed otherwise, all results correspond to our\ntwo-dimensional relative self-attention mechanism. Experimental details can be found in the Appendix.\n4.1. CIFAR-100 image classi\ufb01cation\nWe \ufb01rst investigate how Attention Augmentation performs on CIFAR-100 [22], a standard benchmark for lowresolution imagery, using a Wide ResNet architecture [51].\nThe Wide-ResNet-28-10 architecture is comprised of 3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c226c-6db4-11ee-a6ef-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-18", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "The Wide-ResNet-28-10 architecture is comprised of 3\nstages of 4 residual blocks each using two 3\u00023convolutions. We augment the Wide-ResNet-28-10 by augmenting\nthe \ufb01rst convolution of all residual blocks with relative attention using Nh=8 heads and \u0014=2\u001d=0.2 and a minimum of\n20 dimensions per head for the keys. We compare Attention\nAugmentation (AA) against other forms of attention including Squeeze-and-Excitation (SE) [17] and the parameterfree formulation of Gather-Excite (GE) [16]. Table 1 shows\nthat Attention Augmentation improves performance both\nover the baseline network and Squeeze-and-Excitation at a\nsimilar parameter and complexity cost.\nArchitecture Params GFlops top-1 top-5\nWide-ResNet [51] 36.3M 10.4 80.3 95.0\nGE-Wide-ResNet [16] 36.3M 10.4 79.8 95.0\nSE-Wide-ResNet [17] 36.5M 10.4 81.0 95.3\nAA-Wide-ResNet (ours) 36.2M 10.9 81.6 95.2\nTable 1. Image classi\ufb01cation on the CIFAR-100 dataset [22] using"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c226d-6db4-11ee-90c5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-19", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Table 1. Image classi\ufb01cation on the CIFAR-100 dataset [22] using\nthe Wide-ResNet 28-10 architecture [51].\n4.2. ImageNet image classi\ufb01cation with ResNet\nWe next examine how Attention Augmentation performs\non ImageNet [9, 21], a standard large-scale dataset for high\nresolution imagery, across an array of architectures. We\nstart with the ResNet architecture [14, 47, 13] because of itswidespread use and its ability to easily scale across several\ncomputational budgets. The building block in ResNet-34\ncomprises two 3x3 convolutions with the same number of\noutput \ufb01lters. ResNet-50 and its larger counterparts use a\nbottleneck block comprising of 1x1, 3x3, 1x1 convolutions\nwhere the last pointwise convolution expands the number\nof \ufb01lters and the \ufb01rst one contracts the number of \ufb01lters.\nWe modify all ResNets by augmenting the 3x3 convolutions as this decreases number of parameters.2We apply\nAttention Augmentation in each residual block of the last 3\nstages of the architecture \u2013 when the spatial dimensions of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c226e-6db4-11ee-96a0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-20", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Attention Augmentation in each residual block of the last 3\nstages of the architecture \u2013 when the spatial dimensions of\nthe activation maps are 28x28, 14x14 and 7x7 \u2013 and downsample only during the \ufb01rst stage. All attention augmented\nnetworks use \u0014=2\u001d=0.2, except for ResNet-34 which uses\n\u0014=\u001d=0.25. The number of attention heads is \ufb01xed to Nh=8.\nArchitecture Params (M) \u0001Infer \u0001Train top-1\nResNet-50 25.6 - - 76.4\nSE [17] 28.1 +12% +92% 77.5 (77.0)\nBAM [31] 25.9 +19% +43% 77.3\nCBAM [46] 28.1 +56% +132% 77.4 (77.4)\nGALA [28] 29.4 +86% +133% 77.5 (77.3)\nAA (\u001d= 0:25) 24.3 +29% +25% 77.7\nTable 2. Image classi\ufb01cation performance of different attention\nmechanisms on the ImageNet dataset. \u0001refers to the increase\nin latency times compared to the ResNet50 on a single Tesla V100"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c226f-6db4-11ee-8f4e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-21", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "mechanisms on the ImageNet dataset. \u0001refers to the increase\nin latency times compared to the ResNet50 on a single Tesla V100\nGPU with Tensor\ufb02ow using a batch size of 128. For fair comparison, we also include top-1 results (in parentheses) when scaling\nnetworks in width to match \u001825:6M parameters as the ResNet50\nbaseline.\nArchitecture GFlops Params top-1 top-5\nResNet-34 [14] 7.4 21.8M 73.6 91.5\nSE-ResNet-34 [17] 7.4 22.0M 74.3 91.8\nAA-ResNet-34 (ours) 7.1 20.7M 74.7 92.0\nResNet-50 [14] 8.2 25.6M 76.4 93.1\nSE-ResNet-50 [17] 8.2 28.1M 77.5 93.7\nAA-ResNet-50 (ours) 8.3 25.8M 77.7 93.8\nResNet-101 [14] 15.6 44.5M 77.9 94.0\nSE-ResNet-101 [17] 15.6 49.3M 78.4 94.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2270-6db4-11ee-97b6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-22", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "SE-ResNet-101 [17] 15.6 49.3M 78.4 94.2\nAA-ResNet-101 (ours) 16.1 45.4M 78.7 94.4\nResNet-152 [14] 23.0 60.2M 78.4 94.2\nSE-ResNet-152 [17] 23.1 66.8M 78.9 94.5\nAA-ResNet-152 (ours) 23.8 61.6M 79.1 94.6\nTable 3. Image classi\ufb01cation on the ImageNet dataset [9] across\na range of ResNet architectures: ResNet-34, ResNet-50, Resnet101, and ResNet-152 [14, 47, 13].\nTable 2 benchmarks Attention Augmentation against\nchannel and spatial attention mechanisms BAM [31],\nCBAM [46] and GALA [28] with channel reduction ratio\u001b= 16 on the ResNet50 architecture. Despite the\n2We found that augmenting the pointwise expansions works just as well\nbut does not save parameters or computations.\nArchitecture GFlops Params top-1 top-5\nMnasNet-0.75 0.45 2.91M 73.3 91.3\nAA-MnasNet-0.75 0.51 3.02M 73.9 91.6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2271-6db4-11ee-be20-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-23", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "AA-MnasNet-0.75 0.51 3.02M 73.9 91.6\nMnasNet-1.0 0.63 3.89M 75.2 92.4\nAA-MnasNet-1.0 0.70 4.06M 75.7 92.6\nMnasNet-1.25 1.01 5.26M 76.7 93.2\nAA-MnasNet-1.25 1.11 5.53M 77.2 93.6\nMnasNet-1.4 1.17 6.10M 77.2 93.5\nAA-MnasNet-1.4 1.29 6.44M 77.7 93.8\nTable 4. Baseline and attention augmented MnasNet [42] accuracies with width multipliers 0.75, 1.0, 1.25 and 1.4.\nlack of specialized kernels (See Appendix A.3), Attention\nAugmentation offers a competitive accuracy/computational\ntrade-off compared to previously proposed attention mechanisms. Table 3 compares the non-augmented networks and\nSqueeze-and-Excitation (SE) [17] across different network\nscales. In all experiments, Attention Augmentation signi\ufb01cantly increases performance over the non-augmented\nbaseline and notably outperforms Squeeze-and-Excitation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2272-6db4-11ee-9339-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-24", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "baseline and notably outperforms Squeeze-and-Excitation\n(SE) [17] while being more parameter ef\ufb01cient (Figure 1).\nRemarkably, our AA-ResNet-50 performs comparably to\nthe baseline ResNet-101 and our AA-ResNet-101 outperforms the baseline ResNet-152. These results suggest that\nattention augmentation is preferable to simply making networks deeper. We include and discuss attention maps visualizations from different pixel positions in the appendix.\n4.3. ImageNet classi\ufb01cation with MnasNet\nIn this section, we inspect the use of Attention Augmentation in a resource constrained setting by conducting\nImageNet experiments with the MnasNet architecture [42],\nwhich is an extremely parameter-ef\ufb01cient architecture. In\nparticular, the MnasNet was found by neural architecture search [54], using only the highly optimized mobile inverted bottleneck block [36] and the Squeeze-andExcitation operation [17] as the primitives in its search\nspace. We apply Attention Augmentation to the mobile\ninverted bottleneck by replacing convolutional channels in\nthe expansion pointwise convolution using \u0014=2\u001d=0.1 and\nNh=4 heads. Our augmented MnasNets use augmented inverted bottlenecks in the last 13 blocks out of 18 in the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2273-6db4-11ee-b72d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-25", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Nh=4 heads. Our augmented MnasNets use augmented inverted bottlenecks in the last 13 blocks out of 18 in the\nMnasNet architecture, starting when the spatial dimension\nis 28x28. We downsample only in the \ufb01rst stage where Attention Augmentation is applied. We leave the \ufb01nal pointwise convolution, also referred to as the \u201chead\u201d, unchanged.\nIn Table 4, we report ImageNet accuracies for the baseline MnasNet and its attention augmented variants at different width multipliers. Our experiments show that Attention Augmentation yields accuracy improvements across\nall width multipliers. Augmenting MnasNets with relative\nself-attention incurs a slight parameter increase, however\n2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0\nnumber of parameters (millions)737475767778top-1 accuracy\n73.3%75.2%76.7%77.2%\n73.9%75.7%77.2%77.7%\nMnasNet\nAA-MnasNet (ours)Figure 3. ImageNet top-1 accuracy as a function of number of parameters for MnasNet (black) and Attention-Augmented-MnasNet"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2274-6db4-11ee-9af3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-26", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "AA-MnasNet (ours)Figure 3. ImageNet top-1 accuracy as a function of number of parameters for MnasNet (black) and Attention-Augmented-MnasNet\n(red) with width multipliers 0:75,1:0,1:25and1:4.\nwe verify in Figure 3 that the accuracy improvements are\nnot just explained by the parameter increase. Additionally,\nwe note that the MnasNet architecture employs Squeezeand-Excitation at multiple locations that were optimally selected via architecture search, further suggesting the bene\ufb01ts of our method.\n4.4. Object Detection with COCO dataset\nWe next investigate the use of Attention Augmentation\non the task of object detection on the COCO dataset [27].\nWe employ the RetinaNet architecture with a ResNet-50\nand ResNet-101 backbone as done in [26], using the opensourced RetinaNet codebase.3We apply Attention Augmentation uniquely on the ResNet backbone, modifying\nthem similarly as in our ImageNet classi\ufb01cation experiments.\nOur relative self-attention mechanism improves the performance of the RetinaNet on both ResNet-50 and ResNet101 as shown in Table 5. Most notably, Attention Augmentation yields a 1.4% mAP improvement over a strong\nRetinaNet baseline from [26]. In contrast to the success"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2275-6db4-11ee-8c6a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-27", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "RetinaNet baseline from [26]. In contrast to the success\nof Squeeze-and-Excitation in image classi\ufb01cation with ImageNet, our experiments show that adding Squeeze-andExcitation operators in the backbone network of the RetinaNet signi\ufb01cantly hurts performance, in spite of grid\nsearching over the squeeze ratio \u001b2f4;8;16g. We hypothesize that localization requires precise spatial information which SE discards during the spatial pooling operation,\nthereby negatively affecting performance. Self-attention on\nthe other hand maintains spatial information and is likely to\n3https://github.com/tensorflow/tpu/tree/master/\nmodels/official/retinanet\nBackbone architecture GFlops Params mAP COCO mAP 50 mAP 75\nResNet-50 [26] 182 33.4M 36.8 54.5 39.5\nSE-ResNet-50 [17] 183 35.9M 36.5 54.0 39.1\nAA-ResNet-50 (ours) 182 33.1M 38.2 56.5 40.7\nResNet-101 [26] 243 52.4M 38.5 56.4 41.2\nSE-ResNet-101 [17] 243 57.2M 37.4 55.0 39.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2276-6db4-11ee-9554-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-28", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "SE-ResNet-101 [17] 243 57.2M 37.4 55.0 39.9\nAA-ResNet-101 (ours) 245 51.7M 39.2 57.8 41.9\nTable 5. Object detection on the COCO dataset [27] using the RetinaNet architecture [26] with different backbone architectures. We report\nmean Average Precision at three different IoU values.\nbe able to identify object boundaries successfully. Visualizations of attention maps (See Figures 9 and 10 in the Appendix) reveal that some heads are indeed delineating objects from their background which might be important for\nlocalization.\n4.5. Ablation Study\nFully-attentional vision models: In this section, we investigate the performance of Attention Augmentation as a\nfunction of the fraction of attentional channels. As we increase this fraction to 100%, we begin to replace a ConvNet with a fully attentional model, only leaving pointwise\nconvolutions and the stem unchanged. Table 6 presents the\nperformance of Attention Augmentation on the ResNet-50\narchitecture for varying ratios \u0014=\u001d2f0:25;0:5;0:75;1:0g.\nPerformance slightly degrades as the ratio of attentional\nchannels increases, which we hypothesize is partly explained by the average pooling operation for downsampling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2277-6db4-11ee-bf2a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-29", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Performance slightly degrades as the ratio of attentional\nchannels increases, which we hypothesize is partly explained by the average pooling operation for downsampling\nat the \ufb01rst stage where Attention Augmentation is applied.\nAttention Augmentation proves however quite robust to the\nfraction of attentional channels. For instance, AA-ResNet50 with\u0014=\u001d=0.75 outperforms its ResNet-50 counterpart,\nwhile being more parameter and \ufb02ops ef\ufb01cient, indicating\nthat mostly employing attentional channels is readily competitive.\nPerhaps surprisingly, these experiments also reveal that\nour proposed self-attention mechanism is a powerful standalone computational primitive for image classi\ufb01cation and\nthat fully attentional models are viable for discriminative visual tasks. In particular, AA-ResNet-50 with \u0014=\u001d=1, which\nuses exclusively attentional channels, is only 2.5% worse\nin accuracy than its fully convolutional counterpart, in spite\nof downsampling with average pooling and having 25% less\nparameters. Notably, this fully attentional architecture4also\noutperforms ResNet-34 while being more parameter and\n\ufb02ops ef\ufb01cient (see Table 6).\n4We consider pointwise convolutions as dense layers. This architecture"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2278-6db4-11ee-96e4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-30", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "\ufb02ops ef\ufb01cient (see Table 6).\n4We consider pointwise convolutions as dense layers. This architecture\nemploys 4 non-pointwise convolutions in the stem and the \ufb01rst stage of the\narchitecture, but we believe such operations can be replaced by attention\ntoo.Architecture GFlops Params top-1 top-5\nResNet-34 [14] 7.4 21.8M 73.6 91.5\nResNet-50 [14] 8.2 25.6M 76.4 93.1\n\u0014=\u001d= 0:25 7.9 24.3M 77.7 93.8\n\u0014=\u001d= 0:5 7.3 22.3M 77.3 93.6\n\u0014=\u001d= 0:75 6.8 20.7M 76.7 93.2\n\u0014=\u001d= 1:0 6.3 19.4M 73.9 91.5\nTable 6. Attention Augmented ResNet-50 with varying ratios of\nattentional channels.\n0 25 50 75 100\nfraction of attention versus convolutional channels707274767880top-1 accuracy76.4%77.5%\n76.6%\n75.4%\n71.1%77.7%\n77.3%\n76.7%\n73.9%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2279-6db4-11ee-8759-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-31", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "76.6%\n75.4%\n71.1%77.7%\n77.3%\n76.7%\n73.9%\nno position\nwith position\nFigure 4. Effect of relative position embeddings as the ratio\nof attentional channels increases on our Attention-Augmented\nResNet50.\nImportance of position encodings: In Figure 4, we show\nthe effect of our proposed two-dimensional relative position encodings as a function of the fraction of attentional\nchannels. As expected, experiments demonstrate that our\nrelative position encodings become increasingly more important as the architecture employs more attentional channels. In particular, the fully self-attentional ResNet-50 gains\n2.8% top-1 ImageNet accuracy when using relative position\nencodings, which indicates the necessity of maintaining position information for fully self-attentional vision models.\nWe additionally compare our proposed two-dimensional\nArchitecture Position Encodings top-1 top-5\nAA-ResNet-34 None 74.4 91.9\nAA-ResNet-34 2d Sine 74.4 92.0\nAA-ResNet-34 CoordConv 74.4 92.0\nAA-ResNet-34 Relative (ours) 74.7 92.0\nAA-ResNet-50 None 77.5 93.7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c227a-6db4-11ee-92c9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-32", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "AA-ResNet-34 Relative (ours) 74.7 92.0\nAA-ResNet-50 None 77.5 93.7\nAA-ResNet-50 2d Sine 77.5 93.7\nAA-ResNet-50 CoordConv 77.5 93.8\nAA-ResNet-50 Relative (ours) 77.7 93.8\nTable 7. Effects of different position encodings in Attention Augmentation on ImageNet classi\ufb01cation.\nPosition Encodings mAP COCO mAP 50 mAP 75\nNone 37.7 56.0 40.2\nCoordConv [29] 37.4 55.5 40.1\nRelative (ours) 38.2 56.5 40.7\nTable 8. Effects of different position encodings in Attention Augmentation on the COCO object detection task using a RetinaNet\nAA-ResNet-50 backbone.\nrelative position encodings to other position encoding\nschemes. We apply Attention Augmentation using the same\nhyperparameters as 4.2 with the following different position encoding schemes: 1) The position-unaware version of\nself-attention (referred to as None ), 2) a two-dimensional\nimplementation of the sinusoidal positional waves (referred"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c227b-6db4-11ee-b627-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-33", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "self-attention (referred to as None ), 2) a two-dimensional\nimplementation of the sinusoidal positional waves (referred\nto as 2d Sine ) as used in [32], 3) CoordConv [29] for which\nwe concatenate (x,y,r) coordinate channels to the inputs of\nthe attention function, and 4) our proposed two-dimensional\nrelative position encodings (referred to as Relative ).\nIn Table 7 and 8, we present the results on ImageNet\nclassi\ufb01cation and the COCO object detection task respectively. On both tasks, Attention Augmentation without position encodings already yields improvements over the fully\nconvolutional non-augmented variants. Our experiments\nalso reveal that the sinusoidal encodings and the coordinate\nconvolution do not provide improvements over the positionunaware version of Attention Augmentation. We obtain additional improvements when using our two-dimensional relative attention, demonstrating the utility of preserving translation equivariance while preventing permutation equivariance.\n5. Discussion and future work\nIn this work, we consider the use of self-attention for vision models as an alternative to convolutions. We introduce\na novel two-dimensional relative self-attention mechanism\nfor images that enables training of competitive fully selfattentional vision models on image classi\ufb01cation for the \ufb01rst"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c227c-6db4-11ee-b99d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-34", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "for images that enables training of competitive fully selfattentional vision models on image classi\ufb01cation for the \ufb01rst\ntime. We propose to augment convolutional operators with\nthis self-attention mechanism and validate the superiority ofthis approach over other attention schemes. Extensive experiments show that Attention Augmentation leads to systematic improvements on both image classi\ufb01cation and object detection tasks across a wide range of architectures and\ncomputational settings.\nSeveral open questions from this work remain. In future work, we will focus on the fully attentional regime\nand explore how different attention mechanisms trade off\ncomputational ef\ufb01ciency versus representational power. For\ninstance, identifying a local attention mechanism may result in an ef\ufb01cient and scalable computational mechanism\nthat could prevent the need for downsampling with average\npooling [34]. Additionally, it is plausible that architectural\ndesign choices that are well suited when exclusively relying\non convolutions are suboptimal when using self-attention\nmechanisms. As such, it would be interesting to see if using Attention Augmentation as a primitive in automated architecture search procedures proves useful to \ufb01nd even better models than those previously found in image classi\ufb01cation [55], object detection [12], image segmentation [6] and\nother domains [5, 1, 35, 8]. Finally, one can ask to which"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c227d-6db4-11ee-972d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-35", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "other domains [5, 1, 35, 8]. Finally, one can ask to which\ndegree fully attentional models can replace convolutional\nnetworks for visual tasks.\nAcknowledgements\nThe authors would like to thank Tsung-Yi Lin, Prajit Ramachandran, Mingxing Tan, Yanping Huang and the\nGoogle Brain team for insightful comments and discussions.\nReferences\n[1] Maximilian Alber, Irwan Bello, Barret Zoph, Pieter-Jan Kindermans, Prajit Ramachandran, and Quoc V . Le. Backprop\nevolution. CoRR , abs/1808.02822, 2018. 8\n[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.\nNeural machine translation by jointly learning to align and\ntranslate. In International Conference on Learning Representations , 2015. 2\n[3] Irwan Bello, Sayali Kulkarni, Sagar Jain, Craig Boutilier,\nEd Huai-hsin Chi, Elad Eban, Xiyang Luo, Alan Mackey,\nand Ofer Meshi. Seq2slate: Re-ranking and slate optimization with rnns. CoRR , abs/1810.02019, 2018. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c227e-6db4-11ee-b6f6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-36", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "and Ofer Meshi. Seq2slate: Re-ranking and slate optimization with rnns. CoRR , abs/1810.02019, 2018. 2\n[4] Irwan Bello, Hieu Pham, Quoc V . Le, Mohammad Norouzi,\nand Samy Bengio. Neural combinatorial optimization with\nreinforcement learning. 2016. 2\n[5] Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V . Le.\nNeural optimizer search with reinforcement learning. In Proceedings of the 34th International Conference on Machine\nLearning - Volume 70 , ICML\u201917, pages 459\u2013468. JMLR.org,\n2017. 8\n[6] Liang-Chieh Chen, Maxwell Collins, Yukun Zhu, George\nPapandreou, Barret Zoph, Florian Schroff, Hartwig Adam,\nand Jon Shlens. Searching for ef\ufb01cient multi-scale architectures for dense image prediction. In Advances in Neural\nInformation Processing Systems , pages 8713\u20138724, 2018. 2,\n8\n[7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng\nYan, and Jiashi Feng. A2-nets: Double attention networks."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c227f-6db4-11ee-b790-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-37", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "[7] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng\nYan, and Jiashi Feng. A2-nets: Double attention networks.\nCoRR , abs/1810.11579, 2018. 3\n[8] Ekin Dogus Cubuk, Barret Zoph, Dandelion Man \u00b4e, Vijay Vasudevan, and Quoc V . Le. Autoaugment: Learning augmentation policies from data. CoRR , abs/1805.09501, 2018. 8\n[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,\nand Li Fei-Fei. Imagenet: A large-scale hierarchical image\ndatabase. In IEEE Conference on Computer Vision and Pattern Recognition . IEEE, 2009. 1, 2, 5\n[10] Xavier Gastaldi. Shake-shake regularization. arXiv preprint\narXiv:1705.07485 , 2017. 11\n[11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Dropblock:\nA regularization method for convolutional networks. In\nAdvances in Neural Information Processing Systems , pages\n10750\u201310760, 2018. 11"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2280-6db4-11ee-ad4d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-38", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "A regularization method for convolutional networks. In\nAdvances in Neural Information Processing Systems , pages\n10750\u201310760, 2018. 11\n[12] Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, and Quoc V\nLe. NAS-FPN: Learning scalable feature pyramid architecture for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 8\n[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\nDeep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.\n2, 5\n[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\nIdentity mappings in deep residual networks. In European\nConference on Computer Vision , 2016. 1, 2, 5, 7, 11\n[15] Sepp Hochreiter and Juergen Schmidhuber. Long short-term\nmemory. Neural Computation , 1997. 2\n[16] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea\nVedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In Advances in Neural Information\nProcessing Systems , pages 9423\u20139433, 2018. 3, 4, 5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2281-6db4-11ee-ad90-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-39", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Processing Systems , pages 9423\u20139433, 2018. 3, 4, 5\n[17] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer\nVision and Pattern Recognition , 2018. 1, 2, 3, 4, 5, 6, 7\n[18] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M Dai,\nMatthew D Hoffman, and Douglas Eck. Music transformer.\nInAdvances in Neural Processing Systems , 2018. 3, 4\n[19] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu,\nAnoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In\nIEEE Conference on Computer Vision and Pattern Recognition, 2017. 2\n[20] Sergey Ioffe and Christian Szegedy. Batch normalization:\nAccelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations , 2015. 4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c510f-6db4-11ee-b72f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-40", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations , 2015. 4\n[21] Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the\nIEEE Conference on Computer Vision and Pattern Recognition, 2019. 2, 5[22] Alex Krizhevsky. Learning multiple layers of features from\ntiny images. Technical report, University of Toronto, 2009.\n2, 5\n[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.\nImagenet classi\ufb01cation with deep convolutional neural networks. In Advances in Neural Information Processing System, 2012. 1, 2\n[24] Yann LeCun, L \u00b4eon Bottou, Yoshua Bengio, and Patrick\nHaffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998. 1\n[25] Tsung-Yi Lin, Piotr Doll \u00b4ar, Ross Girshick, Kaiming He,\nBharath Hariharan, and Serge Belongie. Feature pyramid\nnetworks for object detection. In Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition ,\n2017. 5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5110-6db4-11ee-bbe6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-41", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "networks for object detection. In Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition ,\n2017. 5\n[26] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and\nPiotr Doll \u00b4ar. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer\nvision , pages 2980\u20132988, 2017. 6, 7, 11\n[27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,\nPietro Perona, Deva Ramanan, Piotr Doll \u00b4ar, and C Lawrence\nZitnick. Microsoft coco: Common objects in context. In\nEuropean Conference on Computer Vision , pages 740\u2013755.\nSpringer, 2014. 2, 6, 7\n[28] Drew Linsley, Dan Scheibler, Sven Eberhardt, and Thomas\nSerre. Global-and-local attention networks for visual recognition. CoRR , abs/1805.08819, 2018. 5\n[29] Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski\nSuch, Eric Frank, Alex Sergeev, and Jason Yosinski. An\nintriguing failing of convolutional neural networks and the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5111-6db4-11ee-85eb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-42", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An\nintriguing failing of convolutional neural networks and the\ncoordconv solution. In Advances in Neural Information Processing Systems , pages 9628\u20139639, 2018. 3, 8\n[30] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint\narXiv:1608.03983 , 2016. 11\n[31] Jongchan Park, Sanghyun Woo, Joon-Young Lee, and In So\nKweon. Bam: bottleneck attention module. In British Machine Vision Conference , 2018. 3, 4, 5\n[32] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, \u0141ukasz\nKaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine\nLearning , 2018. 3, 8\n[33] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos,\nEric Wiewiora, and Serge Belongie. Objects in context.\n2007. 1\n[34] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan\nBello, Anselm Levskaya, and Jonathon Shlens. Stand-alone"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5112-6db4-11ee-9264-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-43", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone\nself-attention in vision models. CoRR , abs/1906.05909,\n2019. 8\n[35] Prajit Ramachandran, Barret Zoph, and Quoc V . Le. Searching for activation functions. CoRR , abs/1710.05941, 2017.\n8\n[36] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted\nresiduals and linear bottlenecks. In Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition ,\npages 4510\u20134520, 2018. 6\n[37] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Selfattention with relative position representations. arXiv\npreprint arXiv:1803.02155 , 2018. 2, 3, 4\n[38] David R. So, Chen Liang, and Quoc V . Le. The evolved\ntransformer. CoRR , abs/1901.11117, 2019. 3\n[39] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and\nAlex Alemi. Inception-v4, Inception-Resnet and the impact"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5113-6db4-11ee-9ed3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-44", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "[39] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and\nAlex Alemi. Inception-v4, Inception-Resnet and the impact\nof residual connections on learning. In International Conference on Learning Representations Workshop Track , 2016.\n2\n[40] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,\nScott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent\nVanhoucke, and Andrew Rabinovich. Going deeper with\nconvolutions. In IEEE Conference on Computer Vision and\nPattern Recognition , 2015. 2\n[41] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon\nShlens, and Zbigniew Wojna. Rethinking the Inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition , 2016. 2, 11\n[42] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan,\nand Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. 2,\n5, 6, 11"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5114-6db4-11ee-9bfa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-45", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "5, 6, 11\n[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia\nPolosukhin. Attention is all you need. In Advances in Neural\nInformation Processing Systems , pages 5998\u20136008, 2017. 1,\n2, 3\n[44] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer\nnetworks. In NIPS , pages 2692\u20132700, 2015. 2\n[45] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the\nIEEE Conference on Computer Vision and Pattern Recognition, pages 7794\u20137803, 2018. 3\n[46] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In\nSo Kweon. Cbam: Convolutional block attention module.\nInProceedings of the European Conference on Computer Vision (ECCV) , pages 3\u201319, 2018. 3, 4, 5\n[47] Saining Xie, Ross Girshick, Piotr Doll \u00b4ar, Zhuowen Tu, and\nKaiming He. Aggregated residual transformations for deep"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5115-6db4-11ee-ba2d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-46", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "[47] Saining Xie, Ross Girshick, Piotr Doll \u00b4ar, Zhuowen Tu, and\nKaiming He. Aggregated residual transformations for deep\nneural networks. In Proceedings of the IEEE Conference on\nComputer Vision and Pattern Recognition , 2017. 2, 5\n[48] Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and\nKoichi Kise. Shakedrop regularization for deep residual\nlearning. arXiv preprint arXiv:1802.02375 , 2018. 11\n[49] Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S.\nChao, and Zhaopeng Tu. Convolutional self-attention network. In CoRR , volume abs/1810.13320, 2018. 2\n[50] Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui\nZhao, Kai Chen, Mohammad Norouzi, and Quoc V Le.\nQAnet: Combining local convolution with global selfattention for reading comprehension. In International Conference on Learning Representations , 2018. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5116-6db4-11ee-ab6a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-47", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "QAnet: Combining local convolution with global selfattention for reading comprehension. In International Conference on Learning Representations , 2018. 2\n[51] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference , 2016. 5[52] Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David\nReichert, Timothy Lillicrap, Edward Lockhart, Murray\nShanahan, Victoria Langston, Razvan Pascanu, Matthew\nBotvinick, Oriol Vinyals, and Peter Battaglia. Deep reinforcement learning with relational inductive biases. In ICLR ,\n2019. 2\n[53] Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and\nAugustus Odena. Self-attention generative adversarial networks. arXiv:1805.08318 , 2018. 3\n[54] Barret Zoph and Quoc V . Le. Neural architecture search\nwith reinforcement learning. In International Conference on\nLearning Representations , 2017. 6\n[55] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V\nLe. Learning transferable architectures for scalable image"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5117-6db4-11ee-8285-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-48", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "[55] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V\nLe. Learning transferable architectures for scalable image\nrecognition. In Proceedings of the IEEE conference on\ncomputer vision and pattern recognition , pages 8697\u20138710,\n2018. 2, 8, 11\nA. Appendix\nA.1. Experimental details\nTuning Unless speci\ufb01ed otherwise, we use the default hyperparameters found in reference baseline codebases without tuning.\u0014was searched inf0.1, 0.2, 0.5g,\u001dinf0.0, 0.1,\n0.25, 0.5, 0.75, 1.0 gand the number of heads was chosen\nbased on memory constraints (starting from 8 and decreasing when necessary). We report the \ufb01nal accuracy for each\nrun without performing early stopping.\nCIFAR-100 Given the low resolution of CIFAR-100 images, we do not downsample feature maps before the attention operation and instead resort to a smaller batch size.\nWe train all networks for 500 epochs using synchronous\nSGD with momentum 0.9 distributed across 8 TESLA V100\nGPUs. The learning rate is linearly scaled from 0 to\n0:2B=256, whereBis the total batch size, for the \ufb01rst 5%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5118-6db4-11ee-9524-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-49", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "GPUs. The learning rate is linearly scaled from 0 to\n0:2B=256, whereBis the total batch size, for the \ufb01rst 5%\ntraining epochs and then annealed with cosine decay [30].\nWe use standard CIFAR preprocessing: mean normalizing,\nrandom \ufb02ipping and cropping [55, 10, 48]. Non-augmented\narchitectures are trained with a batch size of 1024 and a\nweight decay of 2e-4. Augmented architectures are trained\nwith batch size of 256 and a weight decay of 5e-4.\nImageNet classi\ufb01cation with ResNet We train all\nResNet architectures for 100 epochs using synchronous\nSGD with momentum 0.9 across 8 TESLA V100 GPUs and\nweight decay of 1e-4. We use the largest batch size per\nworkerB2f32;64;128;256gthat \ufb01ts in a minibatch. The\ninitial learning rate is scaled linearly according to the total\nbatch size using a base learning rate of 0.128 for total batch\nsize of 256. During training, we linearly scale the learning\nrate from 0 to this value for the \ufb01rst 5% of training epochs\nand divide it by 10 at epochs 30, 60, 80 and 90. We use"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5119-6db4-11ee-851a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-50", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "rate from 0 to this value for the \ufb01rst 5% of training epochs\nand divide it by 10 at epochs 30, 60, 80 and 90. We use\nstandard Inception data augmentation as described in [41].\nImageNet classi\ufb01cation with MnasNet We follow the\ntraining setup described in [42] and train all networks for\n350 epochs with the RMSProp optimizer using exponential\nlearning rate decay. When training our augmented MnasNets, we divide the learning rate by 2 and adjusted the learning rate decay so that the \ufb01nal learning rate stays the same.\nObject Detection with COCO dataset We follow the\nsetup described in [26, 11] and train the RetinaNet from\nscratch for 150 epochs without using ImageNet pretraining\nfor the ResNet backbone.\nA.2. Computational & Memory costs\nTable 9 provides the breakdown of self-attention related computational costs per image. Storing attention\nmaps in each layer induces a memory cost of Nh(HW)2bfloat16 . At inference, the memory cost for storing attention maps is only 1.2% of the memory required to store\nmodel parameters (49MB).\nLayer Memory Params FLOPS\nfStage 2 - H=W=14g* 4 600KB 43k 22M"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c511a-6db4-11ee-ada7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-51", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "model parameters (49MB).\nLayer Memory Params FLOPS\nfStage 2 - H=W=14g* 4 600KB 43k 22M\nfStage 3 - H=W=14g* 6 600KB 90k 40M\nfStage 4 - H=W=7g* 3 37.5KB 190k 19M\nTraining 6MB (total) 1.3M 390M\nInference 600KB (max) 1.3M 390M\nTable 9. Computational costs associated with self-attention in the\nforward pass of the ResNet50. During inference, we only consider\nthe largest memory cost since activations are not stored.\nFigures 5 and 6 show the accuracies of our attention\naugmented networks across FLOPS counts, which correlate\nwith running times across hardware platforms.\n5 10 15 20 25\ncomputational demand (GFlops).72737475767778798081top-1 accuracy.\n73.6%76.4%77.9%78.4%\n74.3%77.5%78.4%78.9%\n74.7%77.7%78.7%79.1%\nResNet\nSE-ResNet\nAA-ResNet (ours)\nFigure 5. ImageNet top-1 accuracy as a function of computational"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c511b-6db4-11ee-b513-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-52", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "ResNet\nSE-ResNet\nAA-ResNet (ours)\nFigure 5. ImageNet top-1 accuracy as a function of computational\ndemand for variety of ResNet architectures [14]. From left to right:\nResNet-34, ResNet-50, ResNet-101 and ResNet-152.\n0.4 0.6 0.8 1.0 1.2 1.4\ncomputational demand (GFlops).737475767778top-1 accuracy\n73.3%75.2%76.7%77.2%\n73.9%75.7%77.2%77.7%\nMnasNet\nAA-MnasNet (ours)\nFigure 6. ImageNet top-1 accuracy as a function of computational\ndemand for MnasNet (black) and Attention-Augmented-MnasNet\n(red) with width multipliers 0:75,1:0,1:25and1:4.\nA.3. 2D Relative Self-Attention implementation\nWhile our method is simple and only requires matrix\nmultiplication, addition and the softmax operation (Equations 3 and 4), our implementation relies on non-trivial operations (e.g. tiling, transposing and reshaping) because\nno low-level kernels currently exist for hardware platforms.\nFuture work may develop specialized kernels as previously"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c511c-6db4-11ee-ac36-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-53", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "no low-level kernels currently exist for hardware platforms.\nFuture work may develop specialized kernels as previously\ndone for convolutions. Therefore, we believe that current\nlatency times (Table 2) re\ufb02ect the lack of dedicated engineering as opposed to inef\ufb01ciency in the proposed method.\ndef shape list(x):\n\"\"\"Return list of dims, statically where possible.\"\"\"\nstatic = x.get shape().as list()\nshape = tf.shape(x)\nret = []\nfor i, static dim in enumerate(static):\ndim = static dim or shape[i]\nret.append(dim)\nreturn ret\ndef split heads 2d(inputs, Nh):\n\"\"\"Split channels into multiple heads.\"\"\"\nB, H, W, d = shape list(inputs)\nretshape = [B, H, W, Nh, d // Nh]\nsplit = tf.reshape(inputs, ret shape)\nreturn tf.transpose(split, [0, 3, 1, 2, 4])\ndef combine heads 2d(inputs):\n\"\"\"Combine heads (inverse of split heads 2d).\"\"\"\ntransposed = tf.transpose(inputs, [0, 2, 3, 1, 4])\nNh, channels = shape list(transposed)[ \u00002:]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c511d-6db4-11ee-a257-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-54", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "transposed = tf.transpose(inputs, [0, 2, 3, 1, 4])\nNh, channels = shape list(transposed)[ \u00002:]\nretshape = shape list(transposed)[: \u00002] + [Nh \u0003channels]\nreturn tf.reshape(transposed, ret shape)\ndef rel toabs(x):\n\"\"\"Converts tensor from relative to aboslute indexing.\"\"\"\n# [B, Nh, L, 2L \u00001]\nB, Nh, L, = shape list(x)\n# Pad to shift from relative to absolute indexing.\ncolpad = tf.zeros((B, Nh, L, 1))\nx = tf.concat([x, col pad], axis=3)\nflat x = tf.reshape(x, [B, Nh, L \u00032\u0003L])\nflat pad = tf.zeros((B, Nh, L \u00001))\nflat xpadded = tf.concat([flat x, flat pad], axis=2)\n# Reshape and slice out the padded elements.\nfinal x = tf.reshape(flat xpadded, [B, Nh, L+1, 2 \u0003L\u00001])"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c511e-6db4-11ee-9a41-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-55", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "final x = tf.reshape(flat xpadded, [B, Nh, L+1, 2 \u0003L\u00001])\nfinal x = final x[:, :, :L, L \u00001:]\nreturn final x\ndef relative logits 1d(q, rel k, H, W, Nh, transpose mask):\n\"\"\"Compute relative logits along one dimenion.\"\"\"\nrellogits = tf.einsum(\u2019bhxyd,md \u0000>bhxym\u2019, q, rel k)\n# Collapse height and heads\nrellogits = tf.reshape(\nrellogits, [ \u00001, Nh \u0003H, W, 2 \u0003W\u00001])\nrellogits = rel toabs(rel logits)\n# Shape it and tile height times\nrellogits = tf.reshape(rel logits, [ \u00001, Nh, H, W, W])\nrellogits = tf.expand dims(rel logits, axis=3)\nrellogits = tf.tile(rel logits, [1, 1, 1, H, 1, 1])\n# Reshape for adding to the logits.\nrellogits = tf.transpose(rel logits, transpose mask)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c511f-6db4-11ee-a41e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-56", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "# Reshape for adding to the logits.\nrellogits = tf.transpose(rel logits, transpose mask)\nrellogits = tf.reshape(rel logits, [ \u00001, Nh, H \u0003W, H\u0003W])\nreturn rel logits\nFigure 7. Helper functions in Tensor\ufb02ow for 2D relative selfattention.def relative logits(q, H, W, Nh, dkh):\n\"\"\"Compute relative logits.\"\"\"\n# Relative logits in width dimension first.\nrelembeddings w = tf.get variable(\n\u2019rwidth\u2019, shape=(2 \u0003W\u00001, dkh),\ninitializer=tf.random normal initializer(dkh \u0003\u0003\u00000.5))\n# [B, Nh, HW, HW]\nrellogits w = relative logits 1d(\nq, rel embeddings w, H, W, Nh, [0, 1, 2, 4, 3, 5])\n# Relative logits in height dimension next.\n# For ease, we 1) transpose height and width,\n# 2) repeat the above steps and\n# 3) transpose to eventually put the logits\n# in their right positions.\nrelembeddings h = tf.get variable("}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5120-6db4-11ee-ac92-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-57", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "# 2) repeat the above steps and\n# 3) transpose to eventually put the logits\n# in their right positions.\nrelembeddings h = tf.get variable(\n\u2019rheight\u2019, shape=(2 \u0003H\u00001, dkh),\ninitializer=tf.random normal initializer(dkh \u0003\u0003\u00000.5))\n# [B, Nh, HW, HW]\nrellogits h = relative logits 1d(\ntf.transpose(q, [0, 1, 3, 2, 4]),\nrelembeddings h, W, H, Nh, [0, 1, 4, 2, 5, 3])\nreturn rel logits h, rel logits w\ndef self attention 2d(inputs, dk, dv, Nh, relative=True):\n\"\"\"2d relative self \u0000attention.\"\"\"\n, H, W, = shape list(inputs)\ndkh = dk // Nh\ndvh = dv // Nh\nflatten hw = lambda x, d: tf.reshape(x, [ \u00001, Nh, H \u0003W, d])\n# Compute q, k, v\nkqv = tf.layers.conv2d(inputs, 2 \u0003dk + dv, 1)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5121-6db4-11ee-a010-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-58", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "# Compute q, k, v\nkqv = tf.layers.conv2d(inputs, 2 \u0003dk + dv, 1)\nk, q, v = tf.split(kqv, [dk, dk, dv], axis=3)\nq\u0003= dkh \u0003\u0003 \u0000 0.5 # scaled dot \u0000product\n# After splitting, shape is [B, Nh, H, W, dkh or dvh]\nq = split heads 2d(q, Nh)\nk = split heads 2d(k, Nh)\nv = split heads 2d(v, Nh)\n# [B, Nh, HW, HW]\nlogits = tf.matmul(flatten hw(q, dkh), flatten hw(k, dkh),\ntranspose b=True)\nif relative:\nrellogits h, rel logits w = relative logits(q, H, W, Nh,\ndkh)\nlogits += rel logits h\nlogits += rel logits w\nweights = tf.nn.softmax(logits)\nattn out = tf.matmul(weights, flatten hw(v, dvh))"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5122-6db4-11ee-abbb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-59", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "weights = tf.nn.softmax(logits)\nattn out = tf.matmul(weights, flatten hw(v, dvh))\nattn out = tf.reshape(attn out, [ \u00001, Nh, H, W, dvh])\nattn out = combine heads 2d(attn out)\n# Project heads\nattn out = tf.layers.conv2d(attn out, dv, 1)\nreturn attn out\ndef augmented conv2d(X, Fout, k, dk, dv, Nh, relative):\nconv out = tf.layers.conv2d(inputs=X, filters=Fout \u0000dv,\nkernel size=k, padding=\u2019same\u2019)\nattn out = self attention 2d(X, dk, dv, Nh, relative=\nrelative)\nreturn tf.concat([conv out, attn out], axis=3)\nFigure 8. Tensor\ufb02ow code for 2D relative self-attention.\nA.4. Attention visualizations.\nIn Figure 10, we present attention maps visualizations\nfor the input image shown in Figure 9. We see that attention\nheads learn to specialize to different content and notably can\ndelineate object boundaries."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5123-6db4-11ee-a7c5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-60", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "for the input image shown in Figure 9. We see that attention\nheads learn to specialize to different content and notably can\ndelineate object boundaries.\nFigure 9. An input image. The red crosses indexed 1 to 4 represent\nthe pixel locations for which we show the attention maps in Figure\n10.\nFigure 10. Visualization of attention maps for an augmented convolution in the Attention-Augmented-ResNet50. Rows correspond to the 8 different heads and columns correspond to the 4\npixel locations depicted in the input image (See Figure 9)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5124-6db4-11ee-b4b7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-0", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Learning Temporal Pose Estimation\nfrom Sparsely-Labeled Videos\nGedas Bertasius1;2, Christoph Feichtenhofer1, Du Tran1, Jianbo Shi2, Lorenzo Torresani1\n1Facebook AI,2University of Pennsylvania\nAbstract\nModern approaches for multi-person pose estimation in video require large amounts\nof dense annotations. However, labeling every frame in a video is costly and labor\nintensive. To reduce the need for dense annotations, we propose a PoseWarper\nnetwork that leverages training videos with sparse annotations (every kframes) to\nlearn to perform dense temporal pose propagation and estimation. Given a pair\nof video frames\u2014a labeled Frame A and an unlabeled Frame B\u2014we train our\nmodel to predict human pose in Frame A using the features from Frame B by\nmeans of deformable convolutions to implicitly learn the pose warping between A\nand B. We demonstrate that we can leverage our trained PoseWarper for several\napplications. First, at inference time we can reverse the application direction\nof our network in order to propagate pose information from manually annotated\nframes to unlabeled frames. This makes it possible to generate pose annotations\nfor the entire video given only a few manually-labeled frames. Compared to\nmodern label propagation methods based on optical \ufb02ow, our warping mechanism"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5125-6db4-11ee-bf8e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-1", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "for the entire video given only a few manually-labeled frames. Compared to\nmodern label propagation methods based on optical \ufb02ow, our warping mechanism\nis much more compact ( 6M vs 39M parameters), and also more accurate ( 88:7%\nmAP vs 83:8%mAP). We also show that we can improve the accuracy of a\npose estimator by training it on an augmented dataset obtained by adding our\npropagated poses to the original manual labels. Lastly, we can use our PoseWarper\nto aggregate temporal pose information from neighboring frames during inference.\nThis allows us to obtain state-of-the-art pose detection results on PoseTrack2017\nand PoseTrack2018 datasets. Code has been made available at: https://github.\ncom/facebookresearch/PoseWarper .\n1 Introduction\nIn recent years, visual understanding methods [ 1\u201315] have made tremendous progress, partly because\nof advances in deep learning [ 16\u201319], and partly due to the introduction of large-scale annotated\ndatasets [ 20,21]. In this paper we consider the problem of pose estimation, which has greatly\nbene\ufb01tted from the recent creation of large-scale datasets [ 22,23]. Most of the recent advances\nin this area, though, have concentrated on the task of pose estimation in still-images [ 3,23\u201327]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5126-6db4-11ee-8217-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-2", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "in this area, though, have concentrated on the task of pose estimation in still-images [ 3,23\u201327].\nHowever, directly applying these image-level models to video is challenging due to nuisance factors\nsuch as motion blur, video defocus, and frequent pose occlusions. Additionally, the process of\ncollecting annotated pose data in multi-person videos is costly and time consuming. A video typically\ncontains hundreds of frames that need to be densely-labeled by human annotators. As a result,\ndatasets for video pose estimation [ 22] are typically smaller and less diverse compared to their image\ncounterparts [ 21]. This is problematic because modern deep models require large amounts of labeled\ndata to achieve good performance. At the same time, videos have high informational redundancy as\nthe content changes little from frame to frame. This raises the question of whether every single frame\nin a training video needs to be labeled in order to achieve good pose estimation accuracy.\n33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.arXiv:1906.04016v3  [cs.CV]  11 Dec 2019\nTo reduce the reliance on densely annotated video pose data, in this work, we introduce the PoseWarper network, which operates on sparsely annotated videos, i.e., videos where pose annotations"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5127-6db4-11ee-bdeb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-3", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "are given only every kframes. Given a pair of frames from the same video\u2014a labeled Frame A\nand an unlabeled Frame B\u2014we train our model to detect pose in Frame A, using the features from\nFrame B. To achieve this goal, our model leverages deformable convolutions [ 28] across space and\ntime. Through this mechanism, our model learns to sample features from an unlabeled Frame B to\nmaximize pose detection accuracy in a labeled Frame A.\nOur trained PoseWarper can then be used for several applications. First, we can leverage PoseWarper\nto propagate pose information from a few manually-labeled frames across the entire video. Compared\nto modern optical \ufb02ow propagation methods such as FlowNet2 [ 29], our PoseWarper produces more\naccurate pose annotations ( 88:7%mAP vs 83:8%mAP), while also employing a much more compact\nwarping mechanism ( 6M vs 39M parameters). Furthermore, we show that our propagated poses can\nserve as effective pseudo labels for training a more accurate pose detector. Finally, our PoseWarper\ncan be used to aggregate temporal pose information from neighboring frames during inference. This\nnaturally renders the approach more robust to occlusion or motion blur in individual frames, and leads\nto state-of-the-art pose detection results on the PoseTrack2017 and PoseTrack2018 datasets [22]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5128-6db4-11ee-92ae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-4", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "to state-of-the-art pose detection results on the PoseTrack2017 and PoseTrack2018 datasets [22].\n2 Related Work\nMulti-Person Pose Detection in Images. The traditional approaches for pose estimation leverage\npictorial structures model [ 30\u201334], which represents human body as a tree-structured graph with\npairwise potentials between the connected body parts. These approaches have been highly successful\nin the past, but they tend to fail if some of body parts are occluded. These issues have been partially\naddressed by the models that assume a non-tree graph structure [ 35\u201338]. However, most modern\napproaches for single image pose estimation are based on convolutional neural networks [ 3,6,23\u2013\n27,39\u201345]. The method in [ 3] regresses (x;y)joint coordinates directly from the images. More\nrecent work [ 25] instead predicts pose heatmaps, which leads to an easier optimization problem.\nSeveral approaches [ 24,26,39,46] propose an iterative pose estimation pipeline where the predictions\nare re\ufb01ned at different stages inside a CNN or via a recurrent network. The methods in [ 6,23,45]\ntackle pose estimation problem in a top-down fashion, \ufb01rst detecting bounding boxes of people, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5129-6db4-11ee-a235-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-5", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "tackle pose estimation problem in a top-down fashion, \ufb01rst detecting bounding boxes of people, and\nthen predicting the pose heatmaps from the cropped images. The work in [ 24] proposes part af\ufb01nity\n\ufb01elds module that captures pairwise relationships between different body parts. The approaches\nin [42,43] leverage a bottom-up pipeline \ufb01rst predicting the keypoints, and then assembling them\ninto instances. Lastly, a recent work in [ 27], proposes an architecture that preserves high resolution\nfeature maps, which is shown to be highly bene\ufb01cial for the multi-person pose estimation task.\nMulti-Person Pose Detection in Video. Due to a limited number of large scale benchmarks for\nvideo pose detection, there has been signi\ufb01cantly fewer methods in the video domain. Several prior\nmethods [ 22,47,48] tackle a video pose estimation task as a two-stage problem, \ufb01rst detecting the\nkeypoints in individual frames, and then applying temporal smoothing techniques. The method in [ 49]\nproposes a spatiotemporal CRF, which is jointly optimized for the pose prediction in video. The\nwork in [ 50] proposes a personalized video pose estimation framework, which is accomplished by"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c512a-6db4-11ee-90eb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-6", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "proposes a spatiotemporal CRF, which is jointly optimized for the pose prediction in video. The\nwork in [ 50] proposes a personalized video pose estimation framework, which is accomplished by\n\ufb01netuning the model on a few frames with high con\ufb01dence keypoints in each video. The approaches\nin [51,52] leverage \ufb02ow based representations for aligning features temporally across multiple\nframes, and then using such aligned features for pose detection in individual frames.\nIn contrast to these prior methods, our primary objective is to learn an effective video pose detector\nfrom sparsely labeled videos. Our approach has similarities to the methods in [ 51,52], which use\n\ufb02ow representations for feature alignment. However, unlike our model, the methods in [ 51,52] do\nnot optimize their \ufb02ow representations end-to-end with respect to the pose detection task. As we will\nshow in our experiments, this is important for strong performance.\n3 The PoseWarper Network\nOverview. Our goal is to design a model that learns to detect pose from sparsely labeled videos.\nSpeci\ufb01cally, we assume that pose annotations in training videos are available every kframes. Inspired\nby a recent self-supervised approach for learning facial attribute embeddings [ 53], we formulate the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c512b-6db4-11ee-a237-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-7", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "by a recent self-supervised approach for learning facial attribute embeddings [ 53], we formulate the\nfollowing task. Given two video frames\u2014a labeled Frame A and an unlabeled Frame B\u2014our model\n2\n1. Training (Unlabeled      Labeled) \n2. Video Pose Propagation (Labeled      Unlabeled)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c512c-6db4-11ee-a0b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-8", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Unlabeled Frame B (time t+  )\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c512d-6db4-11ee-b022-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c512e-6db4-11ee-a04e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c512f-6db4-11ee-9426-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5130-6db4-11ee-af71-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5131-6db4-11ee-b037-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5132-6db4-11ee-be85-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5133-6db4-11ee-92c1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5134-6db4-11ee-a16a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5135-6db4-11ee-95e2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-17", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Frame A (time t)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5136-6db4-11ee-9341-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-18", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Features Relating the          Two Frames\nPredicted Pose in Frame A\nForward Pass\nBackward Pass"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5137-6db4-11ee-95dc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-19", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Labeled Frame A (time t+  )\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5138-6db4-11ee-9c46-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5139-6db4-11ee-84ff-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c513a-6db4-11ee-a0b5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c513b-6db4-11ee-bc82-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c513c-6db4-11ee-8e73-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c513d-6db4-11ee-b8b3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c513e-6db4-11ee-bc4e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c513f-6db4-11ee-947f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5140-6db4-11ee-acaf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-28", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Frame B (time t)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5141-6db4-11ee-a6ce-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-29", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Ground Truth Pose Propagated to Frame BFeature Warping\nGround Truth Pose in Frame AFeatures Relating the          Two FramesPose Detection\nFeature WarpingFigure 1: A high level overview of our approach for using sparsely labeled videos for pose detection.\nFaces in the \ufb01gure are arti\ufb01cially masked for privacy reasons. In each training video, pose annotations\nare available only every kframes. During training, our system considers a pair of frames\u2013a labeled\nFrame A, and an unlabeled Frame B, and aims to detect pose in Frame A, using the features from\nFrame B. Our training procedure is designed to achieve two goals: 1) our model must be able to\nextract motion offsets relating these two frames. 2) Using these motion offsets our model must\nthen be able to rewarp the detected pose heatmap extracted from an unlabeled Frame B in order to\noptimize the accuracy of pose detection in a labeled Frame A. After training, we can apply our model\nin reverse order to propagate pose information across the entire video from ground truth poses given\nfor only a few frames.\nis allowed to compare Frame A to Frame B but it must predict Pose A (i.e., the pose in Frame A)\nusing the features from Frame B, as illustrated in Figure 1 (top)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5142-6db4-11ee-a400-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-30", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "using the features from Frame B, as illustrated in Figure 1 (top).\nAt \ufb01rst glance, this task may look overly challenging: how can we predict Pose A by merely using\nfeatures from Frame B? However, suppose that we had body joint correspondences between Frame A\nand Frame B. In such a scenario, this task would become trivial, as we would simply need to spatially\n\u201cwarp\u201d the feature maps computed from frame B according to the set of correspondences relating\nframe B to frame A. Based on this intuition, we design a learning scheme that achieves two goals: 1)\nBy comparing Frame A and Frame B, our model must be able to extract motion offsets relating these\ntwo frames. 2) Using these motion offsets our model must be able to rewarp the pose extracted from\nan unlabeled Frame B in order to optimize pose detection accuracy in a labeled Frame A.\nTo achieve these goals, we \ufb01rst feed both frames through a backbone CNN that predicts pose heatmaps\nfor each of the frames. Then, the resulting heatmaps from both frames are used to determine which\npoints from Frame B should be sampled for detection in Frame A. Finally, the resampled pose\nheatmap from Frame B is used to maximize accuracy of Pose A."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5143-6db4-11ee-bd0e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-31", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "points from Frame B should be sampled for detection in Frame A. Finally, the resampled pose\nheatmap from Frame B is used to maximize accuracy of Pose A.\nBackbone Network. Due to its high ef\ufb01ciency and accuracy, we use the state-of-the-art High\nResolution Network (HRNet-W48) [ 27] as our backbone CNN. However, we note that our system can\neasily integrate other architectures as well. Thus, we envision that future improvements in still-image\npose estimation will further improve the effectiveness of our approach.\nDeformable Warping. Initially, we feed Frame A and Frame B through our backbone CNN, which\noutputs pose heatmaps fAandfB. Then, we compute the difference  A;B=fA\u0000fB. The resulting\nfeature tensor  A;Bis provided as input to a stack of 3\u00023simple residual blocks (as in standard\nResNet-18 or ResNet-34 models), which output a feature tensor \u001eA;B. The feature tensor \u001eA;Bis\nthen fed into \ufb01ve 3\u00023convolutional layers each using a different dilation rate d2f3;6;12;18;24g\n3\nLabeled Frame A (time t)\nOffsets"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5144-6db4-11ee-812c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-32", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Warped  Feature MapDifference <latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5145-6db4-11ee-882f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5146-6db4-11ee-8f94-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5147-6db4-11ee-a509-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5148-6db4-11ee-b654-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5149-6db4-11ee-a3db-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c514a-6db4-11ee-b328-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c514b-6db4-11ee-94ad-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c514c-6db4-11ee-8342-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c514d-6db4-11ee-b5ad-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-41", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Unlabeled Frame B (time t+  )\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c514e-6db4-11ee-933e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c514f-6db4-11ee-87e8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5150-6db4-11ee-8f6e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5151-6db4-11ee-b6e8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5152-6db4-11ee-993d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5153-6db4-11ee-90ec-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5154-6db4-11ee-a08e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5155-6db4-11ee-abeb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5156-6db4-11ee-a48c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-50", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Pose      in Frame A A Stack of Residual            3x3 Blocks  Def. Conv. (dilation = 12)   2D Conv. (dilation = 12)   2D Conv. (dilation = 6)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5157-6db4-11ee-a2d4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-51", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "2D Conv. (dilation = 24)\n  Def. Conv. (dilation = 6)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5158-6db4-11ee-b14f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-52", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Def. Conv. (dilation = 24)\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5159-6db4-11ee-969b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c515a-6db4-11ee-949d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c515b-6db4-11ee-8d34-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c515c-6db4-11ee-98cd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c515d-6db4-11ee-9f81-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c515e-6db4-11ee-81f1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c515f-6db4-11ee-9a3f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5160-6db4-11ee-8f3a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5161-6db4-11ee-89d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-61", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Def. Conv. (dilation = 3)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5162-6db4-11ee-a457-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-62", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Def. Conv. (dilation = 18)   2D Conv. (dilation = 18)\n   2D Conv. (dilation = 3)\nFigure 2: An illustration of our PoseWarper architecture. Given a labeled Frame A and an unlabeled\nFrame B, which are separated by \u000esteps in time, our goal is to detect pose in a labeled Frame A\nusing the features from an unlabeled Frame B. First, we predict pose heatmaps for both frames. Then,\nwe compute the difference between pose heatmaps in Frame A and Frame B and feed it through a\nstack of 3\u00023residual blocks. Afterwards, we attach \ufb01ve 3\u00023convolutional layers with dilation\nratesd2f3;6;12;18;24gand predict \ufb01ve sets of offsets o(d)(pn)for each pixel location pn. The\npredicted offsets are used to rewarp pose heatmap B. All \ufb01ve rewarped heatmaps are then summed\nand the resulting tensor is used to predict pose in Frame A.\nto predict \ufb01ve sets of offsets o(d)(pn)at all pixel locations pn. The motivation for using different\ndilation rates at the offset prediction stage comes from the need to consider motion cues at different"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5163-6db4-11ee-beba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-63", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "dilation rates at the offset prediction stage comes from the need to consider motion cues at different\nspatial scales. When the body motion is small, a smaller dilation rate may be more useful as it\ncaptures subtle motion cues. Conversely, if the body motion is large, using large dilation rate allows\nus to incorporate relevant information further away. Next, the predicted offsets are used to spatially\nrewarp the pose heatmap fB. We do this for each of the \ufb01ve sets of offsets o(d), and then sum up all\n\ufb01ve rewarped pose heatmaps to obtain a \ufb01nal output gA;B, which is used to predict pose in Frame A.\nWe implement the warping mechanism via a deformable convolution [ 28], which takes 1) the offsets\no(d)(pn), and 2) the pose heatmap fBas its inputs, and then outputs a newly sampled pose heatmap\ngA;B. The subscript (A;B)is used to indicate that even though gA;Bwas resampled from tensor fB,\nthe offsets for rewarping were computed using  A;B, which contains information from both frames.\nAn illustration of our architecture is presented in Figure 2.\nLoss Function. As in [ 27], we use a standard pose estimation loss function which computes a mean"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5164-6db4-11ee-850c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-64", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "An illustration of our architecture is presented in Figure 2.\nLoss Function. As in [ 27], we use a standard pose estimation loss function which computes a mean\nsquared error between the predicted, and the ground-truth heatmaps. The ground-truth heatmap is\ngenerated by applying a 2D Gaussian around the location of each joint.\nPose Annotation Propagation. During training, we force our model to warp pose heatmap fBfrom\nan unlabeled frame B such that it would match the ground-truth pose heatmap in a labeled Frame A.\nAfterwards, we can reverse the application direction of our network. This then, allows us to propagate\npose information from manually annotated frames to unlabeled frames (i.e. from a labeled Frame\nA to an unlabeled Frame B). Speci\ufb01cally, given a pose annotation in Frame A, we can generate its\nrespective ground-truth heatmap yAby applying a 2D Gaussian around the location of each joint\n(identically to how it was done in [ 23,27]. Then, we can predict the offsets for warping ground-truth\nheatmapyAto an unlabeled Frame B, from the feature difference  B;A=fB\u0000fA. Lastly, we"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5165-6db4-11ee-820a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-65", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "heatmapyAto an unlabeled Frame B, from the feature difference  B;A=fB\u0000fA. Lastly, we\nuse our deformable warping scheme to warp the ground-truth pose heatmap yAto Frame B, thus,\npropagating pose annotations to unlabeled frames in the same video. See Figure 1 (bottom) for a\nhigh-level illustration of this scheme.\nTemporal Pose Aggregation at Inference Time. Instead of using our model to propagate pose\nannotations on training videos, we can also use our deformable warping mechanism to aggregate pose\ninformation from nearby frames during inference in order to improve the accuracy of pose detection.\nFor every frame at time t, we want to aggregate information from all frames at times t+\u000ewhere\n\u000e2f\u0000 3;\u00002;\u00001;0;1;2;3g. Such a pose aggregation procedure renders pose estimation more robust\nto occlusions, motion blur, and video defocus.\n4\n   Ground Truth Reference             Frame (time t)PoseWarper (time t+1)FlowNet2 (time t+1)\nFigure 3: The results of a video pose propagation task by our PoseWarper and FlowNet2 [ 29]. The\n\ufb01rst frame in each 3-frame sequence illustrates a labeled reference frame at time t. For simplicity,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5166-6db4-11ee-9e84-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-66", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "\ufb01rst frame in each 3-frame sequence illustrates a labeled reference frame at time t. For simplicity,\nwe show only the \u201cright ankle\u201d body joint for one person, denoted by a pink circle in each of the\nframes (please zoom in for a clearer view). The second frame depicts our propagated \u201cright ankle\u201d\ndetection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame shows\nthe propagated detection in frame t+1 produced by the FlowNet2 baseline. In contrast to our method,\nFlowNet2 fails to propagate poses when there is large motion, blurriness or occlusions.\nConsider a pair of frames, ItandIt+\u000e. In this case, we want to use pose information from frame\nIt+\u000eto improve pose detection in frame It. To do this, we \ufb01rst feed both frames through our trained\nPoseWarper model, and obtain a spatially rewarped (resampled) pose heatmap gt;t+\u000e, which is aligned\nwith respect to frame Itusing the features from frame It+\u000e. We can repeat this procedure for every \u000e\nvalue, and then aggregate pose information from multiple frames via a summation asP\n\u000egt;t+\u000e."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5167-6db4-11ee-8ead-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-67", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "value, and then aggregate pose information from multiple frames via a summation asP\n\u000egt;t+\u000e.\nImplementation Details. Following the framework in [ 27], for training, we crop a 384\u0002288\nbounding box around each person and use it as input to our model. During training, we use ground\ntruth person bounding boxes. We also employ random rotations, scaling, and horizontal \ufb02ipping to\naugment the data. To learn the network, we use the Adam optimizer [ 54] with a base learning rate\nof10\u00004, which is reduced to 10\u00005and10\u00006after10, and 15epochs, respectively. The training is\nperformed using 4Tesla M40 GPUs, and is terminated after 20epochs. We initialize our model with\na HRNet-W48 [ 27] pretrained for a COCO keypoint estimation task. To train the deformable warping\nmodule, we select Frame B, with a random time-gap \u000e2[\u00003;3]relative to Frame A. To compute\nfeatures relating the two frames, we use twenty 3\u00023residual blocks each with 128channels. Even\nthough this seems like many convolutional layers, due to a small number of channels in each layer,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5168-6db4-11ee-9b9f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-68", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "though this seems like many convolutional layers, due to a small number of channels in each layer,\nthis amounts to only 5:8M parameters (compared to 39M required to compute optical \ufb02ow in [ 29]).\nTo compute the offsets o(d), we use \ufb01ve 3\u00023convolutional layers, each using a different dilation\nrate (d= 3;6;12;18;24). To resample the pose heatmap fB, we employ \ufb01ve 3\u00023deformable\nconvolutional layers, each applied to one of the \ufb01ve predicted offset maps o(d). The \ufb01ve deformable\nconvolution layers too employ different dilation rates of 3;6;12;18;24. During testing, we follow\nthe same two-stage framework used in [ 27,23]: \ufb01rst, we detect the bounding boxes for each person in\nthe image using the detector in [ 48], and then feed the cropped images to our pose estimation model.\n4 Experiments\nIn this section, we present our results on the PoseTrack [ 22] dataset. We demonstrate the effectiveness\nof our approach on three applications: 1) video pose propagation, 2) training a network on annotations\naugmented with propagated pose pseudo-labels, 3) temporal pose aggregation during inference.\n5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5169-6db4-11ee-987a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-69", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "augmented with propagated pose pseudo-labels, 3) temporal pose aggregation during inference.\n5\nTable 1: The results of video pose propagation on the PoseTrack2017 [ 22] validation set (measured in\nmAP). We propagate pose information across the entire video from the manual annotations provided\nin few frames. To study the effect of different levels of dilated convolutions in our PoseWarper\narchitecture, we also include several ablation baselines (see the bottom half of the table).\nMethod Head Shoulder Elbow Wrist Hip Knee Ankle Mean\nPseudo-labeling w/ HRNet [27] 79.1 86.5 81.4 74.7 81.4 79.4 72.3 79.3\nOptical Flow Propagation (Farneback [55]) 76.5 82.3 74.3 69.2 80.8 74.8 70.1 75.5\nOptical Flow Propagation (FlowNet2 [29]) 82.7 91.0 83.8 78.4 89.7 83.6 78.1 83.8\nPoseWarper (no dilated convs) 86.1 91.7 88.0 83.5 90.2 87.3 84.6 87.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c516a-6db4-11ee-9aca-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-70", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "PoseWarper (no dilated convs) 86.1 91.7 88.0 83.5 90.2 87.3 84.6 87.2\nPoseWarper (1 dilated conv) 85.0 91.6 88.0 83.7 89.6 87.3 84.7 87.0\nPoseWarper (2 dilated convs) 85.8 92.4 88.8 84.9 91.0 88.4 86.0 88.0\nPoseWarper (3 dilated convs) 86.1 92.6 89.2 85.5 91.3 88.8 86.3 88.4\nPoseWarper (4 dilated convs) 86.3 92.6 89.5 85.9 91.9 88.8 86.4 88.6\nPoseWarper (5 dilated convs) 86.0 92.7 89.5 86.0 91.5 89.1 86.6 88.7\n4.1 Video Pose Propagation\nQuantitative Results. To verify that our model learns to capture pose correspondences, we apply it\nto the task of video pose propagation, i.e., propagating poses across time from a few labeled frames.\nInitially, we train our PoseWarper in a sparsely labeled video setting according to the procedure"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c516b-6db4-11ee-aecf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-71", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Initially, we train our PoseWarper in a sparsely labeled video setting according to the procedure\ndescribed above. In this setting, every 7thframe of a training video is labeled, i.e. there are 6\nunlabeled frames between each pair of manually labeled frames. Since each video contains on\naverage 30 frames, we have approximately 5annotated frames uniformly spaced out in each video.\nOur goal then, is to use our learned PoseWarper to propagate pose annotations from manually-labeled\nframes to all unlabeled frames in the same video. Speci\ufb01cally, for each labeled frame in a video,\nwe propagate its pose information to the three preceding and three subsequent frames. We train our\nPoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22] and then perform\nour evaluations on the validation set.\nTo evaluate the effectiveness of our approach, we compare our model to several relevant baselines.\nAs our weakest baseline, we use our trained HRNet [ 27] model that simply predicts pose for every\nsingle frame in a video. Furthermore, we also include a few propagation baselines based on warping\nannotations using optical \ufb02ow. The \ufb01rst of these uses a standard Farneback optical \ufb02ow [ 55] to\nwarp the manually-labeled pose in each labeled frame to its three preceding and three subsequent"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c516c-6db4-11ee-b1ae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-72", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "warp the manually-labeled pose in each labeled frame to its three preceding and three subsequent\nframes. We also include a more advanced optical \ufb02ow propagation baseline that uses FlowNet2\noptical \ufb02ow [29]. Finally, we evaluate our PoseWarper model.\nIn Table 1, we present our quantitative results for video pose propagation. The evaluation is done\nusing an mAP metric as in [ 42]. Our best model achieves a 88:7%mAP, while the optical \ufb02ow\npropagation baseline using FlowNet2 [ 29] yields an accuracy of 83:8%mAP. We also note that\ncompared to the FlowNet2 [ 29] propagation baseline, our PoseWarper warping mechanism is not\nonly more accurate, but also signi\ufb01cantly more compact ( 6M vs 39M parameters).\nAblation Studies on Dilated Convolution. In Table 1, we also present the results investigating the\neffect of different levels of dilated convolutions in our PoseWarper architecture. We evaluate all these\nvariants on the task of video pose propagation. First, we report that removing dilated convolution\nblocks from the original architecture reduces the accuracy from 88:7mAP to 87:2mAP. We also\nnote that a network with a single dilated convolution (using a dilation rate of 3) yields 87:0mAP."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c516d-6db4-11ee-86cd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-73", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "note that a network with a single dilated convolution (using a dilation rate of 3) yields 87:0mAP.\nAdding a second dilated convolution level (using dilation rates of 3;6) improves the accuracy to 88:0.\nThree dilation levels (with dilation rates of 3;6;12) yield a mAP of 88:4and four levels (dilation\nrates of 3;6;12;18) give a mAP of 88:6. A network with 5dilated convolution levels yields 88:7\nmAP. Adding more dilated convolutions does not improve the performance further. Additionally, we\nalso experimented with two networks that use dilation rates of 1;2;3;4;5, and 4;8;16;24;32, and\nreport that such models yield mAPs of 88:6and88:5, respectively, which are slightly lower.\nQualitative Comparison to FlowNet2. In Figure 3, we include an illustration of the motion encoded\nby PoseWarper, and compare it to the optical \ufb02ow computed by FlowNet2 for the video pose\npropagation task. The \ufb01rst frame in each 3-frame sequence illustrates a labeled reference frame at\ntime t. For a cleaner visualization, we show only the \u201cright ankle\u201d body joint for one person, which is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c516e-6db4-11ee-8310-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-74", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "time t. For a cleaner visualization, we show only the \u201cright ankle\u201d body joint for one person, which is\nmarked with a pink circle in each of the frames. The second frame depicts our propagated \u201cright\n6\na) Training HRNet with Propagated Pose Pseudo Labelsb) Temporal Pose Aggregation during Inference0 100 200 300# Labeled Training Videos707580Accuracy (mAP)GT (7x)GT (1x) + pGT (6x)GT (1x)02468# Labeled Frames Per Video707580Accuracy (mAP)GT (7x)GT (1x) + pGT (6x)GT (1x)02468# Labeled Frames Per Video707580Accuracy (mAP)GT (7x)GT (1x) + T-Agg.GT (1x)0 100 200 300# Labeled Training Videos707580Accuracy (mAP)GT (7x)GT (1x) + T-Agg.GT (1x)Figure 4: A \ufb01gure illustrating the value of a) training a standard HRNet [ 27] using our propagated\npose pseudo labels (left), and b) our temporal pose aggregation scheme during inference. In both\nsettings, we study pose detection performance as a function of 1) number of sparsely-labeled training"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c516f-6db4-11ee-af1a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-75", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "settings, we study pose detection performance as a function of 1) number of sparsely-labeled training\nvideos (with 1manually-labeled frame per video), and 2) number of labeled frames per video (with\n50sparsely-labeled videos in total). All baselines are based on retraining the standard HRNet [ 27]\nmodel on the different training sets. The \"GT (1x)\" baseline is trained in a standard way on sparsely\nlabeled video data. The \"GT (7x)\" baseline uses 7x more manually annotated data relative to the\n\"GT (1x)\" baseline. Our approach on the left sub\ufb01gure (\"GT (1x) + pGT (6x)\"), augments the\noriginal sparsely labeled video data with our propagated pose pseudo labels ( 6nearby frames for\nevery manually-labeled frame). Lastly, in b) \"GT (1x) + T-Agg\" denotes the use of PoseWarper to\nfuse pose information from multiple neighboring frames during inference (training is done as in \"GT\n(1x)\" baseline). From the results, we observe that both application modalities of PoseWarper provide\nan effective way to achieve strong pose accuracy while reducing the number of manual annotations.\nankle\u201d detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5170-6db4-11ee-b70b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-76", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "ankle\u201d detection from the labeled frame in time t to the unlabeled frame in time t+1. The third frame\nshows the propagated detection in frame t+1 produced by the FlowNet2 baseline. These results\nsuggest that FlowNet2 struggles to accurately warp poses if 1) there is large motion, 2) occlusions, or\n3) blurriness. In contrast, our PoseWarper handles these cases robustly, which is also indicated by our\nresults in Table 1 (i.e., 88:7vs83:8mAP w.r.t. FlowNet2).\n4.2 Data Augmentation with PoseWarper\nHere we consider the task of propagating poses on sparsely labeled training videos using PoseWarper,\nand then using them as pseudo-ground truth labels (in addition to the original manual labels) to\ntrain a standard HRNet-W48 [ 27]. For this experiment, we study the pose detection accuracy as\na function of two variables: 1) the total number of sparsely-labeled videos, and 2) the number of\nmanually-annotated frames per video. We aim to study how much we can reduce manual labeling\nthrough our mechanism of pose propagation, while maintaining strong pose accuracy. Note, that we\n\ufb01rst train our PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5171-6db4-11ee-8dec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-77", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "\ufb01rst train our PoseWarper on sparsely labeled videos from the training set of PoseTrack2017 [ 22].\nThen, we propagate pose annotations on the same set of training videos. Afterwards, we retrain the\nmodel on the joint training set comprised of sparse manual pose annotations and our propagated\nposes. Lastly, we evaluate this trained model on the validation set.\nAll results are based on a standard HRNet [ 27] model trained on different forms of training data. \"GT\n(1x)\" refers to a model trained on sparsely labeled videos using ground-truth annotations only. \"GT\n(7x)\" baseline employs 7x more manually-annotated poses relative to \"GT (1x)\" (the annotations\nare part of the PoseTrack2017 training set). In comparison, our approach (\"GT (1x) + pGT (6x)\"),\nis trained on a joint training set consisting of sparse manual pose annotations (same as \"GT (1x)\"\nbaseline) and our propagated poses (on the training set of PoseTrack2017), which we use as pseudo\nground truth data (pGT). As before, for every labeled frame we propagate the ground truth pose to\nthe3previous and the 3subsequent frames, which allows us to expand the training set by 7times."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5172-6db4-11ee-91fc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-78", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "the3previous and the 3subsequent frames, which allows us to expand the training set by 7times.\nBased on the results in the left sub\ufb01gure of Figure 4, we can draw several conclusions. First, we\nnote that when there are very few labeled videos (i.e., 5), all three baselines perform poorly (leftmost\n\ufb01gure). This indicates that in this setting there is not enough data to learn an effective pose detection\nmodel. Second, we observe that when the number of labeled videos is somewhat reasonable (e.g.,\n50\u0000100), our approach signi\ufb01cantly outperforms the \"GT (1x)\" baseline, and is only slightly worse\nrelative to the \"GT (7x)\" baseline. As we increase the number of labeled videos, the gaps among the\nthree methods shrink, suggesting that the model becomes saturated.\n7\nTable 2: Multi-person pose estimation results on the validation and test sets of PoseTrack2017\nand PoseTrack2018 datasets. Even though our model is designed to improve pose detection in\nscenarios involving sparsely-labeled videos, here we show that our temporal pose aggregation scheme\nduring inference is also useful for models trained on densely labeled videos. We improve upon the\nstate-of-the-art single-frame baselines [23, 27, 56]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5173-6db4-11ee-80a4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-79", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "during inference is also useful for models trained on densely labeled videos. We improve upon the\nstate-of-the-art single-frame baselines [23, 27, 56].\nDataset Method Head Shoulder Elbow Wrist Hip Knee Ankle Mean\nPoseTrack17 Val SetGirdhar et al. [48] 72.8 75.6 65.3 54.3 63.5 60.9 51.8 64.1\nXiu et al. [57] 66.7 73.3 68.3 61.1 67.5 67.0 61.3 66.5\nBin et al [23] 81.7 83.4 80.0 72.4 75.3 74.8 67.1 76.7\nHRNet [27] 82.1 83.6 80.4 73.3 75.5 75.3 68.5 77.3\nMDPN [56] 85.2 88.5 83.9 77.5 79.0 77.0 71.4 80.7\nPoseWarper 81.4 88.3 83.9 78.0 82.4 80.5 73.6 81.2\nPoseTrack17 Test SetGirdhar et al. [48] - - - - - - - 59.6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5174-6db4-11ee-ad6a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-80", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "PoseTrack17 Test SetGirdhar et al. [48] - - - - - - - 59.6\nXiu et al. [57] 64.9 67.5 65.0 59.0 62.5 62.8 57.9 63.0\nBin et al [23] 80.1 80.2 76.9 71.5 72.5 72.4 65.7 74.6\nHRNet [27] 80.1 80.2 76.9 72.0 73.4 72.5 67.0 74.9\nPoseWarper 79.5 84.3 80.1 75.8 77.6 76.8 70.8 77.9\nPoseTrack18 Val SetAlphaPose [58] 63.9 78.7 77.4 71.0 73.7 73.0 69.7 71.9\nMDPN [56] 75.4 81.2 79.0 74.1 72.4 73.0 69.9 75.0\nPoseWarper 79.9 86.3 82.4 77.5 79.8 78.8 73.2 79.7\nPoseTrack18 Test SetAlphaPose++ [56, 58] - - - 66.2 - - 65.0 67.6\nMDPN [56] - - - 74.5 - - 69.0 76.4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5175-6db4-11ee-83e1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-81", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "MDPN [56] - - - 74.5 - - 69.0 76.4\nPoseWarper 78.9 84.4 80.9 76.8 75.6 77.5 71.8 78.0\nAs we vary the number of labeled frames per video (second leftmost \ufb01gure), we notice several\ninteresting patterns. First, we note that for a small number of labeled frames per video (i.e., 1\u00002) our\napproach outperforms the \"GT (1x)\" baseline by a large margin. Second, we note that the performance\nof our approach and the \"GT (7x)\" becomes very similar as we add 2or more labeled frames per\nvideo. These \ufb01ndings further strengthen our previous observation that PoseWarper allows us to reduce\nthe annotation cost without a signi\ufb01cant loss in performance.\n4.3 Improved Pose Estimation via Temporal Pose Aggregation\nIn this subsection we assess the ability of PoseWarper to improve the accuracy of pose estimation at\ntest time by using our deformable warping mechanism to aggregate pose information from nearby\nframes. We visualize our results in Figure 4 b), where we evaluate the effectiveness of our temporal\npose aggregation during inference for models trained a) with a different number of labeled videos"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5176-6db4-11ee-a75e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-82", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "frames. We visualize our results in Figure 4 b), where we evaluate the effectiveness of our temporal\npose aggregation during inference for models trained a) with a different number of labeled videos\n(second rightmost \ufb01gure), and b) with a different number of manually-labeled frames per video\n(rightmost \ufb01gure). We compare our approach (\"GT (1x) + T-Agg.\") to the same \"GT (7x)\" and \"GT\n(1x)\" baselines de\ufb01ned in the previous subsection. Note that our method in this case is trained exactly\nas \"GT (1x)\" baseline, the only difference comes from the inference procedure.\nWhen the number of training videos and/or manually labeled frames is small, our approach provides\na signi\ufb01cant accuracy boost with respect to the \"GT (1x)\" baseline. However, once, we increase the\nnumber of labeled videos/frames, the gap between all three baselines shrinks, and the model becomes\nmore saturated. Thus, our temporal pose aggregation scheme during inference is another effective\nway to maintain strong performance in a sparsely-labeled video setting.\n4.4 Comparison to State-of-the-Art\nWe also test the effectiveness of our temporal pose aggregation scheme, when the model is trained on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5177-6db4-11ee-9cfd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-83", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "4.4 Comparison to State-of-the-Art\nWe also test the effectiveness of our temporal pose aggregation scheme, when the model is trained on\nthe full PoseTrack [ 22] dataset. Table 2 compares our method to the most recent approaches in this\narea [ 48,57,23,27]. These results suggest that although we designed our method to improve pose\nestimation when training videos are sparsely-labeled, our temporal pose aggregation scheme applied\nat inference is also useful for models trained on densely-labeled videos. Our PoseWarper obtains\n81:2mAP and 77:9mAP on PoseTrack2017 validation and test sets respectively, and 79:7mAP and\n78:0mAP on PoseTrack2018 validation and test sets respectively, thus outperforming prior single\nframe baselines [48, 57, 23, 27].\n8\nFrame tFrame t+5Farneback FlowPredicted Human         MotionOffset MagnitudesChannel 99 (x,y)Channel 123 (x,y)\nFigure 5: In the \ufb01rst two columns, we show a pair of video frames used as input for our model.\nThe3rdand4thcolumns depict 2randomly selected offset channels visualized as a motion \ufb01eld.\nDifferent channels appear to capture the motion of different body parts. In the 5thcolumn, we display"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5178-6db4-11ee-af56-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-84", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Different channels appear to capture the motion of different body parts. In the 5thcolumn, we display\nthe offset magnitudes, which highlight salient human motion. Finally, the last two columns illustrate\nthe standard Farneback \ufb02ow, and the human motion predicted from our learned offsets. To predict\nhuman motion we train a linear classi\ufb01er to regress the ground-truth (x;y)displacement of each\njoint from the offset maps. The color wheel, at the bottom right corner encodes motion direction.\n4.5 Interpreting Learned Offsets\nUnderstanding what information is encoded in our learned offsets is nearly as dif\ufb01cult as analyzing\nany other CNN features [ 59,60]. The main challenge comes from the high dimensionality of offsets:\nwe are predicting c\u0002kh\u0002kw(x;y)displacements for every pixel for each of the \ufb01ve dilation\nratesd, wherecis the number of channels, and kh;kware the convolutional kernel height and width\nrespectively.\nIn columns 3;4of Figure 5, we visualize two randomly-selected offset channels as a motion \ufb01eld.\nBased on this \ufb01gure, it appears that different offset maps encode different motions rather than all"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5179-6db4-11ee-85e7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-85", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Based on this \ufb01gure, it appears that different offset maps encode different motions rather than all\npredicting the same solution (say, the optical \ufb02ow between the two frames). This makes sense, as the\nnetwork may decide to ignore motions of uninformative regions, and instead capture the motion of\ndifferent human body parts in different offset maps (say, a hand as opposed to the head). We also note\nthat the magnitudes of our learned offsets encode salient human motion (see Column 5 of Figure 5).\nLastly, to verify that our learned offsets encode human motion, for each point pndenoting a body joint,\nwe extract our predicted offsets and train a linear classi\ufb01er to regress the ground truth (x;y)motion\ndisplacement of this body joint. In Column 7 of Figure 5, we visualize our predicted motion outputs\nfor every pixel. We show Farneback\u2019s optical \ufb02ow in Column 6. Note that in regions containing\npeople, our predicted human motion matches Farneback optical \ufb02ow. Furthermore, we point out that\ncompared to the standard Farneback optical \ufb02ow, our motion \ufb01elds look less noisy.\n5 Conclusions\nIn this work, we introduced PoseWarper, a novel architecture for pose detection in sparsely labeled"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c517a-6db4-11ee-a9c8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-86", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "5 Conclusions\nIn this work, we introduced PoseWarper, a novel architecture for pose detection in sparsely labeled\nvideos. Our PoseWarper can be effectively used for multiple applications, including video pose\npropagation, and temporal pose aggregation. In these settings, we demonstrated that our approach\nreduces the need for densely labeled video data, while producing strong pose detection performance.\nFurthermore, our state-of-the-art results on PoseTrack2017 and PoseTrack2018 datasets demonstrate\nthat our PoseWarper is useful even when the training videos are densely-labeled. Our future work\ninvolves improving our model ability to propagate labels and aggregate temporal information when\nthe input frames are far away from each other. We are also interested in exploring self-supervised\nlearning objectives for our task, which may further reduce the need of pose annotations in video. We\nwill release our source code and our trained models upon publication of the paper.\nReferences\n[1]Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Deepedge: A multi-scale bifurcated deep network\nfor top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR) , June 2015.\n9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c517b-6db4-11ee-a518-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-87", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "for top-down contour detection. In The IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR) , June 2015.\n9\n[2]Saining Xie, Ross Girshick, Piotr Doll\u00e1r, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR , 2017.\n[3]Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In\nCVPR , 2014.\n[4]Gedas Bertasius, Lorenzo Torresani, Stella X. Yu, and Jianbo Shi. Convolutional random walk networks\nfor semantic image segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR) , July 2017.\n[5]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional\nnetworks for visual recognition. In Computer Vision \u2013 ECCV 2014 , 2014.\n[6]Kaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross Girshick. Mask R-CNN. In Proceedings of the\nInternational Conference on Computer Vision (ICCV) , 2017."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c517c-6db4-11ee-a2e2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-88", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "International Conference on Computer Vision (ICCV) , 2017.\n[7]Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll\u00e1r. Focal Loss for Dense Object\nDetection. In Proceedings of the International Conference on Computer Vision (ICCV) , 2017.\n[8]Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object\ndetection with region proposal networks. In Neural Information Processing Systems (NIPS) , 2015.\n[9]Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV) ,\n2015.\n[10] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate\nobject detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision\nand Pattern Recognition (CVPR) , 2014.\n[11] Saurabh Gupta, Ross Girshick, Pablo Arbelaez, and Jitendra Malik. Learning rich features from RGB-D\nimages for object detection and segmentation. In ECCV , 2014.\n[12] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c517d-6db4-11ee-a4c2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-89", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "[12] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and\nAlexander C. Berg. Ssd: Single shot multibox detector. In ECCV , 2016.\n[13] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-fcn: Object detection via region-based fully convolutional\nnetworks. In Advances in Neural Information Processing Systems 29 , pages 379\u2013387. Curran Associates,\nInc., 2016.\n[14] Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. You only look once: Uni\ufb01ed,\nreal-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR\n2016, Las Vegas, NV , USA, June 27-30, 2016 , pages 779\u2013788, 2016.\n[15] Joseph Redmon and Ali Farhadi. YOLO9000: better, faster, stronger. In 2017 IEEE Conference on\nComputer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages\n6517\u20136525, 2017."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c517e-6db4-11ee-94b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-90", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages\n6517\u20136525, 2017.\n[16] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classi\ufb01cation with deep convolutional\nneural networks. In NIPS , 2012.\n[17] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru\nErhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Computer Vision\nand Pattern Recognition (CVPR) , 2015.\n[18] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In\nICLR , 2015.\n[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.\n2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770\u2013778, 2016.\n[20] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c517f-6db4-11ee-b53e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-91", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Database. In CVPR09 , 2009.\n[21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll\u00e1r,\nand C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on\nComputer Vision (ECCV) , Z\u00fcrich, September 2014.\n[22] Umar Iqbal, Anton Milan, and Juergen Gall. Posetrack: Joint multi-person pose estimation and tracking.\nInIEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.\n10\n[23] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In\nEuropean Conference on Computer Vision (ECCV) , 2018.\n[24] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using\npart af\ufb01nity \ufb01elds. In CVPR , 2017.\n[25] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In\nComputer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5180-6db4-11ee-9317-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-92", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14,\n2016, Proceedings, Part VIII , pages 483\u2013499, 2016.\n[26] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In\nCVPR , pages 4724\u20134732. IEEE Computer Society, 2016.\n[27] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human\npose estimation. In CVPR , 2019.\n[28] J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei. Deformable convolutional networks. In 2017\nIEEE International Conference on Computer Vision (ICCV) , volume 00, pages 764\u2013773, Oct. 2017.\n[29] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.\nFlownet 2.0: Evolution of optical \ufb02ow estimation with deep networks. CoRR , abs/1612.01925, 2016.\n[30] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5181-6db4-11ee-8517-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-93", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "[30] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and\narticulated pose estimation. In CVPR , pages 1014\u20131021. IEEE Computer Society, 2009.\n[31] M. Andriluka, S. Roth, and B. Schiele. Monocular 3d pose estimation and tracking by detection. In 2010\nIEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 623\u2013630, June\n2010.\n[32] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose\nestimation. In Proc. BMVC , pages 12.1\u201311, 2010. doi:10.5244/C.24.12.\n[33] Leonid Pishchulin, Mykhaylo Andriluka, Peter V . Gehler, and Bernt Schiele. Strong appearance and\nexpressive spatial models for human pose estimation. In ICCV , pages 3487\u20133494. IEEE Computer Society,\n2013.\n[34] Yi Yang and D. Ramanan. Articulated pose estimation with \ufb02exible mixtures-of-parts. In Proceedings of\nthe 2011 IEEE Conference on Computer Vision and Pattern Recognition , CVPR \u201911, pages 1385\u20131392,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5182-6db4-11ee-8736-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-94", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "the 2011 IEEE Conference on Computer Vision and Pattern Recognition , CVPR \u201911, pages 1385\u20131392,\nWashington, DC, USA, 2011. IEEE Computer Society.\n[35] M. Dantone, J. Gall, C. Leistner, and L. van Gool. Human pose estimation using body parts dependent\njoint regressors. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pages 3041\u20133048,\nPortland, OR, USA, June 2013. IEEE.\n[36] Xiangyang Lan and D. P. Huttenlocher. Beyond trees: common-factor models for 2d human pose recovery.\nInTenth IEEE International Conference on Computer Vision (ICCV\u201905) Volume 1 , volume 1, pages\n470\u2013477 V ol. 1, Oct 2005.\n[37] L. Sigal and M. J. Black. Measure locally, reason globally: Occlusion-sensitive articulated pose estimation.\nInProc. IEEE Conf. on Computer Vision and Pattern Recognition, CVPR , volume 2, pages 2041\u20132048,\nNew York, NY , June 2006.\n[38] Yang Wang and Greg Mori. Multiple tree models for occlusion and spatial constraints in human pose\nestimation. In David Forsyth, Philip Torr, and Andrew Zisserman, editors, Computer Vision \u2013 ECCV 2008 ,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5183-6db4-11ee-b4c6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-95", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "estimation. In David Forsyth, Philip Torr, and Andrew Zisserman, editors, Computer Vision \u2013 ECCV 2008 ,\npages 710\u2013724, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg.\n[39] J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. Human pose estimation with iterative error feedback.\nIn2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4733\u20134742, June\n2016.\n[40] Xianjie Chen and Alan L Yuille. Articulated pose estimation by a graphical model with image dependent\npairwise relations. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger,\neditors, Advances in Neural Information Processing Systems 27 , pages 1736\u20131744. Curran Associates,\nInc., 2014.\n[41] W. Ouyang, X. Chu, and X. Wang. Multi-source deep learning for human pose estimation. In 2014 IEEE\nConference on Computer Vision and Pattern Recognition , pages 2337\u20132344, June 2014.\n11"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5184-6db4-11ee-b345-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-96", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Conference on Computer Vision and Pattern Recognition , pages 2337\u20132344, June 2014.\n11\n[42] Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schieke. Deepercut:\nA deeper, stronger, and faster multi-person pose estimation model. In European Conference on Computer\nVision (ECCV) , 2016.\n[43] Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter Gehler, and\nBernt Schiele. Deepcut: Joint subset partition and labeling for multi person pose estimation. In IEEE\nConference on Computer Vision and Pattern Recognition (CVPR) , 2016.\n[44] Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of a convolutional\nnetwork and a graphical model for human pose estimation. In Z. Ghahramani, M. Welling, C. Cortes, N. D.\nLawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages\n1799\u20131807. Curran Associates, Inc., 2014."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5185-6db4-11ee-90ec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-97", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages\n1799\u20131807. Curran Associates, Inc., 2014.\n[45] George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, Chris Bregler, and\nKevin Murphy. Towards accurate multi-person pose estimation in the wild. In CVPR , pages 3711\u20133719.\nIEEE Computer Society, 2017.\n[46] Vasileios Belagiannis and Andrew Zisserman. Recurrent human pose estimation. In International\nConference on Automatic Face and Gesture Recognition . IEEE, 2017.\n[47] Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres,\nand Bernt Schiele. Articulated multi-person tracking in the wild. In 2017 IEEE Conference on Computer\nVision and Pattern Recognition (CVPR) , pages 1293\u20131301. IEEE, July 2017. Oral.\n[48] Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, and Du Tran. Detect-and-Track:\nEf\ufb01cient Pose Estimation in Videos. In CVPR , 2018."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5186-6db4-11ee-947c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-98", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "Ef\ufb01cient Pose Estimation in Videos. In CVPR , 2018.\n[49] Jie Song, Limin Wang, Luc Van Gool, and Otmar Hilliges. Thin-slicing network: A deep structured model\nfor pose estimation in videos. In CVPR , pages 4420\u20134229, 2017.\n[50] J. Charles, T. P\ufb01ster, D. Magee, D. Hogg, and A. Zisserman. Personalizing human video pose estimation.\nInIEEE Conference on Computer Vision and Pattern Recognition , 2016.\n[51] T. P\ufb01ster, J. Charles, and A. Zisserman. Flowing convnets for human pose estimation in videos. In IEEE\nInternational Conference on Computer Vision , 2015.\n[52] Dingwen Zhang, Guangyu Guo, Dong Huang, and Junwei Han. Pose\ufb02ow: A deep motion representation\nfor understanding human behaviors in videos. In The IEEE Conference on Computer Vision and Pattern\nRecognition (CVPR) , June 2018.\n[53] O. Wiles, A.S. Koepke, and A. Zisserman. Self-supervised learning of a facial attribute embedding from\nvideo. In British Machine Vision Conference , 2018."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5187-6db4-11ee-8ae7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-99", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "video. In British Machine Vision Conference , 2018.\n[54] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015.\n[55] Gunnar Farneb\u00e4ck. Two-frame motion estimation based on polynomial expansion. In Proceedings of the\n13th Scandinavian Conference on Image Analysis , SCIA\u201903, pages 363\u2013370, Berlin, Heidelberg, 2003.\nSpringer-Verlag.\n[56] Hengkai Guo, Tang Tang, Guozhong Luo, Riwei Chen, Yongchen Lu, and Linfu Wen. Multi-domain pose\nnetwork for multi-person pose estimation and tracking. In Computer Vision - ECCV 2018 Workshops Munich, Germany, September 8-14, 2018, Proceedings, Part II , pages 209\u2013216, 2018.\n[57] Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu. Pose Flow: Ef\ufb01cient online pose\ntracking. In BMVC , 2018.\n[58] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi-person pose estimation. In\nThe IEEE International Conference on Computer Vision (ICCV) , Oct 2017."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5188-6db4-11ee-a846-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1906.04016-100", "title": "Learning Temporal Pose Estimation from Sparsely-Labeled Videos", "source": "http://arxiv.org/pdf/1906.04016", "start_index": 0, "chroma:document": "The IEEE International Conference on Computer Vision (ICCV) , Oct 2017.\n[59] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. CoRR ,\nabs/1311.2901, 2013.\n[60] Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural\nnetworks through deep visualization. CoRR , abs/1506.06579, 2015.\n12"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5189-6db4-11ee-9235-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-0", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Classifying, Segmenting, and Tracking\nObject Instances in Video with Mask Propagation\nGedas Bertasius, Lorenzo Torresani\nFacebook AI\nAbstract\nWe introduce a method for simultaneously classifying,\nsegmenting and tracking object instances in a video sequence. Our method, named MaskProp, adapts the popular Mask R-CNN to video by adding a mask propagation\nbranch that propagates frame-level object instance masks\nfrom each video frame to all the other frames in a video clip.\nThis allows our system to predict clip-level instance tracks\nwith respect to the object instances segmented in the middle frame of the clip. Clip-level instance tracks generated\ndensely for each frame in the sequence are \ufb01nally aggregated to produce video-level object instance segmentation\nand classi\ufb01cation. Our experiments demonstrate that our\nclip-level instance segmentation makes our approach robust\nto motion blur and object occlusions in video. MaskProp\nachieves the best reported accuracy on the YouTube-VIS\ndataset, outperforming the ICCV 2019 video instance segmentation challenge winner despite being much simpler and\nusing orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).\n1. Introduction"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c518a-6db4-11ee-8d37-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-1", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "using orders of magnitude less labeled data (1.3M vs 1B images and 860K vs 14M bounding boxes).\n1. Introduction\nIn this paper, we tackle the recently introduced video instance segmentation problem [42]. This task requires segmenting all instances of a prede\ufb01ned set of object classes\nin each frame, classifying them, and linking individual instances over the entire sequence.\nIn recent years, convolutional networks have obtained remarkable results in still-image object detection [16, 33, 14,\n15], and segmentation [27, 45, 8, 4]. However, extending\nthese models to video instance segmentation is challenging. In order to localize objects precisely, these methods\nhave to operate at very large spatial resolution. As a result,\ndetectors based on the popular ResNet-101 or ResNet-152\nbackbones [17] can rarely \ufb01t more than one image per GPU\nduring training. In the context of video instance segmentation this is problematic because tracking objects over time\nrequires analyzing multiple video frames simultaneously.\nTo address this issue, one could reduce the spatial resFigure 1: In this paper, we tackle the problem of video instance segmentation, which requires classifying, segmenting, and tracking object instances in a given video sequence.\nOur proposed Mask Propagation framework (MaskProp)\nprovides a simple and effective way for solving this task."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c518b-6db4-11ee-876a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-2", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Our proposed Mask Propagation framework (MaskProp)\nprovides a simple and effective way for solving this task.\nolution of the input and \ufb01t more video frames in a GPU.\nHowever, doing so typically leads to a signi\ufb01cant drop in\nsegmentation or detection performance. Alternatively, one\ncould perform high-resolution instance segmentation on individual frames and then link segmentations temporally in\na separate post-processing stage. However, performing instance segmentation and tracking in two disjoint steps often produces suboptimal results, because these two tasks\nare closely intertwined. The key challenge then becomes\ndesigning a uni\ufb01ed model that can track objects in video\nwhile maintaining strong detection accuracy.\nCurrently, the best method for video instance segmentation is the ICCV 2019 challenge winner [28]. It tackles\nvideo instance segmentation by dividing it into four problems: 1) detection, 2) classi\ufb01cation, 3) segmentation, and\n4) tracking. These four problems are solved independently\nusing several off-the-shelf components and their respective\nsolutions are combined and adapted to the video instance\nsegmentation task. However, despite effective performance,\nsuch an approach is disadvantageous because it requires designing and tuning a separate model (or, in some cases, an"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c518c-6db4-11ee-ba5d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-3", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "segmentation task. However, despite effective performance,\nsuch an approach is disadvantageous because it requires designing and tuning a separate model (or, in some cases, an\nensemble of models) for each of the four tasks. This renders the approach costly and cumbersome. On the other\nend of the complexity spectrum, MaskTrack R-CNN [42] is\na simple uni\ufb01ed approach trained end-to-end but it achieves\nsigni\ufb01cantly lower performance ( 30:3vs44:8video mAP).\nTo address the shortcomings of these prior methods wearXiv:1912.04573v4  [cs.CV]  9 Jul 2021\nMaskTrack R-CNN [42] ICCV19 Challenge Winner [28] MaskProp\nModelClassi\ufb01cation cls head Mask R-CNN [16], ResNeXt-101 32x48d [30] cls head\nLocalization bbox head Mask R-CNN [16] bbox head\nSegmentation mask head DeepLabv3 [9], Box2Seg [29] mask head\nTracking tracking head UnOVOST [47], ReID Net [18, 31] mask propagation head\nOptical Flow - PWC-Net [35] Pre-training DatasetsImageNet [34] (1.3M images) X X X"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c518d-6db4-11ee-84c1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-4", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Optical Flow - PWC-Net [35] Pre-training DatasetsImageNet [34] (1.3M images) X X X\nCOCO [25] (860K bboxes) X X X\nInstagram [30] (1B images) - X OpenImages [23] (14M bboxes) - X Performancevideo mAP 30.3 44.8 46.6\nvideo AP@75 32.6 48.9 51.2\nTable 1: A table comparing our work to prior video instance segmentation methods [42, 28]. The ICCV 2019 Challenge\nWinner [28] decomposes video instance segmentation into four different problems, solves each of them independently using\nensembles of different models, and then combines these solutions. In contrast, our approach relies on a single uni\ufb01ed model\ntrained end-to-end. Despite being simpler, and using several orders of magnitude less pretraining data (1.3M vs 1B images\nand 860K vs 14M bounding boxes) our model achieves higher accuracy. Furthermore, compared to MaskTrack R-CNN [42],\nour work yields a 16.3% gain in mAP (46.6% vs 30.3%).\nintroduce MaskProp, a simple mask propagation framework\nfor simultaneously classifying, segmenting and tracking object instances in video. Our method adapts the popular"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c518e-6db4-11ee-8e9a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-5", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "introduce MaskProp, a simple mask propagation framework\nfor simultaneously classifying, segmenting and tracking object instances in video. Our method adapts the popular\nMask R-CNN [16] to video by adding a branch that propagates frame-level instance masks from each video frame\nto other frames within a temporal neighborhood (which we\nrefer to as a clip). This allows our method to compute cliplevel instance tracks centered at each individual frame of the\nvideo. These densely estimated clip-level tracks are then\naggregated to form accurate and coherent object instance\nsequences for the entire video, regardless of its length. This\nrenders our approach capable of handling challenging cases\nof occlusions, disocclusions, and motion blur. Our method\nachieves the best reported accuracy on the YouTube-VIS\ndataset [42], outperforming the ICCV 2019 challenge winner [28] despite being much simpler and using signi\ufb01cantly\nless labeled data (1000x fewer images and 10x fewer bounding boxes). In Table 1, we compare our approach vs these\nprior methods in terms of accuracy and other characteristics.\n2. Related Work\nInstance Segmentation in Images. Compared to instance\nsegmentation in images [11, 16, 44, 1, 2, 22, 26], the problem considered in this paper requires not only to segment\nobject instances in individual frames, but also to determine"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c518f-6db4-11ee-b777-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-6", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "object instances in individual frames, but also to determine\ninstance correspondences across multiple frames. We leverage the Mask R-CNN model [16] for still-image instance\nsegmentation and adapt it to track object instances in video.\nObject Detection in Video. Object detection in video requires classifying and localizing objects in every frame of\na given video. Most modern video object detection systems [46, 5, 40, 13] implement some form of spatiotemporal feature alignment for improving object detection accuracy in individual video frames. However, these systems\nare typically not designed for tracking object instances. In\ncontrast, our mask propagation produces clip-level instancesegmentations rather than frame-level bounding boxes.\nVideo Object Segmentation. The task of video object\nsegmentation requires segmenting foreground objects in a\nclass-agnostic fashion [43, 21, 36, 38], often by leveraging\nground truth masks available for the \ufb01rst frame during inference [6, 32, 10, 19, 37]. Instead, video instance segmentation requires \ufb01nding all instances of a prede\ufb01ned set of\nobject classes in each frame, classifying them and linking\nthem over the entire sequence.\nVideo Instance Segmentation. The recently introduced\nvideo instance segmentation task [42] requires classifying,\nsegmenting and tracking object instances in videos. This"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5190-6db4-11ee-a03f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-7", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Video Instance Segmentation. The recently introduced\nvideo instance segmentation task [42] requires classifying,\nsegmenting and tracking object instances in videos. This\nis the task considered in this work. There are only a few\nvideo instance segmentation methods we can compare our\napproach to. The MaskTrack R-CNN [42] presents a uni\ufb01ed model for video instance segmentation. It augments the\noriginal Mask R-CNN [16] with a tracking branch that establishes associations among object instances segmented in\nseparate frames. Furthermore, we include the ICCV 2019\nvideo instance segmentation challenge winner [28] in our\ncomparison. This approach divides video instance segmentation into four separate subproblems: classi\ufb01cation, detection, segmentation, and tracking. A separate model (or\nan ensemble of models) is used to solve each of these subproblems, and these solutions are then combined to produce\nvideo instance segmentation results. For brevity, from now\non we refer to it as EnsembleVIS to indicate that it is an ensemble approach designed for video instance segmentation.\nOur MaskProp framework provides advantages over\nboth of these methods [42, 28]. Similarly to MaskTrack\nR-CNN [42], our method is a uni\ufb01ed and simple approach."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5191-6db4-11ee-9f51-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-8", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "both of these methods [42, 28]. Similarly to MaskTrack\nR-CNN [42], our method is a uni\ufb01ed and simple approach.\nHowever, our mask propagation branch is much more effective than the tracking branch of MaskTrack R-CNN, achieving much higher accuracy relative to this baseline. Furthermore, compared to EnsembleVIS [28], our method 1)\nis much simpler, 2) uses signi\ufb01cantly less labeled data, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5192-6db4-11ee-a49a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-9", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "clip-level     instance tracktemporally propagated instance featuresfeature tensorinstance feature computation at time t    frame-level    instance mask time ttime t\u2326<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5193-6db4-11ee-ba31-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5194-6db4-11ee-af85-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5195-6db4-11ee-9580-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5196-6db4-11ee-9f02-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5197-6db4-11ee-b639-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5198-6db4-11ee-bef9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c5199-6db4-11ee-b0ca-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c784c-6db4-11ee-8e02-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c784d-6db4-11ee-947c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-18", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "t+5time t-5time t-10"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c784e-6db4-11ee-a09b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-19", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "time ttime t+10time t-5time t+5time t+10time ttime t-10propagated instance        segmentationfeature tensors for each frame             in a video clip\ntime t-5time t+5time t+10time ttime t-10\nFigure 2: An illustration of our MaskProp system, which takes as input a video clip centered around frame t, and outputs\na clip-level instance track. Our mask propagation framework can be summarized in three high-level steps: 1) An instancespeci\ufb01c feature map at time tis computed by masking the frame features at time twith the given instance segmentation\nfor framet(one for each instance detected in frame t). 2) Next, we use our mask propagation mechanism to temporally\npropagate instance-speci\ufb01c features from frame tto all the other frames in the clip. 3) Lastly, our model predicts instancespeci\ufb01c segmentations in every frame of the clip by implicitly matching the propagated instance features with the frame-level\nfeatures computed at each time step. This last step yields clip-level instance tracks centered around frame t.\n3) produces higher accuracy on YouTube-VIS [42].\n3. Video Instance Segmentation\nProblem De\ufb01nition. Let us denote with V2RL\u00023\u0002H\u0002W"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c784f-6db4-11ee-87d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-20", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "3. Video Instance Segmentation\nProblem De\ufb01nition. Let us denote with V2RL\u00023\u0002H\u0002W\nan input video consisting of LRGB frames of spatial size\nH\u0002W. The aim of our system is to segment and temporally\nlink all object instances that are visible for at least one frame\ninVand that belong to a prede\ufb01ned set of categories C=\nf1;:::;Kg. To achieve this goal, our model outputs a videolevel instance mask track Mi2RL\u0002H\u0002Wwith a category\nlabelci2f1;:::;Kgand a con\ufb01dence score si2[0;1]for\neach object instance idetected in the video.\nEvaluation Metric. Video instance segmentation is evaluated according to the metrics of average precision (AP) and\naverage recall (AR). Unlike in the image domain, these metrics are evaluated over the video sequence. Thus, to evaluate spatiotemporal consistency of the predicted mask sequences, the video Intersection over Union (IoU) between\na predicted object instance iand a ground truth object instancejis computed as:\nIoU(i;j) =PT\nt=1jMi(t)\\~Mj(t)j\nPT"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7850-6db4-11ee-aacc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-21", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "IoU(i;j) =PT\nt=1jMi(t)\\~Mj(t)j\nPT\nt=1jMi(t)[~Mj(t)j(1)\nwhere ~Mj(t)is the ground-truth segmentation of object j\nin framet. To achieve a large IoU, a model must not only\naccurately classify and segment object instances at a framelevel, but also reliably track them over the video sequence.\nAs in the COCO benchmark for image segmentation [25], the metrics of AP and AR are computed separatelyfor each object category, and then averaged over ten IoU\nthresholds from 50% to95% at increments of 5%. Lastly,\nthe resulting AP and AR metrics are averaged over the category set, which yields the \ufb01nal evaluation metric.\n4. Mask Propagation\nMaskProp takes a video Vof arbitrary length Las input and outputs video-level instance segmentation tracks\nMi, category labels ciand con\ufb01dence scores sifor all\nobjectsidetected in the video. In order to achieve this\ngoal, our method \ufb01rst builds clip-level object instance tracks\nMi\nt\u0000T:t+T2 R(2T+1)\u00021\u0002H\u0002Wfor each individual clip"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7851-6db4-11ee-a072-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-22", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Mi\nt\u0000T:t+T2 R(2T+1)\u00021\u0002H\u0002Wfor each individual clip\nVt\u0000T:t+T2R(2T+1)\u00023\u0002H\u0002Wof length (2T+ 1) in the\nvideo, i.e., for t= 1;2;:::;L (clips at the beginning and\nthe end of the video will include fewer frames).\nWe want to use clips that are long enough to allow us\nto jointly solve instance segmentation and tracking while\nhandling challenging cases of occlusion and motion blur.\nAt the same time, the clip should be short enough to allow\nus to \ufb01t it at high spatial resolution in the memory of a GPU.\nThe resulting clip-level instance masks Mi\nt\u0000T:t+Tproduced densely for all overlapping clips t= 1;:::;L are\nthen aggregated to produce video-level instance masks Mi.\nOur approach for clip-level instance segmentation is described in subsections 4.1 and 4.2. We also illustrate it in\nFigure 2. The subsequent clip-level instance mask aggregation method is presented in subsection 4.3.\nFrame t\nBbox...\nClsMask\nFrame t+...     Mask  Propagation     CNN"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7852-6db4-11ee-8829-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-23", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "CNN\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7853-6db4-11ee-9aa4-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7854-6db4-11ee-a9fa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7855-6db4-11ee-8741-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7856-6db4-11ee-bdb7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7857-6db4-11ee-a777-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7858-6db4-11ee-940e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7859-6db4-11ee-a04e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c785a-6db4-11ee-b636-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c785b-6db4-11ee-b83c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-32", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Figure 3: We adapt Mask R-CNN [16] to video by adding\na mask propagation branch, for tracking object segmentation instances in video. Given a video clip centered around\nframet, our system outputs a clip-level instance segmentation track as well as a classi\ufb01cation score and a bounding\nbox for each object instanced detected in frame t. For compactness, in this \ufb01gure, we illustrate our system processing a\npair of frames but typically the propagation is applied from\nthe middle frame to all the other frames in the clip.\n4.1. Video Mask R-CNN\nOur video instance segmentation system is based on the\nMask R-CNN [16] model, which we adapt to video by\nadding a mask propagation branch (See Figure 3). We train\nour system with a multi-task loss Lt=Lcls\nt+Lbox\nt+\nLmask\nt +Lprop\nt wheretdenotes a time-step of a center\nframe. We use identical loss terms Lcls\nt;Lbox\nt,Lmask\nt as\nin Mask R-CNN. The mask propagation loss is de\ufb01ned as:\nLprop\nt =~NtX\nit+TX\nt0=t\u0000T1\u0000sIoU (Mi\nt\u0000T:t+T(t0);~Mi"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c785c-6db4-11ee-b9ac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-33", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "t =~NtX\nit+TX\nt0=t\u0000T1\u0000sIoU (Mi\nt\u0000T:t+T(t0);~Mi\nt\u0000T:t+T(t0))(2)\nwhereMi\nt\u0000T:t+T(t0)2[0;1]is the segmentation at time\nt0for an instance ipredicted from a clip centered at tand\n~Mi\nt\u0000T:t+T(t0)is the corresponding ground truth mask at\ntimet0.~Ntis the number of ground truth object instances\nin framet, andsIoU is de\ufb01ned as:\nsIoU (A;B) =P\npA(p)B(p)P\npA(p) +B(p)\u0000A(p)B(p)(3)\nwhere the summations in numerator and denominator are\nperformed over every pixel location p. The loss above is a\nsoft IoU loss, which we observed to work slightly better\nthan the standard cross entropy loss for our task.\n4.2. Mask Propagation Branch\nOverview. Our main technical contribution is the design of\na mask propagation branch, that allows our method to track"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c785d-6db4-11ee-8b92-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-34", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "4.2. Mask Propagation Branch\nOverview. Our main technical contribution is the design of\na mask propagation branch, that allows our method to track\nobject instances. Given a video clip Vt\u0000T:t+Tcenteredat framet, our system outputs clip-level instance masks\nMi\nt\u0000T:t+Tfor each predicted object instance iin framet.\nOur mask propagation branch can be described in three\nhigh-level steps: 1) instance-speci\ufb01c feature computation,\n2) temporal propagation of instance features, and 3) propagated instance segmentation. We will now describe each of\nthese steps in more detail. We introduce our mask propagation with the example of propagating object instance masks\nfrom frametto framet+\u000ewhere\u000e2[\u0000T:T].\nComputing Instance Speci\ufb01c Features. The mask\nbranch of our model predicts frame-level instance masks\nMi\nt2R1\u0002H0\u0002W0from single frame inputs. We then\nuse these frame-level instance masks to compute instancespeci\ufb01c features for frame t. Speci\ufb01cally, for each object instancei, we compute an element-wise product between Mi\nt\nand the feature tensor from the backbone network ft. This\nthen yields a set of new feature tensors fi"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c785e-6db4-11ee-8feb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-35", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "t\nand the feature tensor from the backbone network ft. This\nthen yields a set of new feature tensors fi\nt2RC\u0002H0\u0002W0,\nwherei= 1;::;Nt, andNtis the number of object instances\ndetected in frame t. In other words, for each object instance\ni, we are zeroing out the feature values in ftthat correspond\nto pixels not belonging to that object instance.\nTemporally Propagating Instance Features. Given\nframe-level features ft;ft+\u000e2RC\u0002H0\u0002W0and instancespeci\ufb01c feature tensor fi\nt, our method generates a propagated instance feature tensor gi\nt;t+\u000e. Intuitively, gi\nt;t+\u000erepresents the features predicted by our model for object instanceiin framet+\u000efrom an instance-speci\ufb01c feature tensorfi\nt. The tensor gi\nt;t+\u000eis generated by warping features\nfi\ntusing the alignment computed from frame-level features\nftandft+\u000e. We implement the propagation mechanism via\na deformable convolution [12], which has previously been\nused for aligning features computed from separate frames"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c785f-6db4-11ee-9b25-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-36", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "ftandft+\u000e. We implement the propagation mechanism via\na deformable convolution [12], which has previously been\nused for aligning features computed from separate frames\nof a video [5, 3]. Speci\ufb01cally, we compute the elementwise difference of tensors ft;ft+\u000eand feed it through a\nsimple residual block [17], which predicts motion offsets\not;t+\u000e2R2k2\u0002H0\u0002W0. These offsets contain (x;y)sampling locations for each entry of a k\u0002kdeformable convolution kernel [12]. The propagation step takes as inputs 1)\nthe offsetsot;t+\u000eand 2) the instance feature tensor fi\nt, and\nthen applies deformable convolution to output the propagated instance feature tensor gi\nt;t+\u000efor each instance i. We\nuse subscript t;t+\u000eto denote the propagated instance feature because, although gis obtained by propagating the feature tensorfi\nt, the offset computation uses both frame tand\nframet+\u000e. We stress that no explicit ground truth alignment is available between frames. The deformable convolutional kernels are supervised implicitly by optimizing Eq. 2.\nSegmenting Propagated Instances. Lastly, we use our\npropagated feature map gi"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7860-6db4-11ee-b6e5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-37", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Segmenting Propagated Instances. Lastly, we use our\npropagated feature map gi\nt;t+\u000efor predicting a corresponding object instance mask in frame t+\u000e. To do this we\n\ufb01rst, construct a new feature tensor \u001ei\nt;t+\u000e=gi\nt;t+\u000e+ft+\u000e.\nThe addition effectively overimposes the tensor gi\nt;t+\u000epredicted from time tfor object instance iin framet+\u000e, with\nTensor t"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7861-6db4-11ee-bc05-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-38", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7862-6db4-11ee-953a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7863-6db4-11ee-9001-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7864-6db4-11ee-a5e8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7865-6db4-11ee-a245-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7866-6db4-11ee-9742-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7867-6db4-11ee-9502-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7868-6db4-11ee-9cc5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7869-6db4-11ee-8ef5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c786a-6db4-11ee-8164-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-47", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "t+  \u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c786b-6db4-11ee-99fb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c786c-6db4-11ee-b98f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c786d-6db4-11ee-a485-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c786e-6db4-11ee-b9c3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c786f-6db4-11ee-80c1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7870-6db4-11ee-9798-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7871-6db4-11ee-a74f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7872-6db4-11ee-b70f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7873-6db4-11ee-85e9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-56", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Residual      Block"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7874-6db4-11ee-b962-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-57", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "\u2326<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7875-6db4-11ee-8336-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7876-6db4-11ee-8af9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7877-6db4-11ee-8248-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7878-6db4-11ee-b2d6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7879-6db4-11ee-a33b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c787a-6db4-11ee-a363-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c787b-6db4-11ee-a03e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c787c-6db4-11ee-ae27-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-65", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit>\u2326<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c787d-6db4-11ee-b7bd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c787e-6db4-11ee-a6c6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c787f-6db4-11ee-b543-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7880-6db4-11ee-9beb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7881-6db4-11ee-ac9f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7882-6db4-11ee-8176-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7883-6db4-11ee-9b55-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7884-6db4-11ee-8fdc-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7885-6db4-11ee-9141-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-74", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Deform. Conv. Offsets\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7886-6db4-11ee-bb84-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7887-6db4-11ee-bbd7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7888-6db4-11ee-8f9f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7889-6db4-11ee-930a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c788a-6db4-11ee-8681-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c788b-6db4-11ee-9048-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c788c-6db4-11ee-b26e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c788d-6db4-11ee-85e2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-82", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "5W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c788e-6db4-11ee-84b9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c788f-6db4-11ee-a17f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7890-6db4-11ee-9521-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7891-6db4-11ee-a327-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7892-6db4-11ee-94b2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7893-6db4-11ee-8a53-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7894-6db4-11ee-9cea-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7895-6db4-11ee-9af7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7896-6db4-11ee-b1c4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-91", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "2D Conv."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7897-6db4-11ee-9ece-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-92", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Softmax   Attention\u2326<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7898-6db4-11ee-8983-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c7899-6db4-11ee-b7bf-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c789a-6db4-11ee-a05f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c789b-6db4-11ee-904e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c789c-6db4-11ee-8545-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c789d-6db4-11ee-89ff-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c789e-6db4-11ee-aa2e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c789f-6db4-11ee-a0cd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-100", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit>\u2326<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a0-6db4-11ee-a8cb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a1-6db4-11ee-94aa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a2-6db4-11ee-a4e2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a3-6db4-11ee-8c08-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a4-6db4-11ee-89cb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a5-6db4-11ee-80f7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a6-6db4-11ee-a6a7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a7-6db4-11ee-a70f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a8-6db4-11ee-9e7c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-109", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Propagated Instance Segmentation (time t+  )\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78a9-6db4-11ee-98e5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78aa-6db4-11ee-a059-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78ab-6db4-11ee-8cd9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78ac-6db4-11ee-9e7c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78ad-6db4-11ee-943e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78ae-6db4-11ee-a46d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78af-6db4-11ee-b599-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b0-6db4-11ee-a627-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b1-6db4-11ee-921b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-118", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "2) Instance Feature Propagation (from time t to time t+  )\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b2-6db4-11ee-beaa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b3-6db4-11ee-a661-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b4-6db4-11ee-8667-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b5-6db4-11ee-ba46-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b6-6db4-11ee-8d80-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b7-6db4-11ee-8fda-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b8-6db4-11ee-b818-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78b9-6db4-11ee-b8d8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78ba-6db4-11ee-befb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-127", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Instance Feature Computation (time t)Instancej"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78bb-6db4-11ee-a6e0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-128", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78bc-6db4-11ee-b1fc-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78bd-6db4-11ee-bec0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78be-6db4-11ee-94f9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78bf-6db4-11ee-9934-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c0-6db4-11ee-a4f5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c1-6db4-11ee-9267-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c2-6db4-11ee-ab31-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c3-6db4-11ee-82e9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-136", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "oOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit>Instancei<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c4-6db4-11ee-a355-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c5-6db4-11ee-b7ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-138", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c6-6db4-11ee-bcba-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c7-6db4-11ee-b87b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-140", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c8-6db4-11ee-b1f2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78c9-6db4-11ee-b44b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-142", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78ca-6db4-11ee-a030-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78cb-6db4-11ee-9580-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-144", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit>Instancej"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78cc-6db4-11ee-a2ae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-145", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78cd-6db4-11ee-bafe-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78ce-6db4-11ee-8de3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78cf-6db4-11ee-94c0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d0-6db4-11ee-9cc7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d1-6db4-11ee-b999-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d2-6db4-11ee-adfb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d3-6db4-11ee-961f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d4-6db4-11ee-9313-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-153", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "oOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit>Instancei<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d5-6db4-11ee-b53e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d6-6db4-11ee-a9d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-155", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d7-6db4-11ee-8c04-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d8-6db4-11ee-9be1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-157", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78d9-6db4-11ee-942b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78da-6db4-11ee-aa7f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-159", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78db-6db4-11ee-8ca1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78dc-6db4-11ee-99dd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-161", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit>Figure"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78dd-6db4-11ee-8673-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-162", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "4: An illustration of the 3 steps implemented by our mask propagation branch: 1)For every detected instance in frame"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78de-6db4-11ee-a27e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-163", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "t, we compute an instance-speci\ufb01c feature tensor via element-wise multiplication between tensor tand the given frame-level\ninstance mask at frame t.2)Next, the element-wise difference of the feature tensors associated with frames tandt+\u000eis\nused to predict motion offsets between frames tandt+\u000e. The resulting offsets are used to propagate the instance-speci\ufb01c\ntensors from time tto timet+\u000evia deformable convolution. The propagated tensors represent instance-speci\ufb01c features\npredicted for time t+\u000eusing the tensors computed at time t.3)Lastly, we add the propagated instance feature tensors to the\ntensor effectively computed at t+\u000e. A convolutional layer applied to these tensors predicts instance masks in frame t+\u000e.\nThe pixels that do not belong to any object instances are zeroed out using an instance-agnostic attention map.\nthe tensorft+\u000eeffectively computed from the frame at time\nt+\u000e. If the object instance prediction is consistent with the\nfeature computation, the feature tensors will be aligned and\nthus, they will reinforce each other in the predicted region.\nFinally, the resulting feature tensors \u001ei\nt;t+\u000eare fed into\na1\u00021convolution layer that outputs instance masks for"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78df-6db4-11ee-8d62-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-164", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Finally, the resulting feature tensors \u001ei\nt;t+\u000eare fed into\na1\u00021convolution layer that outputs instance masks for\neach object instance iin framet+\u000e. The masks are normalized with the softmax nonlinearity across all Ntinstances. To zero-out pixels that do not belong to any object\ninstance, we use a single 3\u00023convolution that computes\nan instance-agnostic attention map At+\u000efrom feature tensorft+\u000e. We then multiply At+\u000ewith each of our predicted\ninstance masks. A detailed illustration of our mask propagation branch is presented in Figure 4.\n4.3. Video-Level Segmentation Instances\nGiven a video of length L, our goal is to produce videolevel segmentation instances Mi2 RL\u0002H\u0002W. Conceptually, this requires linking clip-level instance tracks\nMi\nt\u0000T:t+TandMj\nt0\u0000T:t0+Twheniandjrepresent the same\nobject instance, i.e., when the instances are matching. We\nachieve this by assigning a video-level instance ID to each\nof our predicted clip-level instance tracks. Matching instance tracks are assigned the same video-level instance ID.\nMatching Clip-Level Instance Tracks. Consider a pair of\nclip-level instance tracks Mi"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78e0-6db4-11ee-bd4c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-165", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Matching Clip-Level Instance Tracks. Consider a pair of\nclip-level instance tracks Mi\nt\u0000T:t+TandMj\nt0\u0000T:t0+T, that\nare centered around frames tandt0respectively. They overlap in time ifjt\u0000t0j<2T+ 1. Let us denote their overlapping time interval as \\t;t0. Given two overlapping in-stance tracks, we can check if they match by comparing\ntheir predicted instance masks in the overlapping frames.\nIf the masks of the two instance tracks align well, then they\nare likely to encode the same object instance. Otherwise,\neach track represents a different object instance. We compute a matching score mi;j\nt;t0between two clip-level instance\ntracks using our previously de\ufb01ned soft IoU metric as:\nmi;j\nt;t0=1\nj\\t;t0jX\n~t2\\t;t0sIoU (Mi\nt\u0000T:t+T(~t); Mj\nt0\u0000T:t0+T(~t))(4)\nVideo-Level Instance ID Assignment. We denote withY\nthe set of video-level instance IDs. The set Yis built incrementally by matching clip-level instance tracks Mi\nt\u0000T:t+T"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78e1-6db4-11ee-9c51-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-166", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "the set of video-level instance IDs. The set Yis built incrementally by matching clip-level instance tracks Mi\nt\u0000T:t+T\nin order from time t= 1 to timet=L. Initially, we set\nY=f1;2;:::;N 1gwhereN1is the number of object instances detected at time t= 1 (i.e., in the \ufb01rst clip of the\nvideo). Let yi\nt2Y denote the video-level instance ID assigned to clip-level instance track Mi\nt\u0000T:t+T. As we move\nforward in time t>1, the video-level instance ID yi\ntis assigned by matching clip-level instance Mi\nt\u0000T:t+Tto all previously processed instance tracks Mj\nt0\u0000T:t0+Tthat overlap\nwith this clip, i.e., such that \\t;t06=;. For each video-level\ninstance IDy2Y already in the ID set, we compute a score\nqi\nt(y)capturing how well Mi\nt\u0000T:t+Tmatches the tracks that\nhave been already assigned video-level ID y:\nqi\nt(y) =P\nt0s.t.\\t;t06=;PNt0\nj=11fyj"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78e2-6db4-11ee-8dd9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-167", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "qi\nt(y) =P\nt0s.t.\\t;t06=;PNt0\nj=11fyj\nt0=yg\u0001mi;j\nt;t0\nP\nt0s.t.\\t;t06=;PNt0\nj=11fyj\nt0=yg(5)\ntime ttime t+5time t+10time t+15time t-15time t-10time t-5\nFigure 5: An illustration of instance-speci\ufb01c features propagated from frame tto other frames in the given video clip. Here,\nwe visualize propagated activations from one randomly selected feature channel. The activations in the two rows correspond\nto two different object instances detected at time t. Our visualizations suggest that MaskProp reliably propagates features\nthat are speci\ufb01c to each instance even when instances appear next to each other, and despite the changes in shape, pose and\nthe nuisances effects of deformation and occlusion.\nwhere 1fyj\nt0=ygis an indicator function that is equal\nto1whenyj\nt0=y, and 0otherwise.Nt0is the number\nof detected instances at time t0, andmi;j\nt;t0is the previously\nde\ufb01ned matching score between instance tracks Mi"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c78e3-6db4-11ee-9a00-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-168", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "of detected instances at time t0, andmi;j\nt;t0is the previously\nde\ufb01ned matching score between instance tracks Mi\nt\u0000T:t+T\nandMj\nt0\u0000T:t0+T. The score qi\nt(y)is effectively an average\nof the soft IoU computed between instance track Mi\nt\u0000T:t+T,\nand all its overlapping instance tracks that have been assigned video-level instance ID y.\nLetq\u0003= maxy2Yqi\nt(y)be the maximum score obtained\nby considering all possible video-level instance IDs y2Y.\nIfq\u0003is greater than a certain threshold, the current instance track Mi\nt\u0000T:t+Tis assigned the video-level instance\nIDy\u0003= arg maxy2Yqi\nt(y). Otherwise, the clip-level track\ndoes not match any of the current video-level instances. In\nsuch case we create a new video-level instance ID and assign it to the the clip-level track while also expanding the\nsetY, i.e.,yi\nt=jYj+ 1andY=Y[fjYj + 1g.\nFinally, for each video-level instance ID y2 Y , we\ngenerate the \ufb01nal sequence of segmentation instance masks"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f65-6db4-11ee-8bea-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-169", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Finally, for each video-level instance ID y2 Y , we\ngenerate the \ufb01nal sequence of segmentation instance masks\nMy2RL\u0002H\u0002Was:\nMy(t) =(\nMi\nt\u0000T:t+T(t)ifyi\nt=y\n0 otherwise .4.4. Implementation Details\nBackbone Network. As our backbone we use a Spatiotemporal Sampling Network [5] based on a Deformable\nResNeXt-101-64x4d [12, 41] with a feature pyramid network (FPN) [24] attached on top of it.\nDetection Network. For detection, we use a Hybrid Task\nCascade Network [7] with a 3stage cascade.\nMask Propagation Branch. The residual block in the\nmask propagation branch consists of two 3\u00023convolutions with 128output channels each. The instance feature\npropagation is applied to the FPN feature map with the second largest spatial resolution. To propagate instance features from one frame to another, we predict 9 (x;y)offsets\nfor every pixel, which are then used as input to a 3\u00023deformable convolutional layer with 256channels. To capture\nmotion at different scales, we use three levels of dilated deformable convolutions with dilation rates 3;6;12as in [3]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f66-6db4-11ee-ad49-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-170", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "motion at different scales, we use three levels of dilated deformable convolutions with dilation rates 3;6;12as in [3].\nHigh-Resolution Mask Re\ufb01nement. Predicting masks\nfrom RoI features typically leads to low resolution predictions. We address this issue via a high-resolution mask re\ufb01nement step. Given the center of a detected bounding box,\nwe crop a 384\u0002384patch around the object, preserving\nthe original aspect ratio. We then feed the RGB patch and\nthe predicted low-resolution mask through 3residual blocks\neach with 128channels to obtain a high-resolution mask.\nMethod Pre-training Data mAP AP@75 AR@1 AR@10\nDeepSORTz[39] Imagenet [34], COCO [25] 26.1 26.1 27.8 31.3\nFEELVOSz[37] Imagenet [34], COCO [25] 26.9 29.7 29.9 33.4\nOSMNz[43] Imagenet [34], COCO [25] 27.5 29.1 28.6 33.1\nMaskTrack R-CNNz[42] Imagenet [34], COCO [25] 30.3 32.6 31.0 35.5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f67-6db4-11ee-841b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-171", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "MaskTrack R-CNNz[42] Imagenet [34], COCO [25] 30.3 32.6 31.0 35.5\nMaskTrack R-CNN\u0003Imagenet [34], COCO [25] 36.9 40.2 34.3 42.9\nEnsembleVIS [28] Imagenet [34], COCO [25], Instagram [30], OpenImages [23] 44.8 48.9 42.7 51.7\nMaskProp Imagenet [34], COCO [25] 46.6 51.2 44.0 52.6\nMaskProp Imagenet [34], COCO [25], OpenImages [23] 50.0 55.9 44.6 54.5\nTable 2: The results of video instance segmentation on the YouTube-VIS [42] validation dataset. We evaluate the performance\nof each method according to mean average precision (mAP), average precision at 75% IoU threshold (AP@75), and average\nrecall given top 1(AR@1) and top 10(AR@10) detections. The baselines denoted withzwere implemented by the authors\nin [42], whereas the methods marked with\u0003were implemented by us, and use the same backbone and detection networks as\nour approach. Despite its simplicity, MaskProp outperforms all prior video instance segmentation methods by a large margin."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f68-6db4-11ee-88b5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-172", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "our approach. Despite its simplicity, MaskProp outperforms all prior video instance segmentation methods by a large margin.\nScoring Video-Level Sequences. Each video-level sequence contains a list of classi\ufb01cation scores and predicted\nobject labels. To assign a con\ufb01dence score to each videolevel sequence, we average classi\ufb01cation scores associated\nwith that sequence (separately for each object category).\nTraining. We use a similar training setup as in [7]. The loss\nweights for each of the three cascade stages are set to 1;0:5;\nand0:25respectively. The loss weight for the semantic segmentation branch is set to 0:1. We train our model on pairs\nof frames, where the second frame in a pair is randomly selected with a time gap \u000e2[\u000025;25]relative to \ufb01rst frame.\nWe use a multi-scale training approach implemented by resizing the shorter side of the frame randomly between 400\nand800pixels. Our model is trained in a distributed setting using 64GPUs, each GPU holding a single clip. The\ntraining is done for 20epochs with an initial learning rate\nof0:008, which is decreased by 10at16and19epochs.\nWe initialize our model with a Mask R-CNN pretrained on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f69-6db4-11ee-83ce-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-173", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "of0:008, which is decreased by 10at16and19epochs.\nWe initialize our model with a Mask R-CNN pretrained on\nCOCO for the instance segmentation. The hyperparameters\nof RPN and FPN are the same as in [7].\nInference. During testing, we run the bounding box prediction branch on 1000 proposals, apply non-maximum suppression, and use boxes with a score higher than 0:1as input to the mask prediction and mask propagation branches.\nDuring inference, our MaskProp is applied to video clips\nconsisting of 13frames.\n5. Experimental Results\nIn this section, we evaluate MaskProp for video instance\nsegmentation on YouTube-VIS [42], which contains 2;238\ntraining, 302validation, and 343test videos. Each video\nis annotated with per-pixel segmentation, category, and instance labels. The dataset contains 40object categories.\nSince the evaluation on the test set is currently closed, we\nperform our evaluations on the validation set.\n5.1. Quantitative Results\nVideo instance segmentation is a very recent task [42],\nand thus, there are only a few established baselines that we\n7 9 11 13 15 17 19 21"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f6a-6db4-11ee-ae5e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-174", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Video instance segmentation is a very recent task [42],\nand thus, there are only a few established baselines that we\n7 9 11 13 15 17 19 21\nInstance Track Clip Length43.54444.54545.54646.547Accuracy (Video mAP)Figure 6: We plot video mAP as a function of an instance\ntrack clip length (denoted as 2T+ 1in the paper). Based on\nthese results, we observe that optimal video instance segmentation performance is achieved when we propagate instance masks to T= 6previous and subsequent frames.\ncan compare our work to. We include in our comparison\nMaskTrack R-CNN [42], and the EnsembleVIS method,\nwhich won the ICCV 2019 video instance segmentation\nchallenge [28]. Additionally, to make the comparison with\nMaskTrack R-CNN more fair, we reimplement it using the\nsame backbone and detection networks as our MaskProp\n(see MaskTrack R-CNN* in Table 2).\nWe present our quantitative results in Table 2, where we\nassess each method according to 1) mean video average precision (mAP), 2) video average precision at IoU threshold\nof75%, and 3) average recall given 1and10highest scored\ninstances per video. From these results, we observe that\nour MaskProp outperforms all the other baselines according"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f6b-6db4-11ee-89b6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-175", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "of75%, and 3) average recall given 1and10highest scored\ninstances per video. From these results, we observe that\nour MaskProp outperforms all the other baselines according\nto all four evaluation metrics, thus achieving state-of-theart results in video instance segmentation on YouTube-VIS.\nIt can be noted that we outperform EnsembleVIS [28] by\n1:8mAP even though our approach is much simpler and\neven if it uses orders of magnitude less labeled data for\npre-training. Furthermore, compared to the MaskTrack RCNN, our method achieves a 16:3improvement in mAP. We\nalso note that our implementation of MaskTrack [42] signi\ufb01cantly improves upon the original work, but it is still 9:7\nMethod mAP AP@75\nFlowNet2 Propagation 31.4 33.6\nMaskTrack R-CNN\u000336.9 40.2\nMaskProp 46.6 51.2\nTable 3: Here, we study the effectiveness of our mask propagation branch. If we replace it with the FlowNet2 propagation scheme, where masks are propagated using the optical\n\ufb02ow predicted by a FlowNet2 network [20], the accuracy\ndrops from 46:6mAP to 31:4mAP. Similarly, if we replace\nour mask propagation branch with the tracking branch from"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f6c-6db4-11ee-a935-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-176", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "drops from 46:6mAP to 31:4mAP. Similarly, if we replace\nour mask propagation branch with the tracking branch from\nMaskTrack R-CNN, the accuracy drops to 36:9mAP. Note\nthat all of these baselines are implemented using the same\nbackbone and detection networks.\nworse in mAP compared to our model. Lastly, we note that\nadditionally pre-training MaskProp on the OpenImages [23]\ndataset as was done in [28] further boosts our performance\nto50:0mAP.\n5.2. Ablation Experiments\nMask Propagation Branch. To investigate the effectiveness of our mask propagation branch, we compare our\nmethod with a FlowNet2 [20] propagation baseline. For\nthis baseline, we use exactly the same setup as for our\nMaskProp, except that instance masks are propagated using the optical \ufb02ow predicted by a FlowNet2 network [20]\nrather than our proposed mask propagation scheme. For a\nmore complete comparison, we also include the MaskTrack\nR-CNN\u0003from Table 2, which uses the originally proposed\ntracking branch [42], but is implemented using the same\nbackbone and detection networks as our MaskProp.\nThese baselines allow us to directly compare the effectiveness of our mask propagation scheme versus the propagation mechanisms employed by FlowNet2 and MaskTrack"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f6d-6db4-11ee-8cab-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-177", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "backbone and detection networks as our MaskProp.\nThese baselines allow us to directly compare the effectiveness of our mask propagation scheme versus the propagation mechanisms employed by FlowNet2 and MaskTrack\nR-CNN [42] methods. The results in Table 3 show that\nMaskProp outperforms these baselines by a large margin.\nInstance Track Clip Length. Due to occlusions, object instances in video may not be visible in some frames. If there\nare occlusions separated by 2T0+ 2time-steps, we can use\nT >T0to predict longer clip-level instance tracks. In Figure 6, we study video instance segmentation performance as\na function of instance track clip length (denoted as 2T+ 1\nin the paper). Our results indicate that the best accuracy is\nachieved when we use a clip length of 13, meaning that we\npropagate instances to 6previous and 6subsequent frames.\nHigh-Resolution Mask Re\ufb01nement. We also study the impact of our high-resolution mask re\ufb01nement, described in\nSubsection 4.4. We report that removing this re\ufb01nement\ncauses a drop of 1:9%in video instance segmentation mAP.\nImportance of Frame-Level Instance Masks. As described in our main draft, we use frame-level instance masks"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f6e-6db4-11ee-a2da-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-178", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Importance of Frame-Level Instance Masks. As described in our main draft, we use frame-level instance masks\nfor instance-speci\ufb01c feature computation. To investigate theBackbone Network mAP AP@75\nResNet-50 40.0 42.9\nResNet-101 42.5 45.6\nResNeXt-101-64x4d 44.3 48.3\nSTSN [5]-ResNeXt-101-64x4d 46.6 51.2\nTable 4: We study the effect of frame-level instance masks\nto our system\u2019s performance. We evaluate our method\u2019s\naccuracy when using instance masks obtained from Mask\nR-CNN with several different backbones. Our results indicate that frame-level instance masks obtained from stronger\nmodels lead to better video instance segmentation results.\ncontribution of these masks to the performance of our system, we experiment with masks obtained from several different Mask R-CNN models. In Table 4, we present our\nresults for this ablation. Our results indicate that framelevel instance masks obtained from stronger models allow\nus to achieve better video instance segmentation performance. Thus, we expect that future improvements in image\ninstance segmentation will further bene\ufb01t our method.\n5.3. Qualitative Results\nIn Figure 7, we compare our predicted clip-level instance"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f6f-6db4-11ee-be6d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-179", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "instance segmentation will further bene\ufb01t our method.\n5.3. Qualitative Results\nIn Figure 7, we compare our predicted clip-level instance\ntracks (last row of predictions for each clip) with the MaskTrack R-CNN predictions (\ufb01rst row of predictions). We\nuse different colors to represent different object instances.\nOur qualitative results suggest that our MaskProp produces\nmore robust and temporally coherent instance tracks than\nMaskTrack R-CNN. Such differences in performance are\nespecially noticeable when a video contains large object\nmotion, occlusions, or overlapping objects.\nIn Figure 5, we also visualize instance-speci\ufb01c features\nthat are propagated from frame tto other frames in the\ngiven video clip for two different object instances detected\nin framet. Here, we show activations from a randomly selected feature channel. Based on these results, we observe\nthat our MaskProp reliably propagates features that are speci\ufb01c to each instance despite motion blur, object deformations and large variations in object appearance.\n6. Conclusion\nIn this work, we introduced MaskProp, a novel architecture for video instance segmentation. Our method is conceptually simple, it does not require large amounts of labeled data for pre-training, and it produces state-of-the-art"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f70-6db4-11ee-a15e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-180", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "results on YouTube-VIS dataset. In future, we plan to extend MaskProp to scenarios where only bounding box annotations are available. We are also interested in applying our\nmethod to problems such as pose estimation and tracking.\nVideo ClipMaskPropMaskTrack      R-CNN\nVideo ClipMaskPropMaskTrack      R-CNN\nVideo Clip\nMaskProp\nMaskTrack      R-CNNVideo Clip\nMaskPropMaskTrack      R-CNN\nFigure 7: We compare our video instance segmentation results with MaskTrack R-CNN [42] predictions. Different object\ninstances are encoded with different colors. The \ufb01rst row for each video shows the original frames. The second row illustrates\nthe mask predictions of MaskTrack R-CNN and the third row those obtained with our MaskProp. Compared to MaskTrack\nR-CNN, our MaskProp tracks object instances more robustly even when they are occluded or overlap with each other.\nReferences\n[1] Anurag Arnab and Philip H. S. Torr. Pixelwise instance segmentation with a dynamically instantiated network. In 2017\nIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,\npages 879\u2013888, 2017. 2\n[2] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f71-6db4-11ee-ada2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-181", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "pages 879\u2013888, 2017. 2\n[2] M. Bai and R. Urtasun. Deep watershed transform for instance segmentation. In 2017 IEEE Conference on Computer\nVision and Pattern Recognition (CVPR) , pages 2858\u20132866,\nJuly 2017. 2\n[3] Gedas Bertasius, Christoph Feichtenhofer, Du Tran, Jianbo\nShi, and Lorenzo Torresani. Learning temporal pose estimation from sparsely labeled videos. In Advances in Neural\nInformation Processing Systems 33 . 2019. 4, 6\n[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Semantic segmentation with boundary neural \ufb01elds. In The\nIEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016. 1\n[5] Gedas Bertasius, Lorenzo Torresani, and Jianbo Shi. Object\ndetection in video with spatiotemporal sampling networks.\nInECCV (12) , volume 11216 of Lecture Notes in Computer\nScience , pages 342\u2013357. Springer, 2018. 2, 4, 6, 8\n[6] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f72-6db4-11ee-88a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-182", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "[6] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset,\nLaura Leal-Taix\u00e9, Daniel Cremers, and Luc Van Gool. Oneshot video object segmentation. In Computer Vision and Pattern Recognition (CVPR) , 2017. 2\n[7] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao\nLi, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi,\nWanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid\ntask cascade for instance segmentation. In IEEE Conference\non Computer Vision and Pattern Recognition, CVPR 2019,\nLong Beach, CA, USA, June 16-20, 2019 , pages 4974\u20134983.\nComputer Vision Foundation / IEEE, 2019. 6, 7\n[8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,\nKevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal.\nMach. Intell. , 40(4):834\u2013848, 2018. 1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f73-6db4-11ee-bddf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-183", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Mach. Intell. , 40(4):834\u2013848, 2018. 1\n[9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian\nSchroff, and Hartwig Adam. Encoder-decoder with atrous\nseparable convolution for semantic image segmentation. In\nComputer Vision - ECCV 2018 - 15th European Conference,\nMunich, Germany, September 8-14, 2018, Proceedings, Part\nVII, pages 833\u2013851, 2018. 2\n[10] Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van\nGool. Blazingly fast video object segmentation with pixelwise metric learning. In CVPR , pages 1189\u20131198. IEEE\nComputer Society, 2018. 2\n[11] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In\nCVPR , 2016. 2\n[12] J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei.\nDeformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , volume 00,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f74-6db4-11ee-9101-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-184", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Deformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , volume 00,\npages 764\u2013773, Oct. 2017. 4, 6[13] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.\nDetect to track and track to detect. In International Conference on Computer Vision (ICCV) , 2017. 2\n[14] Ross Girshick. Fast R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV) , 2015. 1\n[15] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra\nMalik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE\nConference on Computer Vision and Pattern Recognition\n(CVPR) , 2014. 1\n[16] Kaiming He, Georgia Gkioxari, Piotr Doll\u00e1r, and Ross Girshick. Mask R-CNN. In Proceedings of the International\nConference on Computer Vision (ICCV) , 2017. 1, 2, 4\n[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\nDeep residual learning for image recognition. 2016 IEEE\nConference on Computer Vision and Pattern Recognition\n(CVPR) , pages 770\u2013778, 2016. 1, 4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f75-6db4-11ee-9d00-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-185", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Deep residual learning for image recognition. 2016 IEEE\nConference on Computer Vision and Pattern Recognition\n(CVPR) , pages 770\u2013778, 2016. 1, 4\n[18] Alexander Hermans*, Lucas Beyer*, and Bastian Leibe. In\nDefense of the Triplet Loss for Person Re-Identi\ufb01cation.\narXiv preprint arXiv:1703.07737 , 2017. 2\n[19] Alba Herrera-Palacio, Carles Ventura, and Xavier Giro-i Nieto. Video object linguistic grounding. In 1st International\nWorkshop on Multimodal Understanding and Learning for\nEmbodied Applications , MULEA \u201919, pages 49\u201351, New\nYork, NY , USA, 2019. ACM. 2\n[20] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper,\nAlexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical \ufb02ow estimation with deep networks. CoRR ,\nabs/1612.01925, 2016. 8\n[21] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion and appearance for fully\nautomatic segmentation of generic objects in videos. In 2017"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f76-6db4-11ee-b048-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-186", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "automatic segmentation of generic objects in videos. In 2017\nIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,\npages 2117\u20132126, 2017. 2\n[22] Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bogdan Savchynskyy, and Carsten Rother. InstanceCut: from\nedges to instances with multicut. In CVPR , 2017. 2\n[23] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan\nPopov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari.\nThe open images dataset v4: Uni\ufb01ed image classi\ufb01cation,\nobject detection, and visual relationship detection at scale,\n2018. 2, 7, 8\n[24] Tsung-Yi Lin, Piotr Doll\u00e1r, Ross Girshick, Kaiming He,\nBharath Hariharan, and Serge Belongie. Feature pyramid\nnetworks for object detection. In CVPR , 2017. 6\n[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f77-6db4-11ee-9450-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-187", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "networks for object detection. In CVPR , 2017. 6\n[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,\nPietro Perona, Deva Ramanan, Piotr Doll\u00e1r, and C. Lawrence\nZitnick. Microsoft coco: Common objects in context. In\nEuropean Conference on Computer Vision (ECCV) , Z\u00fcrich,\nSeptember 2014. 2, 3, 7\n[26] S. Liu, J. Jia, S. Fidler, and R. Urtasun. Sgn: Sequential\ngrouping networks for instance segmentation. In 2017 IEEE\nInternational Conference on Computer Vision (ICCV) , pages\n3516\u20133524, Oct 2017. 2\n[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully\nconvolutional networks for semantic segmentation. In\nCVPR , 2015. 1\n[28] Jonathon Luiten, Philip H.S. Torr, and Bastian Leibe. Video\ninstance segmentation 2019: A winning approach for combined detection, segmentation, classi\ufb01cation and tracking.\nIn2019 IEEE International Conference on Computer Vision\nWorkshops, ICCV Workshops 2019, Seoul, South Korea, October 27-November 2, 2019 . IEEE, 2019. 1, 2, 7, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f78-6db4-11ee-83d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-188", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "Workshops, ICCV Workshops 2019, Seoul, South Korea, October 27-November 2, 2019 . IEEE, 2019. 1, 2, 7, 8\n[29] Jonathon Luiten, Paul V oigtlaender, and Bastian Leibe. Premvos: Proposal-generation, re\ufb01nement and merging for\nvideo object segmentation. In Asian Conference on Computer Vision , 2018. 2\n[30] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan,\nKaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe,\nand Laurens van der Maaten. Exploring the limits of weakly\nsupervised pretraining. In ECCV , 2018. 2, 7\n[31] Aljo\u0161a O\u0161ep, Paul V oigtlaender, Jonathon Luiten, Stefan\nBreuers, and Bastian Leibe. Large-scale object mining for\nobject discovery from unlabeled video. ICRA , 2019. 2\n[32] F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and\nA.Sorkine-Hornung. Learning video object segmentation\nfrom static images. In Computer Vision and Pattern Recognition , 2017. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f79-6db4-11ee-ba2d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-189", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "A.Sorkine-Hornung. Learning video object segmentation\nfrom static images. In Computer Vision and Pattern Recognition , 2017. 2\n[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.\nFaster R-CNN: Towards real-time object detection with region proposal networks. In Neural Information Processing\nSystems (NIPS) , 2015. 1\n[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,\nAditya Khosla, Michael Bernstein, Alexander C. Berg, and\nLi Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575 , 2014. 2, 7\n[35] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.\nPWC-Net: CNNs for optical \ufb02ow using pyramid, warping,\nand cost volume. 2018. 2\n[36] P. Tokmakov, K. Alahari, and C. Schmid. Learning video\nobject segmentation with visual memory. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages\n4491\u20134500, Oct 2017. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f7a-6db4-11ee-8308-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-190", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "object segmentation with visual memory. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages\n4491\u20134500, Oct 2017. 2\n[37] Paul V oigtlaender, Yuning Chai, Florian Schroff, Hartwig\nAdam, Bastian Leibe, and Liang-Chieh Chen. Feelvos: Fast\nend-to-end embedding learning for video object segmentation. In CVPR , 2019. 2, 7\n[38] Paul V oigtlaender, Michael Krause, Aljosa Osep, Jonathon\nLuiten, Berin Balachandar Gnana Sekar, Andreas Geiger,\nand Bastian Leibe. Mots: Multi-object tracking and segmentation. In Proceedings IEEE Conf. on Computer Vision and\nPattern Recognition (CVPR) , June 2019. 2\n[39] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple\nonline and realtime tracking with a deep association metric.\nCoRR , abs/1703.07402, 2017. 7\n[40] Fanyi Xiao and Yong Jae Lee. Video object detection with an\naligned spatial-temporal memory. In European Conference\non Computer Vision (ECCV) , 2018. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f7b-6db4-11ee-b389-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-191", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "[40] Fanyi Xiao and Yong Jae Lee. Video object detection with an\naligned spatial-temporal memory. In European Conference\non Computer Vision (ECCV) , 2018. 2\n[41] Saining Xie, Ross Girshick, Piotr Doll\u00e1r, Zhuowen Tu, and\nKaiming He. Aggregated residual transformations for deep\nneural networks. In CVPR , 2017. 6[42] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV , 2019. 1, 2, 3, 7, 8, 9\n[43] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang,\nand Aggelos K. Katsaggelos. Ef\ufb01cient video object segmentation via network modulation. In 2018 IEEE Conference on\nComputer Vision and Pattern Recognition, CVPR 2018, Salt\nLake City, UT, USA, June 18-22, 2018 , pages 6499\u20136507,\n2018. 2, 7\n[44] Jifeng Dai Xiangyang Ji Yi Li, Haozhi Qi and Yichen Wei.\nFully convolutional instance-aware semantic segmentation.\nInCVPR , 2017. 2\n[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f7c-6db4-11ee-8f19-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1912.04573-192", "title": "Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation", "source": "http://arxiv.org/pdf/1912.04573", "start_index": 0, "chroma:document": "InCVPR , 2017. 2\n[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang\nWang, and Jiaya Jia. Pyramid scene parsing network. In 2017\nIEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,\npages 6230\u20136239, 2017. 1\n[46] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen\nWei. Flow-guided feature aggregation for video object detection. In International Conference on Computer Vision\n(ICCV) , 2017. 2\n[47] Idil Esen Zul\ufb01kar, Jonathon Luiten, and Bastian Leibe. Unovost: Unsupervised of\ufb02ine video object segmentation and\ntracking for the 2019 unsupervised davis challenge. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f7d-6db4-11ee-9a47-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-0", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Quantifying Attention Flow in Transformers\nSamira Abnar\nILLC, University of Amsterdam\ns.abnar@uva.nlWillem Zuidema\nILLC, University of Amsterdam\nw.h.zuidema@uva.nl\nAbstract\nIn the Transformer model, \u201cself-attention\u201d\ncombines information from attended embeddings into the representation of the focal embedding in the next layer. Thus, across layers of the Transformer, information originating\nfrom different tokens gets increasingly mixed.\nThis makes attention weights unreliable as explanations probes. In this paper, we consider\nthe problem of quantifying this \ufb02ow of information through self-attention. We propose two\nmethods for approximating the attention to input tokens given attention weights, attention\nrollout andattention \ufb02ow , as post hoc methods\nwhen we use attention weights as the relative\nrelevance of the input tokens. We show that\nthese methods give complementary views on\nthe \ufb02ow of information, and compared to raw\nattention, both yield higher correlations with\nimportance scores of input tokens obtained using an ablation method and input gradients.\n1 Introduction\nAttention (Bahdanau et al., 2015; Vaswani et al.,\n2017) has become the key building block of neural sequence processing models, and visualizing\nattention weights is the easiest and most popular"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f7e-6db4-11ee-ac41-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-1", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "2017) has become the key building block of neural sequence processing models, and visualizing\nattention weights is the easiest and most popular\napproach to interpret a model\u2019s decisions and to\ngain insights about its internals (Vaswani et al.,\n2017; Xu et al., 2015; Wang et al., 2016; Lee et al.,\n2017; Dehghani et al., 2019; Rockt \u00a8aschel et al.,\n2016; Chen and Ji, 2019; Coenen et al., 2019; Clark\net al., 2019). Although it is wrong to equate attention with explanation (Pruthi et al., 2019; Jain and\nWallace, 2019), it can offer plausible and meaningful interpretations (Wiegreffe and Pinter, 2019;\nVashishth et al., 2019; Vig, 2019). In this paper,\nwe focus on problems arising when we move to the\nhigher layers of a model, due to lack of token identi\ufb01ability of the embeddings in higher layers (Brunner et al., 2020).We propose two simple but effective methods to\ncompute attention scores to input tokens (i.e., token\nattention ) at each layer, by taking raw attentions\n(i.e., embedding attention ) of that layer as well as\nthose from the precedent layers. These methods"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f7f-6db4-11ee-916d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-2", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "attention ) at each layer, by taking raw attentions\n(i.e., embedding attention ) of that layer as well as\nthose from the precedent layers. These methods\nare based on modelling the information \ufb02ow in the\nnetwork with a DAG (Directed Acyclic Graph), in\nwhich the nodes are input tokens and hidden embeddings, edges are the attentions from the nodes\nin each layer to those in the previous layer, and\nthe weights of the edges are the attention weights.\nThe \ufb01rst method, attention rollout , assumes that\nthe identities of input tokens are linearly combined\nthrough the layers based on the attention weights.\nTo adjust attention weights, it rolls out the weights\nto capture the propagation of information from input tokens to intermediate hidden embeddings. The\nsecond method, attention \ufb02ow , considers the attention graph as a \ufb02ow network. Using a maximum\n\ufb02ow algorithm, it computes maximum \ufb02ow values,\nfrom hidden embeddings (sources) to input tokens\n(sinks). In both methods, we take the residual connection in the network into account to better model\nthe connections between input tokens and hidden\nembedding. We show that compared to raw attention, the token attentions from attention rollout and\nattention \ufb02ow have higher correlations with the importance scores obtained from input gradients as"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f80-6db4-11ee-9954-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-3", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "embedding. We show that compared to raw attention, the token attentions from attention rollout and\nattention \ufb02ow have higher correlations with the importance scores obtained from input gradients as\nwell as blank-out , an input ablation based attribution method. Furthermore, we visualize the token\nattention weights and demonstrate that they are better approximations of how input tokens contribute\nto a predicted output, compared to raw attention.\nIt is noteworthy that the techniques we propose\nin this paper, are not toward making hidden embeddings more identi\ufb01able, or providing better attention weights for better performance, but a new set\nof attention weights that take token identity problem into consideration and can serve as a better\ndiagnostic tool for visualization and debugging.arXiv:2005.00928v2  [cs.LG]  31 May 2020\n(a) Embedding attentions\n (b) Attention rollout\n (c) Attention \ufb02ow\nFigure 1: Visualisation of attention weights.\nFigure 2: Raw Attention maps for the CLS token at\ndifferent layers.\n2 Setups and Problem Statement\nIn our analysis, we focus on the verb number prediction task, i.e., predicting singularity or plurality\nof a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f81-6db4-11ee-b209-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-4", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "of a verb of a sentence, when the input is the sentence up to the verb position. We use the subjectverb agreement dataset (Linzen et al., 2016). This\ntask and dataset are convenient choices, as they offer a clear hypothesis about what part of the input\nis essential to get the right solution. For instance,\ngiven \u201c the key to the cabinets \u201d as the input, we\nknow that attending to \u201ckey\u201d helps the model predict singular as output while attending to \u201ccabinets\u201d\n(anagreement attractor , with the opposite number)\nis unhelpful.\nWe train a Transformer encoder, with GPT2 Transformer blocks as described in (Radford\net al., 2019; Wolf et al., 2019) (without masking).\nThe model has 6 layers, and 8 heads, with hidden/embedding size of 128. Similar to Bert (Devlin et al., 2019) we add a CLS token and use\nits embedding in the \ufb01nal layer as the input to\nthe classi\ufb01er. The accuracy of the model on the\nsubject-verb agreement task is 0:96. To facilitate\nreplication of our experiments we will make the\nimplementations of the models we use and algorithms we introduce publicly available at https:\n//github.com/samiraabnar/attention_flow ."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f82-6db4-11ee-9b1e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-5", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "implementations of the models we use and algorithms we introduce publicly available at https:\n//github.com/samiraabnar/attention_flow .\nWe start by visualizing raw attention in Figure 1a\n(like Vig 2019). The example given here is correctly classi\ufb01ed. Crucially, only in the \ufb01rst couple\nof layers, there are some distinctions in the attention patterns for different positions, while in higher\nlayers the attention weights are rather uniform. Fig-ure 2 (left) gives raw attention scores of the CLS\ntoken over input tokens (x-axis) at different layers (y-axis), which similarly lack an interpretable\npattern.These observations re\ufb02ect the fact that as\nwe go deeper into the model, the embeddings are\nmore contextualized and may all carry similar information. This underscores the need to track down\nattention weights all the way back to the input layer\nand is in line with \ufb01ndings of Serrano and Smith\n(2019), who show that attention weights do not\nnecessarily correspond to the relative importance\nof input tokens.\nTo quantify the usefulness of raw attention\nweights, and the two alternatives that we consider\nin the next section, besides input gradients, we\nemploy an input ablation method, blank-out , to estimate an importance score for each input token."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f83-6db4-11ee-bba9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-6", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "in the next section, besides input gradients, we\nemploy an input ablation method, blank-out , to estimate an importance score for each input token.\nBlank-out replaces each token in the input, one\nby one, with UNK and measures how much it affects the predicted probability of the correct class.\nWe compute the Spearman\u2019s rank correlation coef\ufb01cient between the attention weights of the CLS\nembedding in the \ufb01nal layer and the importance\nscores from blank-out. As shown in the \ufb01rst row\nof Table 1, the correlation between raw attention\nweights of the CLS token and blank-out scores is\nrather low, except for the \ufb01rst layer. As we can see\nin Table 2 this is also the case when we compute\nthe correlations with input gradients.\nL1 L2 L3 L4 L5 L6\nRaw 0.69\u00060.27 0.10\u00060.43 -0.11\u00060.49 -0.09\u00060.52 0.20\u00060.45 0.29\u00060.39\nRollout 0.32\u00060.26 0.38\u00060.27 0.51\u00060.26 0.62\u00060.26 0.70\u00060.25 0.71\u00060.24"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f84-6db4-11ee-8bab-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-7", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Flow 0.32\u00060.26 0.44\u00060.29 0.70\u00060.25 0.70\u00060.22 0.71\u00060.22 0.70\u00060.22\nTable 1: SpearmanR correlation of attention based importance with blank-out scores for 2000 samples from\nthe test set for the verb number prediction model.\n3 Attention Rollout and Attention Flow\nAttention rollout and attention \ufb02ow recursively\ncompute the token attentions in each layer of a\nL1 L2 L3 L4 L5 L6\nRaw 0.53\u00060.33 0.16\u00060.38 -0.06\u00060.42 0.00\u00060.47 0.24\u00060.40 0.46\u00060.35\nRollout 0.22\u00060.31 0.27\u00060.32 0.39\u00060.32 0.47\u00060.32 0.53\u00060.32 0.54\u00060.31\nFlow 0.22\u00060.31 0.31\u00060.34 0.54\u00060.32 0.61\u00060.28 0.60\u00060.28 0.61\u00060.28\nTable 2: SpearmanR correlation of attention based importance with input gradients for 2000 samples from\nthe test set for the verb number prediction model."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f85-6db4-11ee-ac99-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-8", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Table 2: SpearmanR correlation of attention based importance with input gradients for 2000 samples from\nthe test set for the verb number prediction model.\ngiven model given the embedding attentions as input. They differ in the assumptions they make\nabout how attention weights in lower layers affect\nthe \ufb02ow of information to the higher layers and\nwhether to compute the token attentions relative to\neach other or independently.\nTo compute how information propagates from\nthe input layer to the embeddings in higher layers, it is crucial to take the residual connections\nin the model into account as well as the attention\nweights. In a Transformer block, both self-attention\nand feed-forward networks are wrapped by residual connections, i.e., the input to these modules is\nadded to their output. When we only use attention\nweights to approximate the \ufb02ow of information in\nTransformers, we ignore the residual connections.\nBut these connections play a signi\ufb01cant role in\ntying corresponding positions in different layers.\nHence, to compute attention rollout and attention\n\ufb02ow, we augment the attention graph with extra\nweights to represent residual connections. Given\nthe attention module with residual connection, we\ncompute values in layer l+1asVl+1=Vl+WattVl,\nwhere Wattis the attention matrix. Thus, we have"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f86-6db4-11ee-8683-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-9", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "compute values in layer l+1asVl+1=Vl+WattVl,\nwhere Wattis the attention matrix. Thus, we have\nVl+1= (Watt+I)Vl. So, to account for residual\nconnections, we add an identity matrix to the attention matrix and re-normalize the weights. This\nresults in A= 0:5Watt+ 0:5I, where Ais the raw\nattention updated by residual connections.\nFurthermore, analyzing individual heads requires accounting for mixing of information between heads through a position-wise feed-forward\nnetwork in Transformer block. Using attention rollout and attention \ufb02ow, it is also possible to analyze\neach head separately. We explain in more details\nin Appendix A.1. However, in our analysis in this\npaper, for simplicity, we average the attention at\neach layer over all heads.\nAttention rollout Attention rollout is an intuitive\nway of tracking down the information propagated\nfrom the input layer to the embeddings in the higher\nlayers. Given a Transformer with Llayers, we want\nto compute the attention from all positions in layerlito all positions in layer lj, where j < i . In the\nattention graph, a path from node vat position k\ninli, to node uat position minlj, is a series of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f87-6db4-11ee-8d8e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-10", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "attention graph, a path from node vat position k\ninli, to node uat position minlj, is a series of\nedges that connect these two nodes. If we look\nat the weight of each edge as the proportion of\ninformation transferred between two nodes, we\ncan compute how much of the information at v\nis propagated to uthrough a particular path by\nmultiplying the weights of all edges in that path.\nSince there may be more than one path between\ntwo nodes in the attention graph, to compute the\ntotal amount of information propagated from vtou,\nwe sum over all possible paths between these two\nnodes. At the implementation level, to compute the\nattentions from litolj, we recursively multiply the\nattention weights matrices in all the layers below.\n~A(li) =\u001a\nA(li)~A(li\u00001)ifi > j\nA(li) ifi=j(1)\nIn this equation, ~Ais attention rollout, Ais raw attention and the multiplication operation is a matrix\nmultiplication. With this formulation, to compute\ninput attention we set j= 0.\nAttention \ufb02ow In graph theory, a \ufb02ow network\nis a directed graph with a \u201ccapacity\u201d associated"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f88-6db4-11ee-bdd5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-11", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "input attention we set j= 0.\nAttention \ufb02ow In graph theory, a \ufb02ow network\nis a directed graph with a \u201ccapacity\u201d associated\nwith each edge. Formally, given G= (V; E)is a\ngraph, where Vis the set of nodes, and Eis the set\nof edges in G;C=fcuv2Rj8u; vwhere eu;v2\nE^u6=vgdenotes the capacities of the edges and\ns; t2Vare the source and target (sink) nodes respectively; \ufb02ow is a mapping of edges to real numbers,f:E!R, that satis\ufb01es two conditions: (a)\ncapacity constraint : for each edge the \ufb02ow value\nshould not exceed its capacity, jfuv\u0014cuvj; (b)\n\ufb02ow conservation : for all nodes except sandtthe\ninput \ufb02ow should be equal to output \ufb02ow \u2013sum\nof the \ufb02ow of outgoing edges should be equal to\nsum of the \ufb02ow of incoming edges. Given a \ufb02ow\nnetwork, a maximum \ufb02ow algorithm \ufb01nds a \ufb02ow\nwhich has the maximum possible value between s\nandt(Cormen et al., 2009)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f89-6db4-11ee-9990-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-12", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "network, a maximum \ufb02ow algorithm \ufb01nds a \ufb02ow\nwhich has the maximum possible value between s\nandt(Cormen et al., 2009).\nTreating the attention graph as a \ufb02ow network,\nwhere the capacities of the edges are attention\nweights, using any maximum \ufb02ow algorithm, we\ncan compute the maximum attention \ufb02ow from any\nnode in any of the layers to any of the input nodes.\nWe can use this maximum-\ufb02ow-value as an approximation of the attention to input nodes. In attention\n\ufb02ow, the weight of a single path is the minimum\nvalue of the weights of the edges in the path, instead of the product of the weights. Besides, we\nattention rollout\n attention \ufb02ow\nFigure 3: Attention maps for the CLS token\n.\ncan not compute the attention for node sto node\ntby adding up the weights of all paths between\nthese two nodes, since there might be an overlap\nbetween the paths and this might result in over\ufb02ow\nin the overlapping edges.\nIt is noteworthy that both of the proposed methods can be computed in polynomial time. O(d\u0003n2)\nfor attention rollout and O(d2\u0003n4)for attention\n\ufb02ow, where dis the depth of the model, and nis"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f8a-6db4-11ee-b468-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-13", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "for attention rollout and O(d2\u0003n4)for attention\n\ufb02ow, where dis the depth of the model, and nis\nthe number of tokens.\n4 Analysis and Discussion\nNow, we take a closer look at these three views of\nattention. Figure 1 depicts raw attention, attention\nrollout and attention \ufb02ow for a correctly classi\ufb01ed\nexample across different layers. It is noteworthy\nthat the \ufb01rst layer of attention rollout and attention\n\ufb02ow are the same, and their only difference with\nraw attention is the addition of residual connections. As we move to the higher layers, we see that\nthe residual connections fade away. Moreover, in\ncontrast to raw attention, the patterns of attention\nrollout and attention \ufb02ow become more distinctive\nin the higher layers.\nFigures 2 and 3 show the weights from raw attention, attention rollout and attention \ufb02ow for the\nCLS embedding over input tokens (x-axis) in all\n6 layers (y-axis) for three examples. The \ufb01rst example is the same as the one in Figure 1. The second example is \u201c the article on NNP large systems\n<?>\u201d. The model correctly classi\ufb01es this example and changing the subject of the missing verb"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f8b-6db4-11ee-9daa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-14", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "<?>\u201d. The model correctly classi\ufb01es this example and changing the subject of the missing verb\nfrom \u201carticle\u201d to \u201carticle s\u201d \ufb02ips the decision of the\nmodel. The third example is \u201c here the NNS differ\nin that the female <?>\u201d, which is a miss-classi\ufb01ed\nexample and again changing \u201cNNS\u201d (plural noun)\nto \u201cNNP\u201d (singular proper noun) \ufb02ips the decision\nof the model.\nFor all cases, the raw attention weights are almost uniform above layer three (discussed before).\nraw attention\nattention rollout\nattention \ufb02ow\n(a) \u201cThe author talked to Sara about mask book .\u201d\nraw attention\nattention rollout\nattention \ufb02ow\n(b)\u201cMary convinced John of mask love.\u201d\nFigure 4: Bert attention maps. We look at the attention\nweights from the mask embedding to the two potential\nreferences for it, e.g. \u201cauthor\u201d and \u201cSara\u201d in (a) and\n\u201cMary\u201d and \u201cJohn\u201d in (b). The bars, at the left, show\nthe relative predicted probability for the two possible"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f8c-6db4-11ee-89bf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-15", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "\u201cMary\u201d and \u201cJohn\u201d in (b). The bars, at the left, show\nthe relative predicted probability for the two possible\npronouns, \u201chis\u201d and \u201cher\u201d.\nIn the case of the correctly classi\ufb01ed example, we\nobserve that both attention rollout and attention\n\ufb02ow assign relatively high weights to both the subject of the verb, \u201carticle\u2019 and the attractor, \u201csystems\u201d. For the miss-classi\ufb01ed example, both attention rollout and attention \ufb02ow assign relatively\nhigh scores to the \u201cNNS\u201d token which is not the\nsubject of the verb. This can explain the wrong\nprediction of the model.\nThe main difference between attention rollout\nand attention \ufb02ow is that attention \ufb02ow weights are\namortized among the set of most attended tokens,\nas expected. Attention \ufb02ow can indicate a set of\ninput tokens that are important for the \ufb01nal decision.\nThus we do not get sharp distinctions among them.\nOn the other hand, attention rollout weights are\nmore focused compared to attention \ufb02ow weights,\nwhich is sensible for the third example but not as\nmuch for the second one.\nL1 L3 L5 L6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f8d-6db4-11ee-9283-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-16", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "more focused compared to attention \ufb02ow weights,\nwhich is sensible for the third example but not as\nmuch for the second one.\nL1 L3 L5 L6\nRaw 0.12\u00060.21 0.09\u00060.21 0.08\u00060.20 0.09\u00060.21\nRollout 0.11\u00060.19 0.12\u00060.21 0.13\u00060.21 0.13\u00060.20\nFlow 0.11\u00060.19 0.11\u00060.21 0.12\u00060.22 0.14\u00060.21\nTable 3: SpearmanR correlation of attention based importance with input gradients for 100 samples from the\ntest set for the DistillBERT model \ufb01ne tuned on SST-2.\nFurthermore, as shown in Table 1 and 2 both\nattention rollout and attention \ufb02ow, are better\ncorrelated with blank-out scores and input gradients compared to raw attention, but attention \ufb02ow\nweights are more reliable than attention rollout.\nThe difference between these two methods is rooted\nin their different views of attention weights. Attention \ufb02ow views them as capacities, and at every\nstep of the algorithm, it uses as much of the capacity as possible. Hence, attention \ufb02ow computes the\nmaximum possibility of token identities to propagate to the higher layers. Whereas attention rollout"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f8e-6db4-11ee-83b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-17", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "maximum possibility of token identities to propagate to the higher layers. Whereas attention rollout\nviews them as proportion factors and at every step,\nit allows token identities to be propagated to higher\nlayers exactly based on this proportion factors. This\nmakes attention rollout stricter than attention \ufb02ow,\nand so we see that attention rollout provides us with\nmore focused attention patterns. However, since\nwe are making many simplifying assumptions, the\nstrictness of attention rollout does not lead to more\naccurate results, and the relaxation of attention \ufb02ow\nseems to be a useful property.\nAt last, to illustrate the application of attention \ufb02ow and attention rollout on different tasks\nand different models, we examine them on two\npretrained BERT models. We use the models\navailable at https://github.com/huggingface/\ntransformers .\nTable 3 shows the correlation of the importance\nscore obtained from raw attention, attention rollout\nand attention \ufb02ow from a DistillBERT (Sanh et al.,\n2019) model \ufb01ne-tuned to solve \u201cSST-2\u201d (Socher\net al., 2013), the sentiment analysis task from the\nglue benchmark (Wang et al., 2018). Even though\nfor this model, all three methods have very low\ncorrelation with the input gradients, we can still see"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f8f-6db4-11ee-8ac2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-18", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "glue benchmark (Wang et al., 2018). Even though\nfor this model, all three methods have very low\ncorrelation with the input gradients, we can still see\nthat attention rollout and attention \ufb02ow are slightly\nbetter than raw attention.\nFurthermore, in Figure 4, we show an example\nof applying these methods to a pre-trained Bert\nto see how it resolves the pronouns in a sentence.\nWhat we do here is to feed the model with a sentence, masking a pronoun. Next, we look at the\nprediction of the model for the masked word and\ncompare the probabilities assigned to \u201cher\u201d and\n\u201chis\u201d. Then we look at raw attention, attention rollout and attention \ufb02ow weights of the embeddings\nfor the masked pronoun at all the layers. In the\n\ufb01rst example, in Figure 4a, attention rollout and\nattention \ufb02ow are consistent with each other and\nthe prediction of the model. Whereas, the \ufb01nal\nlayer of raw attention does not seem to be consistent with the prediction of the models, and it varies\na lot across different layers. In the second example, in Figure 4b, only attention \ufb02ow weights areconsistent with the prediction of the model.\n5 Conclusion\nTranslating embedding attentions to token attentions can provide us with better explanations about"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f90-6db4-11ee-981f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-19", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "5 Conclusion\nTranslating embedding attentions to token attentions can provide us with better explanations about\nmodels\u2019 internals. Yet, we should be cautious about\nour interpretation of these weights, because, we\nare making many simplifying assumptions when\nwe approximate information \ufb02ow in a model with\nthe attention weights. Our ideas are simple and\ntask/architecture agnostic. In this paper, we insisted on sticking with simple ideas that only require attention weights and can be easily employed\nin any task or architecture that uses self-attention.\nWe should note that all our analysis in this paper is\nfor a Transformer encoder, with no casual masking.\nSince in Transformer decoder, future tokens are\nmasked, naturally there is more attention toward\ninitial tokens in the input sequence, and both attention rollout and attention \ufb02ow will be biased toward\nthese tokens. Hence, to apply these methods on a\nTransformer decoder, we should \ufb01rst normalize\nbased on the receptive \ufb01eld of attention.\nFollowing this work, we can build the attention\ngraph with effective attention weights (Brunner\net al., 2020) instead of raw attentions. Furthermore,\nwe can come up with a new method that adjusts the\nattention weights using gradient-based attribution\nmethods (Ancona et al., 2019).\nAcknowledgements"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f91-6db4-11ee-a151-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-20", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "we can come up with a new method that adjusts the\nattention weights using gradient-based attribution\nmethods (Ancona et al., 2019).\nAcknowledgements\nWe thank Mostafa Dehghani, Wilker Aziz, and the\nanonymous reviewers for their valuable feedback\nand comments on this work. The work presented\nhere was funded by the Netherlands Organization\nfor Scienti\ufb01c Research (NWO), through a Gravitation Grant 024.001.006 to the Language in Interaction Consortium.\nReferences\nMarco Ancona, Enea Ceolini, Cengiz \u00a8Oztireli, and\nMarkus Gross. 2019. Gradient-Based Attribution\nMethods , pages 169\u2013191. Springer International\nPublishing.\nDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly\nlearning to align and translate. In proceedings of\nthe 2015 International Conference on Learning Representations .\nGino Brunner, Yang Liu, Damian Pascual, Oliver\nRichter, Massimiliano Ciaramita, and Roger Wattenhofer. 2020. On identi\ufb01ability in transformers. In\nInternational Conference on Learning Representations .\nHanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classi\ufb01ers via data"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f92-6db4-11ee-9ec9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-21", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "International Conference on Learning Representations .\nHanjie Chen and Yangfeng Ji. 2019. Improving the interpretability of neural sentiment classi\ufb01ers via data\naugmentation. arXiv preprint arXiv:1909.04225 .\nKevin Clark, Urvashi Khandelwal, Omer Levy, and\nChristopher D. Manning. 2019. What does BERT\nlook at? an analysis of BERT\u2019s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP:\nAnalyzing and Interpreting Neural Networks for\nNLP, pages 276\u2013286, Florence, Italy. Association\nfor Computational Linguistics.\nAndy Coenen, Emily Reif, Ann Yuan, Been Kim,\nAdam Pearce, Fernanda Vi \u00b4egas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry\nof bert. arXiv preprint arXiv:1906.02715 .\nThomas H. Cormen, Charles E. Leiserson, Ronald L.\nRivest, and Clifford Stein. 2009. Introduction to\nAlgorithms, Third Edition , 3rd edition. The MIT\nPress.\nMostafa Dehghani, Stephan Gouws, Oriol Vinyals,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f93-6db4-11ee-925c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-22", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Algorithms, Third Edition , 3rd edition. The MIT\nPress.\nMostafa Dehghani, Stephan Gouws, Oriol Vinyals,\nJakob Uszkoreit, and \u0141ukasz Kaiser. 2019. Universal transformers. In proceedings of the 2019 International Conference on Learning Representations .\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and\nKristina Toutanova. 2019. BERT: Pre-training of\ndeep bidirectional transformers for language understanding. In proceedings of the 2019 Conference\nof the North American Chapter of the Association\nfor Computational Linguistics: Human Language\nTechnologies . Association for Computational Linguistics.\nSarthak Jain and Byron C. Wallace. 2019. Attention\nis not Explanation. In proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 3543\u20133556. Association\nfor Computational Linguistics.\nJaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim.\n2017. Interactive visualization and manipulation\nof attention-based neural machine translation. In\nproceedings of the 2017 Conference on Empirical\nMethods in Natural Language Processing: System\nDemonstrations , pages 121\u2013126.\nTal Linzen, Emmanuel Dupoux, and Yoav Goldberg."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f94-6db4-11ee-8804-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-23", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Methods in Natural Language Processing: System\nDemonstrations , pages 121\u2013126.\nTal Linzen, Emmanuel Dupoux, and Yoav Goldberg.\n2016. Assessing the ability of LSTMs to learn\nsyntax-sensitive dependencies. Transactions of the\nAssociation for Computational Linguistics , 4:521\u2013\n535.\nDanish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. 2019. Learning\nto deceive with attention-based explanations. arXiv\npreprint arXiv:1909.07913 .Alec Radford, Jeffrey Wu, Rewon Child, David Luan,\nDario Amodei, and Ilya Sutskever. 2019. Language\nmodels are unsupervised multitask learners. OpenAI\nBlog , 1(8).\nTim Rockt \u00a8aschel, Edward Grefenstette, Karl Moritz\nHermann, Tomas Kocisky, and Phil Blunsom. 2016.\nReasoning about entailment with neural attention.\nInInternational Conference on Learning Representations (ICLR) .\nVictor Sanh, Lysandre Debut, Julien Chaumond, and\nThomas Wolf. 2019. Distilbert, a distilled version of\nbert: smaller, faster, cheaper and lighter."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f95-6db4-11ee-962b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-24", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Thomas Wolf. 2019. Distilbert, a distilled version of\nbert: smaller, faster, cheaper and lighter.\nSo\ufb01a Serrano and Noah A. Smith. 2019. Is attention\ninterpretable? In proceedings of the 57th Annual\nMeeting of the Association for Computational Linguistics . Association for Computational Linguistics.\nRichard Socher, Alex Perelygin, Jean Wu, Jason\nChuang, Christopher D. Manning, Andrew Ng, and\nChristopher Potts. 2013. Recursive deep models\nfor semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on\nEmpirical Methods in Natural Language Processing ,\npages 1631\u20131642, Seattle, Washington, USA. Association for Computational Linguistics.\nShikhar Vashishth, Shyam Upadhyay, Gaurav Singh\nTomar, and Manaal Faruqui. 2019. Attention interpretability across nlp tasks. arXiv preprint\narXiv:1909.11218 .\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob\nUszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz\nKaiser, and Illia Polosukhin. 2017. Attention is all\nyou need. In Advances in neural information processing systems , pages 5998\u20136008."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f96-6db4-11ee-a3fd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-25", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Kaiser, and Illia Polosukhin. 2017. Attention is all\nyou need. In Advances in neural information processing systems , pages 5998\u20136008.\nJesse Vig. 2019. Visualizing attention in transformerbased language models. arXiv preprint\narXiv:1904.02679 .\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.\nGLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353\u2013355, Brussels, Belgium.\nAssociation for Computational Linguistics.\nYequan Wang, Minlie Huang, Li Zhao, et al. 2016.\nAttention-based lstm for aspect-level sentiment classi\ufb01cation. In proceedings of the 2016 conference on\nempirical methods in natural language processing ,\npages 606\u2013615.\nSarah Wiegreffe and Yuval Pinter. 2019. Attention is\nnot not explanation. In proceedings of the 2019 Conference on Empirical Methods in Natural Language\nProcessing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP) . Association for Computational Linguistics.\nThomas Wolf, Lysandre Debut, Victor Sanh, Julien"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f97-6db4-11ee-b82f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-26", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "Thomas Wolf, Lysandre Debut, Victor Sanh, Julien\nChaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R\u2019emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface\u2019s transformers: State-of-the-art natural language processing. ArXiv , abs/1910.03771.\nKelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,\nAaron Courville, Ruslan Salakhudinov, Rich Zemel,\nand Yoshua Bengio. 2015. Show, attend and tell:\nNeural image caption generation with visual attention. In proceedings of International Conference on\nMachine Learning , pages 2048\u20132057.\nA Appendices\nA.1 Single Head Analysis\nFor analysing the attention weights, with multihead setup, we could either analyze attention heads\nseparately, or we could average all heads and have a\nsingle attention graph. However, we should be careful that treating attention heads separately could\npotentially mean that we are assuming there is no\nmixing of information between heads, which is not\ntrue as we combine information of heads in the\nposition-wise feed-forward network on top of selfattention in a transformer block. It is possible to\nanalyse the role of each head in isolation of all other"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f98-6db4-11ee-a6d1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.00928-27", "title": "Quantifying Attention Flow in Transformers", "source": "http://arxiv.org/pdf/2005.00928", "start_index": 0, "chroma:document": "position-wise feed-forward network on top of selfattention in a transformer block. It is possible to\nanalyse the role of each head in isolation of all other\nheads using attention rollout and attention \ufb02ow. To\nnot make the assumption that there is no mixing\nof information between heads, for computing the\n\u201cinput attention\u201d, we will treat all the layers below\nthe layer of interest as single head layers, i.e., we\nsum the attentions of all heads in the layers below.\nFor example, we can compute attention rollout for\nheadkat layer ias~A(i; k) =A(i; k)\u0016A(i), where,\n\u0016A(i)is attention rollout computed for layer iwith\nthe single head assumption."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f99-6db4-11ee-bd91-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-0", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "Language (Technology) is Power: A Critical Survey of \u201cBias\u201d in NLP\nSu Lin Blodgett\nCollege of Information and Computer Sciences\nUniversity of Massachusetts Amherst\nblodgett@cs.umass.eduSolon Barocas\nMicrosoft Research\nCornell University\nsolon@microsoft.com\nHal Daum\u00e9 III\nMicrosoft Research\nUniversity of Maryland\nme@hal3.nameHanna Wallach\nMicrosoft Research\nwallach@microsoft.com\nAbstract\nWe survey 146 papers analyzing \u201cbias\u201d in\nNLP systems, \ufb01nding that their motivations\nare often vague, inconsistent, and lacking\nin normative reasoning, despite the fact that\nanalyzing \u201cbias\u201d is an inherently normative\nprocess. We further \ufb01nd that these papers\u2019\nproposed quantitative techniques for measuring or mitigating \u201cbias\u201d are poorly matched to\ntheir motivations and do not engage with the\nrelevant literature outside of NLP. Based on\nthese \ufb01ndings, we describe the beginnings of a\npath forward by proposing three recommendations that should guide work analyzing \u201cbias\u201d\nin NLP systems. These recommendations rest\non a greater recognition of the relationships\nbetween language and social hierarchies,\nencouraging researchers and practitioners\nto articulate their conceptualizations of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f9a-6db4-11ee-8453-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-1", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "in NLP systems. These recommendations rest\non a greater recognition of the relationships\nbetween language and social hierarchies,\nencouraging researchers and practitioners\nto articulate their conceptualizations of\n\u201cbias\u201d\u2014i.e., what kinds of system behaviors\nare harmful, in what ways, to whom, and why,\nas well as the normative reasoning underlying\nthese statements\u2014and to center work around\nthe lived experiences of members of communities affected by NLP systems, while interrogating and reimagining the power relations\nbetween technologists and such communities.\n1 Introduction\nA large body of work analyzing \u201cbias\u201d in natural\nlanguage processing (NLP) systems has emerged\nin recent years, including work on \u201cbias\u201d in embedding spaces (e.g., Bolukbasi et al., 2016a; Caliskan\net al., 2017; Gonen and Goldberg, 2019; May\net al., 2019) as well as work on \u201cbias\u201d in systems\ndeveloped for a breadth of tasks including language\nmodeling (Lu et al., 2018; Bordia and Bowman,2019), coreference resolution (Rudinger et al.,\n2018; Zhao et al., 2018a), machine translation (Vanmassenhove et al., 2018; Stanovsky et al., 2019),\nsentiment analysis (Kiritchenko and Mohammad,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f9b-6db4-11ee-82a2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-2", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "2018; Zhao et al., 2018a), machine translation (Vanmassenhove et al., 2018; Stanovsky et al., 2019),\nsentiment analysis (Kiritchenko and Mohammad,\n2018), and hate speech/toxicity detection (e.g.,\nPark et al., 2018; Dixon et al., 2018), among others.\nAlthough these papers have laid vital groundwork by illustrating some of the ways that NLP\nsystems can be harmful, the majority of them fail\nto engage critically with what constitutes \u201cbias\u201d\nin the \ufb01rst place. Despite the fact that analyzing\n\u201cbias\u201d is an inherently normative process\u2014in\nwhich some system behaviors are deemed good\nand others harmful\u2014papers on \u201cbias\u201d in NLP\nsystems are rife with unstated assumptions about\nwhat kinds of system behaviors are harmful, in\nwhat ways, to whom, and why. Indeed, the term\n\u201cbias\u201d (or \u201cgender bias\u201d or \u201cracial bias\u201d) is used\nto describe a wide range of system behaviors, even\nthough they may be harmful in different ways, to\ndifferent groups, or for different reasons. Even\npapers analyzing \u201cbias\u201d in NLP systems developed\nfor the same task often conceptualize it differently.\nFor example, the following system behaviors"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f9c-6db4-11ee-b311-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-3", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "papers analyzing \u201cbias\u201d in NLP systems developed\nfor the same task often conceptualize it differently.\nFor example, the following system behaviors\nare all understood to be self-evident statements of\n\u201cracial bias\u201d: (a) embedding spaces in which embeddings for names associated with African Americans\nare closer (compared to names associated with\nEuropean Americans) to unpleasant words than\npleasant words (Caliskan et al., 2017); (b) sentiment analysis systems yielding different intensity\nscores for sentences containing names associated\nwith African Americans and sentences containing\nnames associated with European Americans (Kiritchenko and Mohammad, 2018); and (c) toxicityarXiv:2005.14050v2  [cs.CL]  29 May 2020\ndetection systems scoring tweets containing features associated with African-American English as\nmore offensive than tweets without these features\n(Davidson et al., 2019; Sap et al., 2019). Moreover,\nsome of these papers focus on \u201cracial bias\u201d\nexpressed in written text, while others focus on\n\u201cracial bias\u201d against authors. This use of imprecise\nterminology obscures these important differences.\nWe survey 146 papers analyzing \u201cbias\u201d in NLP\nsystems, \ufb01nding that their motivations are often\nvague and inconsistent. Many lack any normative"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f9d-6db4-11ee-b8ee-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-4", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "We survey 146 papers analyzing \u201cbias\u201d in NLP\nsystems, \ufb01nding that their motivations are often\nvague and inconsistent. Many lack any normative\nreasoning for why the system behaviors that are\ndescribed as \u201cbias\u201d are harmful, in what ways, and\nto whom. Moreover, the vast majority of these\npapers do not engage with the relevant literature\noutside of NLP to ground normative concerns when\nproposing quantitative techniques for measuring\nor mitigating \u201cbias.\u201d As a result, we \ufb01nd that many\nof these techniques are poorly matched to their\nmotivations, and are not comparable to one another.\nWe then describe the beginnings of a path\nforward by proposing three recommendations\nthat should guide work analyzing \u201cbias\u201d in NLP\nsystems. We argue that such work should examine\nthe relationships between language and social hierarchies; we call on researchers and practitioners\nconducting such work to articulate their conceptualizations of \u201cbias\u201d in order to enable conversations\nabout what kinds of system behaviors are harmful,\nin what ways, to whom, and why; and we recommend deeper engagements between technologists\nand communities affected by NLP systems. We\nalso provide several concrete research questions\nthat are implied by each of our recommendations.\n2 Method\nOur survey includes all papers known to us"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f9e-6db4-11ee-9195-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-5", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "and communities affected by NLP systems. We\nalso provide several concrete research questions\nthat are implied by each of our recommendations.\n2 Method\nOur survey includes all papers known to us\nanalyzing \u201cbias\u201d in NLP systems\u2014146 papers in\ntotal. We omitted papers about speech, restricting\nour survey to papers about written text only. To\nidentify the 146 papers, we \ufb01rst searched the ACL\nAnthology1for all papers with the keywords \u201cbias\u201d\nor \u201cfairness\u201d that were made available prior to May\n2020. We retained all papers about social \u201cbias,\u201d\nand discarded all papers about other de\ufb01nitions of\nthe keywords (e.g., hypothesis-only bias, inductive\nbias, media bias). We also discarded all papers using \u201cbias\u201d in NLP systems to measure social \u201cbias\u201d\nin text or the real world (e.g., Garg et al., 2018).\nTo ensure that we did not exclude any relevant\n1https://www.aclweb.org/anthology/NLP task Papers\nEmbeddings (type-level or contextualized) 54\nCoreference resolution 20\nLanguage modeling or dialogue generation 17\nHate-speech detection 17\nSentiment analysis 15\nMachine translation 8\nTagging or parsing 5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9f9f-6db4-11ee-b648-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-6", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "Coreference resolution 20\nLanguage modeling or dialogue generation 17\nHate-speech detection 17\nSentiment analysis 15\nMachine translation 8\nTagging or parsing 5\nSurveys, frameworks, and meta-analyses 20\nOther 22\nTable 1: The NLP tasks covered by the 146 papers.\npapers without the keywords \u201cbias\u201d or \u201cfairness,\u201d\nwe also traversed the citation graph of our initial\nset of papers, retaining any papers analyzing \u201cbias\u201d\nin NLP systems that are cited by or cite the papers\nin our initial set. Finally, we manually inspected\nany papers analyzing \u201cbias\u201d in NLP systems from\nleading machine learning, human\u2013computer interaction, and web conferences and workshops, such\nas ICML, NeurIPS, AIES, FAccT, CHI, and WWW,\nalong with any relevant papers that were made\navailable in the \u201cComputation and Language\u201d and\n\u201cComputers and Society\u201d categories on arXiv prior\nto May 2020, but found that they had already been\nidenti\ufb01ed via our traversal of the citation graph. We\nprovide a list of all 146 papers in the appendix. In\nTable 1, we provide a breakdown of the NLP tasks\ncovered by the papers. We note that counts do not"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa0-6db4-11ee-8591-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-7", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "provide a list of all 146 papers in the appendix. In\nTable 1, we provide a breakdown of the NLP tasks\ncovered by the papers. We note that counts do not\nsum to 146, because some papers cover multiple\ntasks. For example, a paper might test the ef\ufb01cacy\nof a technique for mitigating \u201cbias\u201d in embedding spaces in the context of sentiment analysis.\nOnce identi\ufb01ed, we then read each of the 146 papers with the goal of categorizing their motivations\nand their proposed quantitative techniques for measuring or mitigating \u201cbias.\u201d We used a previously\ndeveloped taxonomy of harms for this categorization, which differentiates between so-called allocational andrepresentational harms (Barocas et al.,\n2017; Crawford, 2017). Allocational harms arise\nwhen an automated system allocates resources (e.g.,\ncredit) or opportunities (e.g., jobs) unfairly to different social groups; representational harms arise\nwhen a system (e.g., a search engine) represents\nsome social groups in a less favorable light than\nothers, demeans them, or fails to recognize their\nexistence altogether. Adapting and extending this\ntaxonomy, we categorized the 146 papers\u2019 motivations and techniques into the following categories:\n.Allocational harms.\nPapers\nCategory Motivation Technique"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa1-6db4-11ee-9e59-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-8", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "taxonomy, we categorized the 146 papers\u2019 motivations and techniques into the following categories:\n.Allocational harms.\nPapers\nCategory Motivation Technique\nAllocational harms 30 4\nStereotyping 50 58\nOther representational harms 52 43\nQuestionable correlations 47 42\nVague/unstated 23 0\nSurveys, frameworks, and\nmeta-analyses20 20\nTable 2: The categories into which the 146 papers fall.\n.Representational harms:2\n.Stereotyping that propagates negative generalizations about particular social groups.\n.Differences in system performance for different social groups, language that misrepresents the distribution of different social\ngroups in the population, or language that\nisdenigrating to particular social groups.\n.Questionable correlations between system behavior and features of language that are typically associated with particular social groups.\n.Vague descriptions of \u201cbias\u201d (or \u201cgender\nbias\u201d or \u201cracial bias\u201d) or no description at all.\n.Surveys, frameworks, and meta-analyses .\nIn Table 2 we provide counts for each of the\nsix categories listed above. (We also provide a\nlist of the papers that fall into each category in the\nappendix.) Again, we note that the counts do not\nsum to 146, because some papers state multiple"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa2-6db4-11ee-98ec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-9", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "list of the papers that fall into each category in the\nappendix.) Again, we note that the counts do not\nsum to 146, because some papers state multiple\nmotivations, propose multiple techniques, or propose a single technique for measuring or mitigating\nmultiple harms. Table 3, which is in the appendix,\ncontains examples of the papers\u2019 motivations and\ntechniques across a range of different NLP tasks.\n3 Findings\nCategorizing the 146 papers\u2019 motivations and proposed quantitative techniques for measuring or mitigating \u201cbias\u201d into the six categories listed above enabled us to identify several commonalities, which\nwe present below, along with illustrative quotes.\n2We grouped several types of representational harms into\ntwo categories to re\ufb02ect that the main point of differentiation\nbetween the 146 papers\u2019 motivations and proposed quantitative\ntechniques for measuring or mitigating \u201cbias\u201d is whether or not\nthey focus on stereotyping. Among the papers that do not focus on stereotyping, we found that most lack suf\ufb01ciently clear\nmotivations and techniques to reliably categorize them further.3.1 Motivations\nPapers state a wide range of motivations,\nmultiple motivations, vague motivations, and\nsometimes no motivations at all. We found that\nthe papers\u2019 motivations span all six categories, with\nseveral papers falling into each one. Appropriately,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa3-6db4-11ee-98b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-10", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "sometimes no motivations at all. We found that\nthe papers\u2019 motivations span all six categories, with\nseveral papers falling into each one. Appropriately,\npapers that provide surveys or frameworks for analyzing \u201cbias\u201d in NLP systems often state multiple\nmotivations (e.g., Hovy and Spruit, 2016; Bender,\n2019; Sun et al., 2019; Rozado, 2020; Shah et al.,\n2020). However, as the examples in Table 3 (in the\nappendix) illustrate, many other papers (33%) do\nso as well. Some papers (16%) state only vague\nmotivations or no motivations at all. For example,\n\u201c[N]o human should be discriminated on the basis\nof demographic attributes by an NLP system. \u201d\n\u2014Kaneko and Bollegala (2019)\n\u201c[P]rominent word embeddings [...] encode\nsystematic biases against women and black people\n[...] implicating many NLP systems in scaling up\nsocial injustice. \u201d \u2014May et al. (2019)\nThese examples leave unstated what it might mean\nfor an NLP system to \u201cdiscriminate,\u201d what constitutes \u201csystematic biases,\u201d or how NLP systems\ncontribute to \u201csocial injustice\u201d (itself unde\ufb01ned)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa4-6db4-11ee-801d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-11", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "contribute to \u201csocial injustice\u201d (itself unde\ufb01ned).\nPapers\u2019 motivations sometimes include no normative reasoning. We found that some papers\n(32%) are not motivated by any apparent normative\nconcerns, often focusing instead on concerns about\nsystem performance. For example, the \ufb01rst quote\nbelow includes normative reasoning\u2014namely that\nmodels should not use demographic information\nto make predictions\u2014while the other focuses on\nlearned correlations impairing system performance.\n\u201cIn [text classi\ufb01cation], models are expected to\nmake predictions with the semantic information\nrather than with the demographic group identity\ninformation ( e.g., \u2018gay\u2019, \u2018black\u2019) contained in the\nsentences. \u201d \u2014Zhang et al. (2020a)\n\u201cAn over-prevalence of some gendered forms in the\ntraining data leads to translations with identi\ufb01able\nerrors. Translations are better for sentences\ninvolving men and for sentences containing\nstereotypical gender roles. \u201d\n\u2014Saunders and Byrne (2020)\nEven when papers do state clear motivations,\nthey are often unclear about why the system behaviors that are described as \u201cbias\u201d are harmful, in what ways, and to whom. We found that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa5-6db4-11ee-b1fd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-12", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "they are often unclear about why the system behaviors that are described as \u201cbias\u201d are harmful, in what ways, and to whom. We found that\neven papers with clear motivations often fail to explain what kinds of system behaviors are harmful,\nin what ways, to whom, and why. For example,\n\u201cDeploying these word embedding algorithms in\npractice, for example in automated translation\nsystems or as hiring aids, runs the serious risk of\nperpetuating problematic biases in important\nsocietal contexts. \u201d \u2014Brunet et al. (2019)\n\u201c[I]f the systems show discriminatory behaviors in\nthe interactions, the user experience will be\nadversely affected. \u201d \u2014Liu et al. (2019)\nThese examples leave unstated what \u201cproblematic\nbiases\u201d or non-ideal user experiences might look\nlike, how the system behaviors might result in\nthese things, and who the relevant stakeholders\nor users might be. In contrast, we \ufb01nd that papers\nthat provide surveys or frameworks for analyzing\n\u201cbias\u201d in NLP systems often name who is harmed,\nacknowledging that different social groups may\nexperience these systems differently due to their\ndifferent relationships with NLP systems or\ndifferent social positions. For example, Ruane\net al. (2019) argue for a \u201cdeep understanding of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa6-6db4-11ee-8102-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-13", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "different relationships with NLP systems or\ndifferent social positions. For example, Ruane\net al. (2019) argue for a \u201cdeep understanding of\nthe user groups [sic] characteristics, contexts, and\ninterests\u201d when designing conversational agents.\nPapers about NLP systems developed for the\nsame task often conceptualize \u201cbias\u201d differently. Even papers that cover the same NLP task\noften conceptualize \u201cbias\u201d in ways that differ substantially and are sometimes inconsistent. Rows 3\nand 4 of Table 3 (in the appendix) contain machine\ntranslation papers with different conceptualizations\nof \u201cbias,\u201d leading to different proposed techniques,\nwhile rows 5 and 6 contain papers on \u201cbias\u201d in embedding spaces that state different motivations, but\npropose techniques for quantifying stereotyping.\nPapers\u2019 motivations con\ufb02ate allocational and\nrepresentational harms. We found that the papers\u2019 motivations sometimes (16%) name immediate representational harms, such as stereotyping,\nalongside more distant allocational harms, which,\nin the case of stereotyping, are usually imagined as\ndownstream effects of stereotypes on r\u00e9sum\u00e9 \ufb01ltering. Many of these papers use the imagined downstream effects to justify focusing on particular system behaviors, even when the downstream effects\nare not measured. Papers on \u201cbias\u201d in embedding"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa7-6db4-11ee-93b0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-14", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "are not measured. Papers on \u201cbias\u201d in embedding\nspaces are especially likely to do this because embeddings are often used as input to other systems:\n\u201cHowever, none of these papers [on embeddings]\nhave recognized how blatantly sexist the\nembeddings are and hence risk introducing biases\nof various types into real-world systems. \u201d\n\u2014Bolukbasi et al. (2016a)\u201cIt is essential to quantify and mitigate gender bias\nin these embeddings to avoid them from affecting\ndownstream applications. \u201d \u2014Zhou et al. (2019)\nIn contrast, papers that provide surveys or frameworks for analyzing \u201cbias\u201d in NLP systems treat\nrepresentational harms as harmful in their own\nright. For example, May\ufb01eld et al. (2019) and\nRuane et al. (2019) cite the harmful reproduction\nof dominant linguistic norms by NLP systems (a\npoint to which we return in section 4), while Bender\n(2019) outlines a range of harms, including seeing\nstereotypes in search results and being made invisible to search engines due to language practices.\n3.2 Techniques\nPapers\u2019 techniques are not well grounded in the\nrelevant literature outside of NLP. Perhaps unsurprisingly given that the papers\u2019 motivations are\noften vague, inconsistent, and lacking in normative"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa8-6db4-11ee-9117-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-15", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "Papers\u2019 techniques are not well grounded in the\nrelevant literature outside of NLP. Perhaps unsurprisingly given that the papers\u2019 motivations are\noften vague, inconsistent, and lacking in normative\nreasoning, we also found that the papers\u2019 proposed\nquantitative techniques for measuring or mitigating\n\u201cbias\u201d do not effectively engage with the relevant\nliterature outside of NLP. Papers on stereotyping\nare a notable exception: the Word Embedding\nAssociation Test (Caliskan et al., 2017) draws on\nthe Implicit Association Test (Greenwald et al.,\n1998) from the social psychology literature, while\nseveral techniques operationalize the well-studied\n\u201cAngry Black Woman\u201d stereotype (Kiritchenko\nand Mohammad, 2018; May et al., 2019; Tan\nand Celis, 2019) and the \u201cdouble bind\u201d faced by\nwomen (May et al., 2019; Tan and Celis, 2019), in\nwhich women who succeed at stereotypically male\ntasks are perceived to be less likable than similarly\nsuccessful men (Heilman et al., 2004). Tan and\nCelis (2019) also examine the compounding effects\nof race and gender, drawing on Black feminist\nscholarship on intersectionality (Crenshaw, 1989).\nPapers\u2019 techniques are poorly matched to their"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fa9-6db4-11ee-a2d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-16", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "of race and gender, drawing on Black feminist\nscholarship on intersectionality (Crenshaw, 1989).\nPapers\u2019 techniques are poorly matched to their\nmotivations. We found that although 21% of the\npapers include allocational harms in their motivations, only four papers actually propose techniques\nfor measuring or mitigating allocational harms.\nPapers focus on a narrow range of potential\nsources of \u201cbias.\u201d We found that nearly all of the\npapers focus on system predictions as the potential\nsources of \u201cbias,\u201d with many additionally focusing\non \u201cbias\u201d in datasets (e.g., differences in the\nnumber of gendered pronouns in the training data\n(Zhao et al., 2019)). Most papers do not interrogate\nthe normative implications of other decisions made\nduring the development and deployment lifecycle\u2014\nperhaps unsurprising given that their motivations\nsometimes include no normative reasoning. A\nfew papers are exceptions, illustrating the impacts\nof task de\ufb01nitions, annotation guidelines, and\nevaluation metrics: Cao and Daum\u00e9 (2019) study\nhow folk conceptions of gender (Keyes, 2018) are\nreproduced in coreference resolution systems that\nassume a strict gender dichotomy, thereby maintaining cisnormativity; Sap et al. (2019) focus on\nthe effect of priming annotators with information"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9faa-6db4-11ee-bc54-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-17", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "assume a strict gender dichotomy, thereby maintaining cisnormativity; Sap et al. (2019) focus on\nthe effect of priming annotators with information\nabout possible dialectal differences when asking\nthem to apply toxicity labels to sample tweets, \ufb01nding that annotators who are primed are signi\ufb01cantly\nless likely to label tweets containing features associated with African-American English as offensive.\n4 A path forward\nWe now describe how researchers and practitioners\nconducting work analyzing \u201cbias\u201d in NLP systems\nmight avoid the pitfalls presented in the previous\nsection\u2014the beginnings of a path forward. We\npropose three recommendations that should guide\nsuch work, and, for each, provide several concrete\nresearch questions. We emphasize that these questions are not comprehensive, and are intended to\ngenerate further questions and lines of engagement.\nOur three recommendations are as follows:\n(R1) Ground work analyzing \u201cbias\u201d in NLP systems in the relevant literature outside of NLP\nthat explores the relationships between language and social hierarchies. Treat representational harms as harmful in their own right.\n(R2) Provide explicit statements of why the\nsystem behaviors that are described as \u201cbias\u201d\nare harmful, in what ways, and to whom.\nBe forthright about the normative reasoning\n(Green, 2019) underlying these statements."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fab-6db4-11ee-9fd7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-18", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "are harmful, in what ways, and to whom.\nBe forthright about the normative reasoning\n(Green, 2019) underlying these statements.\n(R3) Examine language use in practice by engaging with the lived experiences of members of\ncommunities affected by NLP systems. Interrogate and reimagine the power relations between technologists and such communities.\n4.1 Language and social hierarchies\nTurning \ufb01rst to (R1) , we argue that work analyzing\n\u201cbias\u201d in NLP systems will paint a much fuller picture if it engages with the relevant literature outside\nof NLP that explores the relationships betweenlanguage and social hierarchies. Many disciplines,\nincluding sociolinguistics, linguistic anthropology,\nsociology, and social psychology, study how\nlanguage takes on social meaning and the role that\nlanguage plays in maintaining social hierarchies.\nFor example, language is the means through which\nsocial groups are labeled and one way that beliefs\nabout social groups are transmitted (e.g., Maass,\n1999; Beukeboom and Burgers, 2019). Group\nlabels can serve as the basis of stereotypes and thus\nreinforce social inequalities: \u201c[T]he label content\nfunctions to identify a given category of people,\nand thereby conveys category boundaries and a\nposition in a hierarchical taxonomy\u201d (Beukeboom"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fac-6db4-11ee-84db-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-19", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "functions to identify a given category of people,\nand thereby conveys category boundaries and a\nposition in a hierarchical taxonomy\u201d (Beukeboom\nand Burgers, 2019). Similarly, \u201ccontrolling\nimages,\u201d such as stereotypes of Black women,\nwhich are linguistically and visually transmitted\nthrough literature, news media, television, and so\nforth, provide \u201cideological justi\ufb01cation\u201d for their\ncontinued oppression (Collins, 2000, Chapter 4).\nAs a result, many groups have sought to bring\nabout social changes through changes in language,\ndisrupting patterns of oppression and marginalization via so-called \u201cgender-fair\u201d language\n(Sczesny et al., 2016; Menegatti and Rubini, 2017),\nlanguage that is more inclusive to people with\ndisabilities (ADA, 2018), and language that is less\ndehumanizing (e.g., abandoning the use of the term\n\u201cillegal\u201d in everyday discourse on immigration in\nthe U.S. (Rosa, 2019)). The fact that group labels\nare so contested is evidence of how deeply intertwined language and social hierarchies are. Taking\n\u201cgender-fair\u201d language as an example, the hope\nis that reducing asymmetries in language about\nwomen and men will reduce asymmetries in their"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fad-6db4-11ee-8293-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-20", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "\u201cgender-fair\u201d language as an example, the hope\nis that reducing asymmetries in language about\nwomen and men will reduce asymmetries in their\nsocial standing. Meanwhile, struggles over language use often arise from dominant social groups\u2019\ndesire to \u201ccontrol both material and symbolic\nresources\u201d\u2014i.e., \u201cthe right to decide what words\nwill mean and to control those meanings\u201d\u2014as was\nthe case in some white speakers\u2019 insistence on\nusing offensive place names against the objections\nof Indigenous speakers (Hill, 2008, Chapter 3).\nSociolinguists and linguistic anthropologists\nhave also examined language attitudes and language ideologies, or people\u2019s metalinguistic beliefs\nabout language: Which language varieties or practices are taken as standard, ordinary, or unmarked?\nWhich are considered correct, prestigious, or appropriate for public use, and which are considered\nincorrect, uneducated, or offensive (e.g., CampbellKibler, 2009; Preston, 2009; Loudermilk, 2015;\nLanehart and Malik, 2018)? Which are rendered invisible (Roche, 2019)?3Language ideologies play\na vital role in reinforcing and justifying social hierarchies because beliefs about language varieties\nor practices often translate into beliefs about their\nspeakers (e.g. Alim et al., 2016; Rosa and Flores,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9fae-6db4-11ee-8660-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-21", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "or practices often translate into beliefs about their\nspeakers (e.g. Alim et al., 2016; Rosa and Flores,\n2017; Craft et al., 2020). For example, in the U.S.,\nthe portrayal of non-white speakers\u2019 language\nvarieties and practices as linguistically de\ufb01cient\nhelped to justify violent European colonialism, and\ntoday continues to justify enduring racial hierarchies by maintaining views of non-white speakers\nas lacking the language \u201crequired for complex\nthinking processes and successful engagement\nin the global economy\u201d (Rosa and Flores, 2017).\nRecognizing the role that language plays in\nmaintaining social hierarchies is critical to the\nfuture of work analyzing \u201cbias\u201d in NLP systems.\nFirst, it helps to explain why representational\nharms are harmful in their own right. Second, the\ncomplexity of the relationships between language\nand social hierarchies illustrates why studying\n\u201cbias\u201d in NLP systems is so challenging, suggesting\nthat researchers and practitioners will need to move\nbeyond existing algorithmic fairness techniques.\nWe argue that work must be grounded in the\nrelevant literature outside of NLP that examines\nthe relationships between language and social\nhierarchies; without this grounding, researchers\nand practitioners risk measuring or mitigating\nonly what is convenient to measure or mitigate,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c9faf-6db4-11ee-a2b4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14050-22", "title": "Language (Technology) is Power: A Critical Survey of \"Bias\" in NLP", "source": "http://arxiv.org/pdf/2005.14050", "start_index": 0, "chroma:document": "the relationships between language and social\nhierarchies; without this grounding, researchers\nand practitioners risk measuring or mitigating\nonly what is convenient to measure or mitigate,\nrather than what is most normatively concerning.\nMore speci\ufb01cally, we recommend that work\nanalyzing \u201cbias\u201d in NLP systems be reoriented\naround the following question: How are social\nhierarchies, language ideologies, and NLP systems\ncoproduced? This question mirrors Benjamin\u2019s\n(2020) call to examine how \u201crace and technology\nare coproduced\u201d\u2014i.e., how racial hierarchies, and\nthe ideologies and discourses that maintain them,\ncreate and are re-created by technology. We recommend that researchers and practitioners similarly\nask how existing social hierarchies and language\nideologies drive the development and deployment\nof NLP systems, and how these systems therefore\nreproduce these hierarchies and ideologies. As\na starting point for reorienting work analyzing\n\u201cbias\u201d in NLP systems around this question, we\n3Language ideologies encompass much more than this; see,\ne.g., Lippi-Green (2012), Alim et al. (2016), Rosa and Flores\n(2017), Rosa and Burdick (2017), and Charity Hudley (2017).provide the following concrete research questions:"}


