# embeddings_queue

created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd3fa-6db4-11ee-b555-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.01261-1024", "title": "Relational inductive biases, deep learning, and graph networks", "source": "http://arxiv.org/pdf/1806.01261", "start_index": 0, "chroma:document": "E0\ni\u0001:= =1\njE0\nijX\nfk:rk=ige0\nk\n38\nAttention-based approaches\nThe various attention-based approaches use a \u001eewhich is factored into a scalar pairwise-interaction\nfunction which returns the unnormalized attention term, denoted \u000be(vrk;vsk)=a0\nk, and a vectorvalued non-pairwise term, denoted \fe(vsk) =b0\nk,\n\u001ee(ek;vrk;vsk;u):=fe(vrk;vsk) = (\u000be(vrk;vsk); \fe(vsk)) = (a0\nk;b0\nk) =e0\nk\nThe single-headed self-attention (SA) in the Transformer architecture (Vaswani et al., 2017),\nimplements the non-local formulation as:\n\u000be(vrk;vsk) = exp (NN \u000bquery(vrk)|\u0001NN\u000bkey(vsk))\n\fe(vsk) = NN \f(vsk)\n\u001ev\u0000\n\u0016 e0\ni;vi;u\u0001:=fv\u0000\n\u0016 e0\ni\u0001\n= NNv\u0000\n\u0016 e0\ni\u0001"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd3fb-6db4-11ee-afd0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.01261-1025", "title": "Relational inductive biases, deep learning, and graph networks", "source": "http://arxiv.org/pdf/1806.01261", "start_index": 0, "chroma:document": "v\u0000\n\u0016 e0\ni;vi;u\u0001:=fv\u0000\n\u0016 e0\ni\u0001\n= NNv\u0000\n\u0016 e0\ni\u0001\nwhere NN\u000bquery,NN\u000bkey, and NN\fare again neural network functions with di\u000berent parameters and\npossibly di\u000berent architectures. They also use a multi-headed version which computes Nhparallel\n\u0016 e0h\niusing di\u000berent NN\u000bquery\nh,NN\u000bkey\nh,NN\fh, wherehindexes the di\u000berent parameters. These are\npassed tofvand concatenated:\nfv\u0010\nf\u0016 e0h\nigh=1:::Nh\u0011\n= NNv\u0010\n[\u0016 e01\ni;:::;\u0016 e0Nh\ni]\u0011\nVertex Attention Interaction Networks (Hoshen, 2017) are very similar to single-headed SA,\nbut use Euclidean distance for the attentional similarity metric, with shared parameters across the\nattention inputs' embeddings, and also use the input node feature in the node update function,\n\u000be(vrk;vsk) = exp\u0000\n\u0000kNN\u000b(vrk)\u0000NN\u000b(vsk)k2\u0001\n\fe(vsk) = NN \f(vsk)\n\u001ev\u0000"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd3fc-6db4-11ee-858e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.01261-1026", "title": "Relational inductive biases, deep learning, and graph networks", "source": "http://arxiv.org/pdf/1806.01261", "start_index": 0, "chroma:document": "\u0000kNN\u000b(vrk)\u0000NN\u000b(vsk)k2\u0001\n\fe(vsk) = NN \f(vsk)\n\u001ev\u0000\n\u0016 e0\ni;vi;u\u0001:=fv\u0000\n\u0016 e0\ni\u0001\n= NNv\u0000\n[\u0016 e0\ni;vi]\u0001\nGraph Attention Networks (Veli\u0014 ckovi\u0013 c et al., 2018) are also similar to multi-headed SA, but use\na neural network as the attentional similarity metric, with shared parameters across the attention\ninputs' embeddings:\n\u000be(vrk;vsk) = exp (NN \u000b0([NN\u000b(vrk);NN\u000b(vsk)))\n\fe(vsk) = NN \f(vsk)\n\u001ev\u0000\n\u0016 e0\ni;vi;u\u0001:=fv\u0010\nf\u0016 e0h\nigh=1:::Nh\u0011\n= NNv\u0010\n[\u0016 e01\ni;:::;\u0016 e0Nh\ni]\u0011\nStretching beyond the speci\fc non-local formulation, Shaw et al. (2018) extended multi-headed\nSA with relative position encodings. \\Relative\" refers to an encoding of the spatial distance between"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd3fd-6db4-11ee-84aa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.01261-1027", "title": "Relational inductive biases, deep learning, and graph networks", "source": "http://arxiv.org/pdf/1806.01261", "start_index": 0, "chroma:document": "SA with relative position encodings. \\Relative\" refers to an encoding of the spatial distance between\nnodes in a sequence or other signal in a metric space. This can be expressed in GN language as an\nedge attribute ek, and replacing the \fe(vsk) from multi-headed SA above with:\n\fe(ek;vsk) = NNe(vsk) +ek\n39\nBelief Propagation embeddings\nFinally, we brie\ry summarize how the general \\structure2vec\" algorithm of Dai et al. (2016) can \ft\ninto our framework. In order to do so, we need to slightly modify our main Equation 1, i.e.:\n\u0016\u000fk=\u001a\u0012\nfelgsl=rk\nrl6=sk\u0013\n:=X\nrl=sk\nsl6=rkel\ne0\nk=\u001ee(\u0016\u000fk) :=f(\u0016\u000fk) = NN( \u0016\u000fk)\n\u0016 e0\ni=\u001a\u0000\nfe0\nkgrk=i\u0001:=X\nfk:rk=igek\nv0\ni=\u001ev\u0000\n\u0016 e0\ni\u0001:=f(\u0016 e0\ni) = NN( \u0016 e0\ni)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd3fe-6db4-11ee-be12-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.01261-1028", "title": "Relational inductive biases, deep learning, and graph networks", "source": "http://arxiv.org/pdf/1806.01261", "start_index": 0, "chroma:document": "v0\ni=\u001ev\u0000\n\u0016 e0\ni\u0001:=f(\u0016 e0\ni) = NN( \u0016 e0\ni)\nEdges' features now takes the meaning of \\message\" between their receiver and sender; note that\nthere is only one set of parameters to learn for both the edges and nodes updates.\n40"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd3ff-6db4-11ee-b61d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-0", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "Object Level Visual Reasoning in Videos\nFabien Baradel1, Natalia Neverova2, Christian Wolf1;3,\nJulien Mille4, and Greg Mori5\n1Universit\u0013 e Lyon, INSA Lyon, CNRS, LIRIS, F-69621, Villeurbanne, France,\nfirstname.lastname@liris.cnrs.fr\n2Facebook AI Research, Paris, France, nneverova@fb.com\n3INRIA, CITI Laboratory, Villeurbanne, France\n4Laboratoire d'Informatique de l'Univ. de Tours, INSA Centre Val de Loire,\n41034, Blois, France, julien.mille@insa-cvl.fr\n5Simon Fraser University, Vancouver, Canada, mori@cs.sfu.ca\nhttps://fabienbaradel.github.io/eccv18_object_level_visual_reasoning/\nAbstract. Human activity recognition is typically addressed by detecting key concepts like global and local motion, features related to object\nclasses present in the scene, as well as features related to the global context. The next open challenges in activity recognition require a level of\nunderstanding that pushes beyond this and call for models with capabilities for \fne distinction and detailed comprehension of interactions"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd400-6db4-11ee-839e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-1", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "understanding that pushes beyond this and call for models with capabilities for \fne distinction and detailed comprehension of interactions\nbetween actors and objects in a scene. We propose a model capable of\nlearning to reason about semantically meaningful spatio-temporal interactions in videos. The key to our approach is a choice of performing\nthis reasoning at the object level through the integration of state of the\nart object detection networks. This allows the model to learn detailed\nspatial interactions that exist at a semantic, object-interaction relevant\nlevel. We evaluate our method on three standard datasets (Twenty-BN\nSomething-Something, VLOG and EPIC Kitchens) and achieve state of\nthe art results on all of them. Finally, we show visualizations of the interactions learned by the model, which illustrate object classes and their\ninteractions corresponding to di\u000berent activity classes.\nKeywords: Video understanding \u0001Human-object interaction\n1 Introduction\nThe \feld of video understanding is extremely diverse, ranging from extracting highly detailed information captured by speci\fcally designed motion capture systems [33] to making general sense of videos from the Web [1]. As in\nthe domain of image recognition, there exist a number of large-scale video\ndatasets [6,26,13,12,23,14], which allow the training of high-capacity deep learning models from massive amounts of data. These models enable detection of key"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd401-6db4-11ee-92ec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-2", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "datasets [6,26,13,12,23,14], which allow the training of high-capacity deep learning models from massive amounts of data. These models enable detection of key\ncues present in videos, such as global and local motion, various object categories\nand global scene-level information, and often achieve impressive performance in\nrecognizing high-level, abstract concepts in the wild.arXiv:1806.06157v3  [cs.CV]  20 Sep 2018\n2 Baradel et al.\nperson 0.86person 0.87knife 0.98carrot 0.99carrot 0.72carrot 0.98\nperson 0.65knife 0.99carrot 0.97carrot 0.97carrot 0.93carrot 0.86carrot 0.95\nFig. 1. Humans can understand what happened in a video (\\the leftmost carrot was\nchopped by the person\") given only a pair of frames. Along these lines, the goal of this\nwork is to explore the capabilities of higher-level reasoning in neural models operating\nat the semantic level of objects and interactions.\nHowever, recent attention has been directed toward a more thorough understanding of human-focused activity in diverse internet videos. These e\u000borts\nrange from atomic human actions [14] to \fne-grained object interactions [13]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd402-6db4-11ee-b9e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-3", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "range from atomic human actions [14] to \fne-grained object interactions [13]\nto everyday, commonly occurring human-object interactions [12]. This returns\nus to a human-centric viewpoint of activity recognition where it is not only the\npresence of certain objects / scenes that dictate the activity present, but the\nmanner, order, and e\u000bects of human interaction with these scene elements that\nare necessary for understanding. In a sense, this is akin to the problems in current\n3D human activity recognition datasets [33], but requires the more challenging\nreasoning and understanding of diverse environments common to internet video\ncollections.\nHumans are able to infer what happened in a video given only a few sample frames. In particular, they can infer complex activities happening between\npairs of frames. This faculty is called reasoning and is a key component of human intelligence. As an example we can consider the pair of images in Figure 1,\nwhich shows a complex situation involving articulated objects (human, carrots\nand knife), the change of location and composition of objects. For humans it is\nstraightforward to draw a conclusion on what happened (a carrot was chopped\nby the human). Humans have this extraordinary ability of performing visual reasoning on very complicated tasks while it remains unattainable for contemporary\ncomputer vision algorithms [37,11].\nThe ability to perform visual reasoning in computer vision algorithms is still"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd403-6db4-11ee-b264-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-4", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "computer vision algorithms [37,11].\nThe ability to perform visual reasoning in computer vision algorithms is still\nan open problem. Attempts have been made for learning interactions between\ndi\u000berent entities in images with promising results on Visual Question Answering.\nThere have been a number of attempts to equip neural models with reasoning\nabilities by training them to solve Visual Question Answering (VQA) problems.\nAmong proposed solutions are prior-less data normalization [27], structuring\nnetworks to model relationships [32,43] as well as more complex attention based\nObject Level Visual Reasoning in Videos 3\nmechanisms [18]. At the same time, it was shown that high performance on existing VQA datasets can be achieved by simply discovering biases in the data [21].\nWe extend these e\u000borts to object level reasoning in videos . Since a video is\na temporal sequence, we leverage time as an explicit causal signal to identify\ncausal object relations. Our approach is related to the concept of the \\arrow of\nthe time\" [28] involving the \\one-way direction\" or \\asymmetry\" of time. Causal\nevent occurs before the event it a\u000bects ( A!B). In Figure 1 the knife was used\nbefore the carrot switched over to the chopped-up state on the right side. For"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd404-6db4-11ee-a70b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-5", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "event occurs before the event it a\u000bects ( A!B). In Figure 1 the knife was used\nbefore the carrot switched over to the chopped-up state on the right side. For\na video classi\fcation problem, we want to identify a causal event Ahappening\nin a video that a\u000bects its label B. But instead of identifying this causal event\ndirectly from pixels we want to identify it from an object level perspective. We\nbelieve that such an approach would be able to learn causal signals.\nFollowing this hypothesis we propose to make a bridge between object detection and activity recognition. Object detection allows us to extract low-level\ninformation from a scene with all the present object instances and their semantic\nmeanings. However, detailed activity understanding requires reasoning over these\nsemantic structures, determining which objects were involved in interactions, of\nwhat nature, and what were the results of these. To compound problems, the\nsemantic structure of a scene may change during a video (e.g. a new object can\nappear, a person may make a move from one point to another one of the scene).\nWe propose an Object Relation Network (ORN), a neural network module for reasoning between detected semantic object instances through space and\ntime. The ORN has potential to address these issues and conduct relational\nreasoning over object interactions for the purpose of activity recognition. A set"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd405-6db4-11ee-af91-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-6", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "time. The ORN has potential to address these issues and conduct relational\nreasoning over object interactions for the purpose of activity recognition. A set\nof object detection masks ranging over di\u000berent object categories and temporal\noccurrences is input to the ORN. The ORN is able to infer pairwise relationships\nbetween objects detected at varying di\u000berent moments in time.\nCode and object masks predictions will be publicly available6.\n2 Related work\nAction Recognition. Action recognition has a long history in computer vision. Pre-deep learning approaches in action recognition focused on handcrafted\nspatio-temporal features including space-time interest points like SIFT-3D, HOG3D,\nIDT and aggregated them using bag-of-words techniques. Some hand-crafted representations, like dense trajectories [42], still give competitive performance and\nare frequently combined with deep learning.\nIn the recent past, work has shifted to deep learning. Early attempts adapt\n2D convolutional networks to videos through temporal pooling and 3D convolutions [2,40]. 3D convolutions are now widely adopted for activity recognition with\nthe introduction of feature transfer by in\rating pre-trained 2D convolutional kernels from image classi\fcation models trained on ImageNet/ILSVRC [31] through\n3D kernels [6]. The downside of 3D kernels is their computational complexity"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd406-6db4-11ee-b379-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-7", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "3D kernels [6]. The downside of 3D kernels is their computational complexity\n6https://github.com/fabienbaradel/object_level_visual_reasoning\n4 Baradel et al.\nand the large number of learnable parameters, leading to the introduction of\n2.5D kernels, i.e. separable \flters in the form of a 2D spatial kernel followed by\na temporal kernel [44]. An alternative to temporal convolutions are Recurrent\nNeural Networks (RNNs) in their various gated forms (GRUs, LSTMs) [17,9].\nKarpathy et al. [20] presented a wide study on di\u000berent ways of connecting\ninformation in spatial and temporal dimensions through convolutions and pooling. On very general datasets with coarse activity classes they have showed that\nthere was a small margin between classifying individual frames and classifying\nvideos with more sophisticated temporal aggregation.\nSimoyan et al. [35] proposed a widely adopted two-stream architecture for\naction recognition which extracts two di\u000berent streams, one processing raw RGB\ninput and one processing pre-computed optical \row images. The method outperformed the state of the art, but relies on rather small scale optical \row computations.\nIn slightly narrower settings, prior information on the video content can allow"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd407-6db4-11ee-bded-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-8", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "In slightly narrower settings, prior information on the video content can allow\nmore \fne-grained models. Articulated pose is widely used in cases where humans\nare guaranteed to be present [33]. Pose estimation and activity recognition as a\njoint (multi-task) problem has recently shown to improve both tasks [25]. Somewhat related to our work, Structural RNNs [19] perform activity recognition\nby integrating features from semantic objects and their relationships. However,\nthey handle the temporal evolution of tracked objects in videos with a set of\nRNNs, each of which corresponds to cliques in a graph which models the spatiotemporal relationships between these objects. This graph is hand-crafted manually for each application, though related work provides learnable connections\nvia gating functions [8].\nAttention models are a way to structure deep networks in an often generic\nway. They are able to iteratively focus attention to speci\fc parts in the data without requiring prior knowledge about part or object positions. In activity recognition, they have gained some traction in recent years, either as soft-attention\non articulated pose (joints) [36], on feature map cells [34,39], on time [45] or on\nparts in raw RGB input through di\u000berentiable crops [3].\nWhen raw video data is globally fed into deep neural networks, they focus"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd408-6db4-11ee-92d9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-9", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "parts in raw RGB input through di\u000berentiable crops [3].\nWhen raw video data is globally fed into deep neural networks, they focus\non extracting spatio-temporal features and perform aggregations. It has been\nshown that these techniques fail on challenging \fne-grained datasets, which require learning long temporal dependencies and human-object interactions. A\nconcentrated e\u000bort has been made to create large scale datasets to overcome\nthese issues [13,12,23,14].\nRelational Reasoning. Relational reasoning is a well studied \feld for many\napplications ranging from visual reasoning [32] to reasoning about physical\nsystems [4]. Battaglia et al. [4] introduce a fully-di\u000berentiable network physics\nengine called Interaction Network (IN). IN learns to predict several physical\nsystems such as gravitational systems, rigid body dynamics, and mass-spring\nsystems. It shows impressive results; however, it learns from a virtual environment, which provides access to virtually unlimited training examples. Following\nthe same perspective, Santoro et al. [32] introduced Relation Network (RN),\nObject Level Visual Reasoning in Videos 5\na plug-in module for reasoning in deep networks. RN shows human-level performance in Visual Question Answering (VQA) by inferring pairwise \\object\""}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd409-6db4-11ee-a5a9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-10", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "a plug-in module for reasoning in deep networks. RN shows human-level performance in Visual Question Answering (VQA) by inferring pairwise \\object\"\nrelations. However, in contrast to our work, the term \\object\" in [32] does not\nrefer to semantically meaningful entities, but to discrete cells in feature maps.\nThe number of interactions therefore grows with feature map resolutions, which\nmakes it di\u000ecult to scale. Furthermore, a recent study [21] has shown that some\nof these results are subject to dataset bias and do not generalize well to small\nchanges in the settings of the dataset.\nIn the same line, a recent work [38] has shown promising results on discovering objects and their interactions in an unsupervised manner using training\nexamples from virtual environments. In [41], attention and relational modules\nare combined on a graph structure. From a di\u000berent perspective, [27] show that\nrelational reasoning can be learned for visual reasoning in a data driven way without any prior using conditional batch normalization with a feature-wise a\u000ene\ntransformation based on conditioning information. In an opposite approach, a\nstrong structural prior is learned in the form of a complex attention mechanism: in [18], an external memory module combined with attention processes\nover input images and text questions, performing iterative reasoning for VQA."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd40a-6db4-11ee-9396-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-11", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "over input images and text questions, performing iterative reasoning for VQA.\nWhile most of the discussed work has been designed for VQA and for predictions on physical systems and environments, extensions have been proposed\nfor video understanding. Reasoning in videos on a mask or segmentation level\nhas been attempted for video prediction [24], where the goal was to leverage semantic information to be able predict further into the future. Zhou et al [5] have\nrecently shown state-of-the-art performance on challenging datasets by extending Relation Network to video classi\fcation. Their chosen entities are frames, on\nwhich they employ RN to reason on a temporal level only through pairwise frame\nrelations. The approach is promising, but restricted to temporal contextual information without an understanding on a local object level, which is provided\nby our approach.\nReasoning over sets of objects is somewhat related to reasoning from unstructured data points, as done in PointNet [29], designed to learn from unordered sets\nof points. PointNet shares many properties with DeepSet [46] which is a more\ngeneral framework for extracting information from sets of objects. To some extent, our work is related to PointNet, as we handle unordered sets of objects in a\npermutation invariant way. However, we have an object relation viewpoint that\ndirectly reasons over relationships between these semantic entities.\n3 Object-level Visual Reasoning in Space and Time"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd40b-6db4-11ee-a814-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-12", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "permutation invariant way. However, we have an object relation viewpoint that\ndirectly reasons over relationships between these semantic entities.\n3 Object-level Visual Reasoning in Space and Time\nOur goal is to extract multiple types of cues from a video sequence: interactions\nbetween predicted objects and their semantic classes, as well as local and global\nmotion in the scene. We formulate this objective as a neural architecture with\ntwo heads: an activity head and an object head . Figure 2 gives a functional\noverview of the model. Both heads share common features up to a certain layer\nshown in red in the \fgure. The activity head , shown in orange in the \fgure,\n6 Baradel et al.\nspatio-temporalblock"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd40c-6db4-11ee-9a36-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-13", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd40d-6db4-11ee-b42a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd40e-6db4-11ee-90a3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd40f-6db4-11ee-b9e9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd410-6db4-11ee-b6a0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd411-6db4-11ee-995f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd412-6db4-11ee-aa6e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd413-6db4-11ee-9b20-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd414-6db4-11ee-8a44-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd415-6db4-11ee-858d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-22", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "network<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd416-6db4-11ee-b495-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd417-6db4-11ee-a296-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd418-6db4-11ee-9248-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd419-6db4-11ee-ad49-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd41a-6db4-11ee-bf82-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd41b-6db4-11ee-a95d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd41c-6db4-11ee-a9da-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd41d-6db4-11ee-993a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd41e-6db4-11ee-bba5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-31", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "object mask detector<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd41f-6db4-11ee-92ed-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd420-6db4-11ee-b044-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd421-6db4-11ee-8921-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd422-6db4-11ee-99c7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd423-6db4-11ee-b67b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd424-6db4-11ee-b636-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd425-6db4-11ee-aa4d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd426-6db4-11ee-8ae6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd427-6db4-11ee-bb4f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-40", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "head<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd428-6db4-11ee-a3f7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd429-6db4-11ee-afde-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd42a-6db4-11ee-b21d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd42b-6db4-11ee-a4bd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd42c-6db4-11ee-a66a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd42d-6db4-11ee-8df7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd42e-6db4-11ee-b0c9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd42f-6db4-11ee-a19e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd430-6db4-11ee-851e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-49", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "head<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd431-6db4-11ee-b0a9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd432-6db4-11ee-b294-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd433-6db4-11ee-a0f5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd434-6db4-11ee-8c5a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd435-6db4-11ee-b8cc-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd436-6db4-11ee-84f5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd437-6db4-11ee-a4aa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd438-6db4-11ee-8a87-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd439-6db4-11ee-b22d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-58", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "reasoningmodule"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd43a-6db4-11ee-8f6c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-59", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd43b-6db4-11ee-b6a4-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd43c-6db4-11ee-ba99-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd43d-6db4-11ee-a44e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd43e-6db4-11ee-be3c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd43f-6db4-11ee-b2c0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd440-6db4-11ee-a115-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd441-6db4-11ee-bb14-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd442-6db4-11ee-be0f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd443-6db4-11ee-916a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-68", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "pool<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd444-6db4-11ee-ae16-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd445-6db4-11ee-892c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd446-6db4-11ee-b886-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd447-6db4-11ee-b417-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd448-6db4-11ee-9168-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd449-6db4-11ee-9faa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd44a-6db4-11ee-a22d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd44b-6db4-11ee-93b6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd44c-6db4-11ee-9272-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-77", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "loss<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd44d-6db4-11ee-a4c5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd44e-6db4-11ee-9ca1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd44f-6db4-11ee-8d25-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd450-6db4-11ee-a058-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd451-6db4-11ee-ad67-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd452-6db4-11ee-ad72-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd453-6db4-11ee-9b41-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd454-6db4-11ee-80fd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd455-6db4-11ee-b141-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-86", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "class loss"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd456-6db4-11ee-b2df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-87", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd457-6db4-11ee-9069-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd458-6db4-11ee-8070-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd459-6db4-11ee-9c4e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd45a-6db4-11ee-9f6b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd45b-6db4-11ee-837c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd45c-6db4-11ee-b5a7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd45d-6db4-11ee-9f8a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd45e-6db4-11ee-8677-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd45f-6db4-11ee-8f4f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-96", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "pairwisetemporal sampling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd460-6db4-11ee-9be0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-97", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd461-6db4-11ee-a3c5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd462-6db4-11ee-8b7a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd463-6db4-11ee-bfcb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd464-6db4-11ee-9eaa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd465-6db4-11ee-8f7e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd466-6db4-11ee-b6e9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd467-6db4-11ee-a0be-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd468-6db4-11ee-b8de-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd469-6db4-11ee-8665-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-106", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "activity features<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd46a-6db4-11ee-b9bf-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd46b-6db4-11ee-bc09-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd46c-6db4-11ee-97bb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd46d-6db4-11ee-a149-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd46e-6db4-11ee-b986-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd46f-6db4-11ee-a525-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd470-6db4-11ee-81b2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd471-6db4-11ee-be46-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd472-6db4-11ee-9b52-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-115", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "ofobject features"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd473-6db4-11ee-b68d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-116", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd474-6db4-11ee-8224-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd475-6db4-11ee-a151-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd476-6db4-11ee-a82b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-119", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "3t0zptkdskfeJ9f7xuoOQ==</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd477-6db4-11ee-8d5b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd478-6db4-11ee-b537-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd479-6db4-11ee-af7e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-122", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "3t0zptkdskfeJ9f7xuoOQ==</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd47a-6db4-11ee-b0ae-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd47b-6db4-11ee-bcb3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd47c-6db4-11ee-8919-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-125", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "3t0zptkdskfeJ9f7xuoOQ==</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd47d-6db4-11ee-861e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd47e-6db4-11ee-b27c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd47f-6db4-11ee-a915-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-128", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "n\u21e52Dsets ofobject masks"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd480-6db4-11ee-a886-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-129", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd481-6db4-11ee-9253-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd482-6db4-11ee-a310-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd483-6db4-11ee-abf5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-132", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "EdzziJzjP7A+foGUQGm2w==</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd484-6db4-11ee-9f64-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd485-6db4-11ee-8b71-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd486-6db4-11ee-a8c5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-135", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "EdzziJzjP7A+foGUQGm2w==</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd487-6db4-11ee-a726-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd488-6db4-11ee-958c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd489-6db4-11ee-8bac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-138", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "EdzziJzjP7A+foGUQGm2w==</latexit><latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd48a-6db4-11ee-9b1f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd48b-6db4-11ee-b397-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd48c-6db4-11ee-a548-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-141", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "EdzziJzjP7A+foGUQGm2w==</latexit>activity"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd48d-6db4-11ee-a3e0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-142", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "loss<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd48e-6db4-11ee-ba5c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd48f-6db4-11ee-bf5a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bd490-6db4-11ee-8728-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaae-6db4-11ee-a2b0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaaf-6db4-11ee-8367-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab0-6db4-11ee-91fb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab1-6db4-11ee-a74d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab2-6db4-11ee-aaff-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab3-6db4-11ee-bb26-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab4-6db4-11ee-92ee-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab5-6db4-11ee-95d2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab6-6db4-11ee-85e3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab7-6db4-11ee-a39a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab8-6db4-11ee-afda-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfab9-6db4-11ee-8a3f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaba-6db4-11ee-8548-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfabb-6db4-11ee-8eaa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfabc-6db4-11ee-b8fa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfabd-6db4-11ee-8a3a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfabe-6db4-11ee-8ec6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfabf-6db4-11ee-a0be-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac0-6db4-11ee-afc7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac1-6db4-11ee-80e2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac2-6db4-11ee-a3f8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac3-6db4-11ee-a2e4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-167", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "B<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac4-6db4-11ee-a334-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac5-6db4-11ee-992c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac6-6db4-11ee-8e55-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac7-6db4-11ee-92e1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac8-6db4-11ee-9232-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfac9-6db4-11ee-8a69-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaca-6db4-11ee-99a7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfacb-6db4-11ee-8aa3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfacc-6db4-11ee-b6fe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-176", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfacd-6db4-11ee-8700-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bface-6db4-11ee-8227-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfacf-6db4-11ee-a3e4-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad0-6db4-11ee-ac47-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad1-6db4-11ee-83d6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad2-6db4-11ee-b9bd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad3-6db4-11ee-b6e8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad4-6db4-11ee-9afb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad5-6db4-11ee-a8e2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-185", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad6-6db4-11ee-8a63-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad7-6db4-11ee-b05e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad8-6db4-11ee-bd3e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfad9-6db4-11ee-af79-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfada-6db4-11ee-acd4-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfadb-6db4-11ee-a1bb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfadc-6db4-11ee-a3b6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfadd-6db4-11ee-9726-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfade-6db4-11ee-8b9d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfadf-6db4-11ee-bae0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae0-6db4-11ee-ba73-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae1-6db4-11ee-a9ea-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae2-6db4-11ee-866f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae3-6db4-11ee-a4a5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae4-6db4-11ee-b7b9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae5-6db4-11ee-9a96-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae6-6db4-11ee-8c46-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae7-6db4-11ee-9ae3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae8-6db4-11ee-9f9a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfae9-6db4-11ee-a5aa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaea-6db4-11ee-a076-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaeb-6db4-11ee-a472-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaec-6db4-11ee-9e4d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaed-6db4-11ee-a1f0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaee-6db4-11ee-bf86-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-210", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "spatialpooling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaef-6db4-11ee-b53c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-211", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf0-6db4-11ee-8069-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf1-6db4-11ee-820a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf2-6db4-11ee-9daa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf3-6db4-11ee-a8af-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf4-6db4-11ee-9d9e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf5-6db4-11ee-a158-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf6-6db4-11ee-a7c6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf7-6db4-11ee-8b99-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf8-6db4-11ee-bbf5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaf9-6db4-11ee-9c22-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfafa-6db4-11ee-99fa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfafb-6db4-11ee-a7f0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfafc-6db4-11ee-997e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfafd-6db4-11ee-831b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfafe-6db4-11ee-8adb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfaff-6db4-11ee-8ba3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb00-6db4-11ee-a0f6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb01-6db4-11ee-acd4-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb02-6db4-11ee-aee9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb03-6db4-11ee-9a75-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb04-6db4-11ee-9911-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb05-6db4-11ee-bf07-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb06-6db4-11ee-b1b9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb07-6db4-11ee-bf78-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb08-6db4-11ee-a754-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-236", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "Fig. 2. A functional overview of the model. A global convolutional model extracts\nfeatures and splits into two heads trained to predict, respectively activity classes and\nobject classes. The latter are predicted by pooling over object instance masks, which\nare predicted by an additional convolutional model. The object instances are passed\nthrough a visual reasoning module.\nis a CNN-based architecture employing convolutional layers, including spatiotemporal convolutions, able to extract global motion features. However, it is not\nable to extract information from an object level perspective. We leverage the\nobject head to perform reasoning on the relationships between predicted object\ninstances.\nOur main contribution is a new structured module called Object Relation\nNetwork (ORN), which is able to perform spatio-temporal reasoning between\ndetected object instances in the video. ORN is able to reason by modeling how\nobjects move, appear and disappear and how they interact between two frames.\nIn this section, we will \frst describe our main contribution, the ORN network.\nWe then provide details about object instance features, about the activity head,\nand \fnally about the \fnal recognition task. In what follows, lowercase letters\ndenote 1D vectors while uppercase letters are used for 2D and 3D matrices or\nhigher order tensors. We assume that the input of our system is a video of T"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb09-6db4-11ee-bf85-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-237", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "denote 1D vectors while uppercase letters are used for 2D and 3D matrices or\nhigher order tensors. We assume that the input of our system is a video of T\nframes denoted by X1:T= (Xt)T\nt=1where Xtis the RGB image at timestep t.\nThe goal is to learn a mapping from X1:Tto activity classes y.\n3.1 Object Relation Network\nORN (Object Relation Network) is a module for reasoning between semantic\nobjects through space and time. It captures object moves, arrivals and interactions in an e\u000ecient manner. We suppose that for each frame t, we have a set\nof objects kwith associated features ok\nt. Objects and features are detected and\ncomputed by the object head described in Section 3.2.\nReasoning about activities in videos is inherently temporal, as activities follow the arrow of time [28], i.e. the causality of the time dimension imposes that\npast actions have consequences in the future but notvice-versa. We handle this\nby sampling: running a process over time t, and for each instant t, sampling a\nObject Level Visual Reasoning in Videos 7\nsecond frame t0with t0<t. Our network reasons on objects which interact between pairs of frames and their corresponding sets of objects Ot0=\b\nok\nt0\tK0"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb0a-6db4-11ee-ac64-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-238", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "second frame t0with t0<t. Our network reasons on objects which interact between pairs of frames and their corresponding sets of objects Ot0=\b\nok\nt0\tK0\nk=1\nandOt=\b\nok\nt\tK\nk=1. The goal is to learn a general function de\fned on the set of\nall input objects from the combined set of both frames:\ngt=g(o1\nt0; : : : ;oK0\nt0;o1\nt; : : : ;oK\nt): (1)\nThe objects in this set are unordered, aside for the frame they belong to.\nThis task is related to a problem raised in the PointNet algorithm [29] discussed in Section 2. PointNet approximates a general function gover an item\nsetS=fx1; x2; : : : ; xNgas a symmetric function g0on transformed elements of\nthe set\ng(x1; x2; : : : ; xN)\u0019g0(h(x1); h(x2); : : : ; h (xN)): (2)\nIn [29], g0is a max pooling operation. The authors show that it allows universal"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb0b-6db4-11ee-8007-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-239", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "In [29], g0is a max pooling operation. The authors show that it allows universal\napproximation of continuous sets of functions given that the hidden representation (the output of the mapping h(\u0001)) is of su\u000eciently high dimension.\nWe argue that the approximation in (2) can be extended as follows:\ng(x1; x2; : : : ; xN)\u0019f\u001e X\nc2Chc([i2cxi)!\n(3)\nwhereCis the set of cliques of a graph de\fned over the item set S,[is the\nconcatenation operator and we chose the sum operator as symmetric function.\nThe input dimension of the non-linearity hc(\u0001) depends on the size of clique c\nbut maps to a \fxed output dimension H. In the case where Cis composed of\nunary cliques only, form (3) decomposes like (2) with the exception of a di\u000berent\nsymmetry operator (sum instead of max pooling). Choosing di\u000berent graphical\nstructures through Swill lead to di\u000berent terms in the summation and allows\nmodeling di\u000berent types of interactions between items in the item set.\nNote that the interactions between items in (3) are not exclusively modeled"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb0c-6db4-11ee-b598-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-240", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "modeling di\u000berent types of interactions between items in the item set.\nNote that the interactions between items in (3) are not exclusively modeled\nthrough hc(\u0001). Indeed, it is interesting to note, that the graphical decomposition provided byCleads to interactions which are di\u000berent from the interactions\nthe same decomposition would provide when used in a probabilistic graphical\nmodel, like for instance a Markov Random Field (MRF). In particular, a decomposition into unary terms only, as given in equation (2), does notlead to\nindependence between items, whereas an MRF with unary terms only is equivalent to a distribution over independent random variables. This is a consequence\nof the global mapping f\u001e(\u0001), which is de\fned on the sum over all direct interactions. Higher-order interactions between several items not directly modeled\nthrough a non-linearity hc(\u0001) can eventually be learned by the model through\nthe joint output space of all hc(\u0001), provided that the dimensionality Hof this\nspace is high enough to incorporate all interactions. However, whereas the mappinghc(\u0001) provides a direct model of interactions between pairs of items, learning\ninteractions between two items ( j; k), which are not directly captured through a\n8 Baradel et al.\nvisual reasoningmodule"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb0d-6db4-11ee-9596-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-241", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb0e-6db4-11ee-bfef-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb0f-6db4-11ee-8715-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb10-6db4-11ee-a483-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb11-6db4-11ee-b542-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb12-6db4-11ee-b56b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb13-6db4-11ee-8332-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb14-6db4-11ee-866c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb15-6db4-11ee-a23a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb16-6db4-11ee-92d5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-250", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "reasoningmodule"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb17-6db4-11ee-894b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-251", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb18-6db4-11ee-a4b2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb19-6db4-11ee-a617-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb1a-6db4-11ee-8929-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb1b-6db4-11ee-8a5e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb1c-6db4-11ee-8e7e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb1d-6db4-11ee-8626-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb1e-6db4-11ee-9524-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb1f-6db4-11ee-9536-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb20-6db4-11ee-942d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-260", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "framet02[0...t\u00002]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb21-6db4-11ee-98d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-261", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb22-6db4-11ee-9ec6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb23-6db4-11ee-b320-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb24-6db4-11ee-a8b0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb25-6db4-11ee-bf11-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb26-6db4-11ee-9d2b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb27-6db4-11ee-b13f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb28-6db4-11ee-a90e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb29-6db4-11ee-999c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb2a-6db4-11ee-9bfe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-270", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "framet002[0...t\u00001]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb2b-6db4-11ee-83e4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-271", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb2c-6db4-11ee-a509-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb2d-6db4-11ee-a3f9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb2e-6db4-11ee-96bf-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb2f-6db4-11ee-8f7c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb30-6db4-11ee-8230-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb31-6db4-11ee-9d4d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb32-6db4-11ee-bd66-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb33-6db4-11ee-9b75-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb34-6db4-11ee-9e12-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-280", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb35-6db4-11ee-8053-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb36-6db4-11ee-a807-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb37-6db4-11ee-b0e1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb38-6db4-11ee-89d9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb39-6db4-11ee-a50e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb3a-6db4-11ee-9d47-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb3b-6db4-11ee-b0ee-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb3c-6db4-11ee-a23f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb3d-6db4-11ee-912e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-289", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb3e-6db4-11ee-aa63-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb3f-6db4-11ee-a010-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb40-6db4-11ee-976c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb41-6db4-11ee-a804-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb42-6db4-11ee-8cd9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb43-6db4-11ee-bd30-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb44-6db4-11ee-ae97-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb45-6db4-11ee-b72c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb46-6db4-11ee-9a5e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-298", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "stream<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb47-6db4-11ee-b6dc-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb48-6db4-11ee-bbba-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb49-6db4-11ee-b246-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb4a-6db4-11ee-b9dd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb4b-6db4-11ee-ab34-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb4c-6db4-11ee-9de2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb4d-6db4-11ee-918f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb4e-6db4-11ee-a321-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb4f-6db4-11ee-a58f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-307", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "RNN"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb50-6db4-11ee-8043-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-308", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb51-6db4-11ee-bf64-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb52-6db4-11ee-a92f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb53-6db4-11ee-9b0e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb54-6db4-11ee-9859-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb55-6db4-11ee-8846-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb56-6db4-11ee-af5e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb57-6db4-11ee-8159-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb58-6db4-11ee-9824-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb59-6db4-11ee-9a83-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-317", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb5a-6db4-11ee-9556-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb5b-6db4-11ee-88f9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb5c-6db4-11ee-a8c0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb5d-6db4-11ee-8cf9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb5e-6db4-11ee-ad99-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb5f-6db4-11ee-bf47-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb60-6db4-11ee-8e13-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb61-6db4-11ee-8ac2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb62-6db4-11ee-ab80-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-326", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb63-6db4-11ee-97b2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb64-6db4-11ee-8c91-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb65-6db4-11ee-9397-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb66-6db4-11ee-9886-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb67-6db4-11ee-ace6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb68-6db4-11ee-9455-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb69-6db4-11ee-b4f8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb6a-6db4-11ee-8f46-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb6b-6db4-11ee-8584-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-335", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1bfb6c-6db4-11ee-b7dd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21cd-6db4-11ee-b16d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21ce-6db4-11ee-8057-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21cf-6db4-11ee-ab07-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d0-6db4-11ee-8b2f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d1-6db4-11ee-b40a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d2-6db4-11ee-9996-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d3-6db4-11ee-9de2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d4-6db4-11ee-a211-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-344", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d5-6db4-11ee-bb4e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d6-6db4-11ee-98c4-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d7-6db4-11ee-a49a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d8-6db4-11ee-9786-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21d9-6db4-11ee-b578-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21da-6db4-11ee-8b1f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21db-6db4-11ee-87b5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21dc-6db4-11ee-aea2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21dd-6db4-11ee-a300-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-353", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21de-6db4-11ee-b28a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21df-6db4-11ee-9c91-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e0-6db4-11ee-b43e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e1-6db4-11ee-bee5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e2-6db4-11ee-ae79-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e3-6db4-11ee-a632-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e4-6db4-11ee-9bd5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e5-6db4-11ee-8c2d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e6-6db4-11ee-bf47-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e7-6db4-11ee-83a7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e8-6db4-11ee-9137-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21e9-6db4-11ee-90df-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21ea-6db4-11ee-b508-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21eb-6db4-11ee-bdb6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21ec-6db4-11ee-96a7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21ed-6db4-11ee-afcb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21ee-6db4-11ee-982b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21ef-6db4-11ee-9727-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f0-6db4-11ee-a3ca-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f1-6db4-11ee-8839-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f2-6db4-11ee-b7fb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f3-6db4-11ee-a99b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f4-6db4-11ee-a0b5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f5-6db4-11ee-a67e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f6-6db4-11ee-a8c7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-378", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "...<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f7-6db4-11ee-8e04-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f8-6db4-11ee-955f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21f9-6db4-11ee-8e13-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21fa-6db4-11ee-a1aa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21fb-6db4-11ee-bcb7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21fc-6db4-11ee-8fe3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21fd-6db4-11ee-ad31-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21fe-6db4-11ee-b023-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c21ff-6db4-11ee-af7f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2200-6db4-11ee-9355-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2201-6db4-11ee-89d6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2202-6db4-11ee-b2d6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2203-6db4-11ee-b2d5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2204-6db4-11ee-aeba-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2205-6db4-11ee-8d01-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2206-6db4-11ee-bb06-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2207-6db4-11ee-ab2d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-395", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "Fig. 3. ORN in the object head operating on detected instances of objects.\nclique cand its corresponding hc(\u0001), requires learning a corresponding subspace\nin the common output space spanned by all hc(\u0001).\nThis leads to the question of how to de\fne the trade-o\u000b between the complexity of the decomposition Cand the output dimensionality Hof the mapping\nhc(\u0001), both of which will determine the complexity of the modeled interactions.\nIncreasing the size of cliques in Cwill increase the input dimension (and therefore the capacity) of the mapping hc(\u0001) as well as the computational complexity\nof the sum operation.\nInspired by relational networks [32], we chose to directly model inter-frame\ninteractions between pairs of objects ( j; k) and leave modeling of higher-order\ninteractions to the output space of the mappings h\u0012and the global mapping f\u001e:\ngt=X\nj;kh\u0012(oj\nt0;ok\nt) (4)\nIn order to better directly model long-range interactions, we make the global\nmapping f\u001e(\u0001;\u0001) recurrent, which leads to the following form:\nrt=f\u001e(gt;rt\u00001) (5)\nwhere rtrepresents the recurrent object reasoning state at time tandgtis the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2208-6db4-11ee-ac65-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-396", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "rt=f\u001e(gt;rt\u00001) (5)\nwhere rtrepresents the recurrent object reasoning state at time tandgtis the\nglobal inter-frame interaction inferred at time tsuch as described in Equation 4.\nIn practice, this is implemented as a GRU, but for simplicity we omitted the\ngates in Equation (5). The pairwise mappings h\u0012(\u0001;\u0001) are implemented as an\nMLP. Figure 3 provides a visual explanation of the object head's operating\nthrough time.\nOur proposed ORN di\u000bers from [32] in three main points:\nObjects have a semantic de\fnition | we model relationships with respect\nto semantically meaningful entities (object instances) instead of feature map\ncells which do not have a semantically meaningful spatial extent. We will show\nin the experimental section that this is a key di\u000berence.\nObject Level Visual Reasoning in Videos 9\nObjects are selected from di\u000berent frames | we infer object pairwise\nrelations only between objects present in two di\u000berent sets. This is a key design\nchoice which allows our model to reason about changes in object relationships\nover time.\nLong range reasoning | integration of the object relations over time is recurrent by using a RNN for f\u001e(\u0001). Since reasoning from a full sequence cannot"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2209-6db4-11ee-9b18-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-397", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "over time.\nLong range reasoning | integration of the object relations over time is recurrent by using a RNN for f\u001e(\u0001). Since reasoning from a full sequence cannot\nbe done by inferring the relations between two frames, f\u001e(\u0001) allows long range\nreasoning on sequences of variable length.\n3.2 Object instance features\nThe object features Ot=\b\nok\nt\tK\nk=1for each frame tused for the ORN module\ndescribed above are computed and collected from local regions predicted by\na mask predictor. Independently for each frame Xtof the input data block,\nwe predict object instances as binary masks Bk\ntand associated object class\npredictions ck\nt, a distribution over Cclasses. We use Mask-RCNN [15], which\nis able to detect objects in a frame using region proposal networks [30] and\nproduces a high quality segmentation mask for each object instance.\nThe objective is to collect features for each object instance, which jointly\ndescribe its appearance, the change in its appearance over time, and its shape,\ni.e. the shape of the binary mask. In theory, appearance could also be described\nby pooling the feature representation learned by the mask predictor (Mask RCNN). However, in practice we choose to pool features from the dedicated object"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c220a-6db4-11ee-8646-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-398", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "by pooling the feature representation learned by the mask predictor (Mask RCNN). However, in practice we choose to pool features from the dedicated object\nhead such as shown in Figure 2, which also include motion through the spatiotemporal convolutions shared with the activity head:\nuk\nt= ROI-Pooling( Ut;Bk\nt) (6)\nwhere Utis the feature map output by the object head ,uk\ntis aD-dimensional\nvector of appearance and appearance change of object k.\nShape information from the binary mask Bk\ntis extracted through the following mapping function: bk\nt=g\u001e(Bk\nt);where g\u001e(\u0001) is a MLP. Information about\nobject kin image Xtis given by a concatenation of appearance, shape, and\nobject class: ok\nt= [bk\ntuk\ntck\nt].\n3.3 Global Motion and Context\nCurrent approaches in video understanding focus on modeling the video from\na high-level perspective. By a stack of spatio-temporal convolution and pooling\nthey focus on learning global scene context information. E\u000bective activity recognition requires integration of both of these sources: global information about the\nentire video content in addition to relational reasoning for making \fne distinctions regarding object interactions and properties.\nIn our method, local low-level reasoning is provided through object head"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c220b-6db4-11ee-927d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-399", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "entire video content in addition to relational reasoning for making \fne distinctions regarding object interactions and properties.\nIn our method, local low-level reasoning is provided through object head\nand the ORN module such as described above in Section 3.1. We complement\n10 Baradel et al.\nthis representation by high-level context information described by Vtwhich are\nfeature outputs from the activity head (orange block in Figure 2).\nWe use spatial global average pooling over Vtto output T D-dimensional\nfeature vectors denoted by vt, where vtcorresponds to the context information\nof the video at timestep t.\nWe model the dynamics of the context information through time by employing a RNN f\r(\u0001) given by:\nst=f\r(vt;st\u00001) (7)\nwhere sis the hidden state of f\r(\u0001) and gives cues about the evolution of the\ncontext though time.\n3.4 Recognition\nGiven an input video sequence X1:T, the two di\u000berent streams corresponding to\nthe activity head and the object head result in the two representations handr,\nrespectively where h=P\nthtandr=P\ntrt. Each representation is the hidden\nstate of the respective GRU, which were described in the preceding subsections.\nRecall that hprovides the global motion context while rprovides the object"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c220c-6db4-11ee-bf18-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-400", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "state of the respective GRU, which were described in the preceding subsections.\nRecall that hprovides the global motion context while rprovides the object\nreasoning state output by the ORN module. We perform independent linear\nclassi\fcation for each representation:\ny1=W h (8)\ny2=Z r (9)\nwhere y1;y2correspond to the logits from the activity head and the object head ,\nrespectively, and WandZare trainable weights (including biases). The \fnal\nprediction is done by averaging logits y1andy2followed by softmax activation.\n4 Network Architectures and feature dimensions\nThe input RGB images Xtare of size R3\u0002W\u0002Hwhere WandHcorrespond to\nthe width and height and are of size 224 each. The object and activity heads\n(orange and green in Figure 2) are a joint convolutional neural network with\nResnet50 architecture pre-trained on ImageNet/ILSVRC [31], with Conv1 and\nConv5 blocks being in\rated to 2.5D convolutions [44] (3D convolutions with a\nseparable temporal dimension). This choice has been optimized on the validation\nset, as explained in Section 6 and shown in Table 5.\nThe last conv5 layers have been split into two di\u000berent heads (activity head"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c220d-6db4-11ee-a6d9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-401", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "set, as explained in Section 6 and shown in Table 5.\nThe last conv5 layers have been split into two di\u000berent heads (activity head\nand object head). The intermediate feature representations UtandVtare of dimensions 2048\u0002T\u000214\u000214 and 2048\u0002T\u00027\u00027, respectively. We provide a higher\nspatial resolution for the feature maps Utof the object head to get more precise\nlocal descriptors. This can be done by changing the stride of the initial conv5\nlayers from 2 to 1. Temporal convolutions have been con\fgured to keep the same\ntime temporal dimension through the network.\nObject Level Visual Reasoning in Videos 11\nGlobal spatial pooling of activity features results in a 2048 dimensional feature vector fed into a GRU with 512 dimensional hidden state st. ROI-Pooling\nof object features results in 2048 dimensional feature vectors uk\nt. The encoder of\nthe binary mask is a MLP with one hidden layer of size 100 and outputs a mask\nembedding bk\ntof dimension 100. The number of object classes is 80, which leads\nin total to a 2229 dimensional object feature vector ok\nt.\nThe non-linearity h\u0012(\u0001) is implemented as an MLP with 2 hidden layers each"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c220e-6db4-11ee-9d2f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-402", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "in total to a 2229 dimensional object feature vector ok\nt.\nThe non-linearity h\u0012(\u0001) is implemented as an MLP with 2 hidden layers each\nwith 512 units and produces an 512 dimensional output space. f\u001e(\u0001) is implemented as a GRU with a 256 dimension hidden state rt. We use ReLU as the\nactivation function after each layer for each network.\n5 Training\nWe train the model with a loss split into two terms:\nLtot=L\u0010^y1+^y2\n2;y\u0011\n+X\ntX\nkL(^ck\nt;ck\nt): (10)\nwhereLis the cross-entropy loss. The \frst term corresponds to supervised activity class losses comparing two di\u000berent activity class predictions to the class\nground truth: ^y1is the prediction of the activity head, whereas ^y2is the prediction of the object head, as given by Equations (8) and (9), respectively.\nThe second term is a loss which pushes the features Uof the object towards\nrepresentations of the semantic object classes. The goal is to obtain features\nrelated to, both, motion (through the layers shared with the activity head), as\nwell as as object classes. As ground-truth object classes are not available, we\nde\fne the loss as the cross-entropy between the class label ck"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c220f-6db4-11ee-9af1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-403", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "well as as object classes. As ground-truth object classes are not available, we\nde\fne the loss as the cross-entropy between the class label ck\ntpredicted by the\nmask predictor and a dedicated linear class prediction ^ck\ntbased on features uk\nt,\nwhich, as we recall, are RoI-pooled from U:\nck\nt=R uk\nt (11)\nwhere Rtrainable parameters (biases integrated) learned end-to-end together\nwith the other parameters of the model.\nWe found that \frst training the object head only and then the full network\nwas performing better. A ResNet50 network pretrained on ImageNet is modi\fed\nby in\rating some of its \flters to 2.5 convolutions (3D convolutions with the time\ndimension separated), as described in Section 4; then by \fne-tuning.\nWe train the model using the Adam optimizer [22] with an initial learning\nrate of 10\u00004on 30 epochs and use early-stopping criterion on the validation set\nfor hyper-parameter optimization. Training takes \u001850 minutes per epoch on 4\nTitan XP GPUs with clips of 8 frames.\n12 Baradel et al.\n6 Experimental results\nWe evaluated the method on three standard datasets, which represent di\u000ecult"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2210-6db4-11ee-bbce-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-404", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "Titan XP GPUs with clips of 8 frames.\n12 Baradel et al.\n6 Experimental results\nWe evaluated the method on three standard datasets, which represent di\u000ecult\n\fne-grained activity recognition tasks: the Something-Something dataset, the\nVLOG dataset and the recently released EPIC Kitchens dataset.\nSomething-Something (SS) is a recent video classi\fcation dataset with\n108,000 example videos and 157 classes [13]. It shows humans performing di\u000berent actions with di\u000berent objects, actions and objects being combined in di\u000berent\nways. Solving SS requires common sense reasoning and the state-of-the-art methods in activity recognition tend to fail, which makes this dataset challenging.\nVLOG is a multi-label binary classi\fcation of human-object interactions\nrecently released with 114,000 videos and 30 classes [12]. Classes correspond\nto objects, and labels of a class are 1 if a person has touched a certain object\nduring the video, otherwise they are 0. It has recently been shown, that stateof-the-art video based methods [6] are outperformed on VLOG by image based\nmethods like ResNet-50 [16], although these video methods outperform image\nbased ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2211-6db4-11ee-8412-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-405", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "methods like ResNet-50 [16], although these video methods outperform image\nbased ResNet-50 on large-scale video datasets like the Kinetics dataset [6]. This\nsuggests a gap between traditional datasets like Kinetics and the \fne-grained\ndataset VLOG, making it particularly di\u000ecult.\nEPIC Kitchens (EPIC) is an egocentric video dataset recently released\ncontaining 55 hours recording of daily activities [7]. This is the largest in \frstperson vision and the activities performed are non-scripted, which makes the\ndataset very challenging and close to real world data. The dataset is densely\nannotated and several tasks exist such as object detection, action recognition and\naction prediction. We focus on action recognition with 39'594 action segments\nin total and 125 actions classes (i.e verbs). Since the test set is not available yet\nwe conducted our experiments on the training set (28'561 videos). We use the\nvideos recorded by person 01 to person 25 for training (22'675 videos) and de\fne\nthe validation set as the remaining videos (5'886 videos).\nFor all datasets we rescale the input video resolution to 256 \u0002256. While\ntraining, we crop space-time blocks of 224 \u0002224 spatial resolution and Lframes,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2212-6db4-11ee-b6d2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-406", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "For all datasets we rescale the input video resolution to 256 \u0002256. While\ntraining, we crop space-time blocks of 224 \u0002224 spatial resolution and Lframes,\nwithL=8 for the SS dataset and L=4 for VLOG and EPIC. We do not perform\nany other data augmentation. While training we extract Lframes from the entire\nvideo by splitting the video into Lsub-sequences and randomly sampling one\nframe per sub-sequence. The output sequence of size Lis called a clip. A clip\naims to represent the full video with less frames. For testing we aggregate results\nof 10 clips. We use lintel [10] for decoding video on the \ry.\nThe ablation study is done by using the train set as training data and we\nreport the result on the validation set. We compare against other state-of-theart approaches on the test set. For the ablation studies, we slightly decreased\nthe computational complexity of the model: the base network (including activity\nand object heads) is a ResNet-18 instead of ResNet-50, a single clip of 4 frames\nis extracted from a video at test time.\nComparison with other approaches. Table 1 shows the performance of\nthe proposed approach on the VLOG dataset. We outperform the state of the art\nObject Level Visual Reasoning in Videos 13"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2213-6db4-11ee-a56d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-407", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "Comparison with other approaches. Table 1 shows the performance of\nthe proposed approach on the VLOG dataset. We outperform the state of the art\nObject Level Visual Reasoning in Videos 13\nTable 1. Results on Hand/Semantic Object Interaction Classi\fcation (Average precision in % on the test set) on VLOG dataset. R50 and I3D implemented by [12].\nmAP\nbag\nbed\nbedding\nbook/papers\nbottle/tube\nbowl\nbox\nbrush\ncabinet\ncell-phone\nclothing\ncup\ndoor\ndrawers\nfood\nfork\nknife\nlaptop\nmicrowave\noven\npen/pencil\npillow\nplate\nrefrigerator\nsink\nspoon\nstu\u000bed animal\ntable\ntoothbrush\ntowel\nR50 [16] 40.5 29.7 68.9 65.8 64.5 58.2 33.1 22.1 19.0 23.9 54.0 45.5 28.6 49.2 28.7 49.6 19.4 37.5 62.9 48.8 23.0 36.9 39.2 12.5 55.9 58.8 31.1 57.4 26.8 39.6 22.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2214-6db4-11ee-b890-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-408", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "I3D [6] 39.7 24.9 71.7 71.4 62.5 57.1 27.1 19.2 33.9 20.7 50.6 45.8 24.7 54.7 19.1 50.8 19.3 41.9 54.0 27.5 21.4 37.4 42.9 12.6 42.5 60.4 33.9 46.0 23.5 59.6 34.7\nOurs 44.7 30.2 72.3 70.7 64.9 59.8 38.2 24.6 26.3 22.4 64.5 47.2 35.4 57.9 25.2 48.5 24.5 40.2 72.0 54.1 26.5 39.9 48.6 15.2 53.5 60.7 36.8 52.8 27.9 64.0 37.6\non this challenging dataset by a margin of \u00194.2 points (44.7% accuracy against\n40.5% by [16]). As mentioned above, traditional video approaches tend to fail\non this challenging \fne-grained dataset, providing inferior results. Table 3 shows\nperformance on SS where we outperform the state of the art given by very recent\nmethods (+2.3 points). On EPIC we re-implement standard baselines and report"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2215-6db4-11ee-9ea4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-409", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "performance on SS where we outperform the state of the art given by very recent\nmethods (+2.3 points). On EPIC we re-implement standard baselines and report\nresults on the validation set (Table 4) since the test set is not available. Our full\nmethod reports an accuracy of 40.89 and outperforms baselines by a large margin\n(\u0019+6.4 and\u0019+7.9 points respectively for against CNN-2D and I3D based on a\nResNet-18).\nE\u000bect of object-level reasoning. Table 2 shows the importance of reasoning on the performance of the method. The baseline corresponds to the performance obtained by the activity head trained alone (in\rated ResNet, in the\nResNet-18 version for this table). No object level reasoning is present in this\nbaseline. The proposed approach (third line) including an object head and the\nORN module gains 0.8, 2.5 and 2.4 points compared to our baseline respectively\non SS, on EPIC and on VLOG. This indicates that the reasoning module is able\nto extract complementary features compared to the activity head.\nUsing semantically de\fned objects proved to be important and led to a gain of\n2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2216-6db4-11ee-a76d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-410", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "2 points on EPIC and 2.3 points on VLOG for the full model (6/12.7 points using\nthe object head only) compared to an extension of Santoro et al. [32] operating\non pixel level. This indicates importance of object level reasoning. The gain on\nSS is smaller (0.7 point with the full model and 7.8 points with the object head\nonly) and can be explained by the di\u000berence in spatial resolution of the videos.\nObject detections and predictions of the binary masks are done using the initial\nvideo resolution. The mean video resolution for VLOG is 660 \u00021183 and for EPIC\nis 640\u0002480 against 100\u0002157 for SS. Mask-RCNN has been trained on images of\nresolution 800\u0002800 and thus performs best on higher resolutions. The quality of\nthe object detector is important for leveraging object level understanding then\nfor the rest of the ablation study we focus on EPIC and VLOG datasets.\nThe function f\u001ein Equation (5) is an important design choice in our model.\nIn our proposed model, f\u001eis recurrent over time to ensure that the ORN module\ncaptures long range reasoning over time, as shown in Equation (5). Removing\nthe recurrence in this equation leads to an MLP instead of a (gated) RNN, as"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2217-6db4-11ee-9508-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-411", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "the recurrence in this equation leads to an MLP instead of a (gated) RNN, as\nevaluated in row 4 of Table 2. Performance decreases by 1.1 point on VLOG\nand 1.4 points on EPIC. The larger gap for EPIC compared to VLOG and\ncan arguably be explained by the fact that in SS actions cover the whole video,\n14 Baradel et al.\nTable 2. Ablation study with ResNet-18 backbone. Results in %: Top-1 accuracy for\nEPIC and SS datasets, and mAP for VLOG dataset.\nMethod Object typeEPIC VLOG SS\nobj. head 2 heads obj. head 2 heads obj. head 2 heads\nBaseline - - 38.33 - 35.03 - 31.31\nORN pixel 23.71 38.83 14.40 35.18 2.51 31.43\nORN COCO 29.94 40.89 27.14 37.49 10.26 32.12"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2218-6db4-11ee-b9dd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-412", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "ORN COCO 29.94 40.89 27.14 37.49 10.26 32.12\nORN-mlp COCO 28.15 39.41 25.40 36.35 - ORN COCO-visual 28.45 38.92 22.92 35.49 - ORN COCO-shape 21.92 37.16 7.18 35.39 - ORN COCO-class 21.96 37.75 13.40 35.94 - ORN COCO-intra 29.25 38.10 26.78 36.28 - ORN clique-1 COCO 28.25 40.18 26.48 36.71 - ORN clique-3 COCO 22.61 37.67 27.05 36.04 - Table 3. Experimental results on the\nSomething-Something dataset (classi\fcation accuracy in % on the test set).\nMethods Top1\nC3D + Avg [13] 21.50\nI3D [13] 27.63\nMultiScale TRN [5] 33.60\nOurs 35.97Table 4. Experimental results on the\nEPIC Kitchens dataset (accuracy in %\non the validation set { methods with\u0003\nhave been re-implemented).\nMethods Top1\nR18 [16]\u000332.05\nI3D-18 [6]\u000334.20"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2219-6db4-11ee-9fea-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-413", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "have been re-implemented).\nMethods Top1\nR18 [16]\u000332.05\nI3D-18 [6]\u000334.20\nOurs 40.89\nwhile solving VLOG requires detecting the right moment when the human-object\ninteraction occurs and thus long range reasoning plays a less important role.\nVisual features extracted from object regions are the most discriminative,\nhowever object shapes and labels also provide complementary information. Finally, the last part of Table 2 evaluates the e\u000bect of the cliques size for modeling the interactions between objects and show that pairwise cliques outperform\ncliques of size 1 and 3. We would like to recall, that even with unary cliques\nonly, interactions between objects are still modeled. However, the model needs\nto \fnd subspaces in the hidden representations associated to each interaction.\nCNN architecture and kernel in\rations. The convolutional architecture\nof the model was optimized over the validation set of the SS dataset, as shown\nin Table 5. The architecture itself (in terms of numbers of layers, \flters etc.)\nis determined by pre-training on image classi\fcation. We optimized the choice\nof \flter in\rations from 2D to 2.5D or 3D for several convolutional blocks. This\nhas been optimized for the single head model and using a ResNet-18 variant"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c221a-6db4-11ee-b2d4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-414", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "has been optimized for the single head model and using a ResNet-18 variant\nto speed up computation. Adding temporal convolutions increases performance\nup to 100% w.r.t. to pure 2D baselines. This indicates, without surprise, that\nmotion is a strong cue. In\rating kernels to 2.5D on the input side and on the\nObject Level Visual Reasoning in Videos 15\nTable 5. E\u000bect of the CNN architecture (choice of kernel in\rations) on a single head\nResNet-18 network. Accuracy in % on the validation set of Something-Something is\nshown. 2.5D kernels are separable kernels: 2D followed by a 1D temporal.\nConv1 Conv2 Conv3 Conv4 Conv5 AggregSS2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D 2D 3D 2.5D GAP RNN\nX- - X- - X- - X- - X- - X -15.73\nX- - X- - X- - X- - X- - -X 15.88\n-X - -X - -X - -X - -X -X -31.42\n- - X - - X - - X - - X - - XX -27.58"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c221b-6db4-11ee-82c1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-415", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "-X - -X - -X - -X - -X -X -31.42\n- - X - - X - - X - - X - - XX -27.58\nX- - X- - X- - X- - -X -X -31.28\nX- - X- - X- - -X - -X -X -32.06\nX- - X- - -X - -X - -X -X -32.25\nX- - X- - X- - X- - - - XX -31.31\nX- - X- - X- - - - X - - XX -32.79\nX- - X- - - - X - - X - - XX -33.77\n-X -X- - X- - X- - X- - X -28.71\n-X - -X -X- - X- - X- - X -31.42\n- - XX- - X- - X- - X- - X -20.05\n- - X - - XX- - X- - X- - X -22.52\noutput side provided best performances, suggesting that temporal integration\nis required at a very low level (motion estimation) as well as on a very high\nlevel, close to reasoning. Our study also corroborates recent research in activity"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c221c-6db4-11ee-b2a0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-416", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "is required at a very low level (motion estimation) as well as on a very high\nlevel, close to reasoning. Our study also corroborates recent research in activity\nrecognition, indicating that 2.5D kernels provide a good trade-o\u000b between highcapacity and learnable numbers of parameters. Finally temporal integration via\nRNN outperforms global average pooling over space and time. The choice of\na (gated) RNN for temporal integration of the activity head features proved\nimportant (see Table 5) compared to global average pooling (GAP) over space\nand time.\nVisualizing the learned object interactions. Figure 4 shows visualizations of the pairwise object relationships the model learned from data, in particular from the VLOG dataset. Each graph is computed for a given activity class,\nand strong arcs between two nodes in the graph indicate strong relationships\nbetween the object classes, i.e. the model detects a high correlation between\nthese relationships and the activity. The graphs were obtained by thresholding\nthe summed activations of each pairwise relationship ( j; k) in equation (4). Each\npair ( j; k) can be assigned a pair of object classes cj\ntandck\ntthrough the predictions of the object instance mask predictor. Integrating over all samples of the\ndataset for a given class leads to the visualizations in Figure 4. We can see that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c221d-6db4-11ee-a379-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-417", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "tthrough the predictions of the object instance mask predictor. Integrating over all samples of the\ndataset for a given class leads to the visualizations in Figure 4. We can see that\nthe object interactions are highly relevant to the detected activities: the persontouches-bed activity is correlated to interactions between relevant object classes\nperson and bed. Similarly, activities human-bowl interaction and human-cup interaction show interactions with the respective objects bowl and cup. Moreover,\nother recovered relationships are highly correlated to the scene (for example,\ndining-table and bowl for activity human-bowl interaction ).\n16 Baradel et al.\nFig. 4. Example of object pairwise interactions learned by our model on VLOG for\nfour di\u000berent classes. Objects co-occurrences are at the top and learned pairwise objects\ninteractions are at the bottom. Line thickness indicates learned importance of a given\nrelation. Interactions have been normalized by the object co-occurrences.\nteddy bearteddy bearteddy bearbedcell phonepersonpersonbedteddy bear\nbottlewine glasspersonpersonpersonpersonbottle\nFig. 5. Examples of failure cases { a) small sized objects (on the left). Our model\ndetects a cell phone and a person but fails to detect hand-cell-phone contact ; b) confusion between semantically similar objects (on the right). The model falsly predicts"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c221e-6db4-11ee-a7d4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-418", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "detects a cell phone and a person but fails to detect hand-cell-phone contact ; b) confusion between semantically similar objects (on the right). The model falsly predicts\nhand-cup contact instead of hand-glass-contact even though the wine glass is detected.\nFinally, Figure 5 shows some failure cases, which are either due to errors\ndone by the object mask prediction (Mask R-CNN) or by the ORN itself.\n7 Conclusion\nWe presented a method for activity recognition in videos which leverages object instance detections for visual reasoning on object interactions over time.\nThe choice of reasoning over semantically well-de\fned objects is key to our approach and outperforms state of the art methods which reason on grid-levels,\nsuch as cells of convolutional feature maps. Temporal dependencies and causal\nrelationships are dealt with by integrating relationships between di\u000berent time\ninstants. We evaluated the method on three di\u000ecult datasets, on which standard\napproaches do not perform well, and report state-of-the-art results.\nAcknowledgements. This work was funded by grant Deepvision (ANR-15CE23-0029, STPGP-479356-15), a joint French/Canadian call by ANR & NSERC.\nObject Level Visual Reasoning in Videos 17\nReferences"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c221f-6db4-11ee-9386-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-419", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "Object Level Visual Reasoning in Videos 17\nReferences\n1. Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G., Varadarajan, B.,\nVijayanarasimhan, S.: Youtube-8m: A large-scale video classi\fcation benchmark.\narXiv preprint arxiv:1609.08675 (2016)\n2. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., Baskurt, A.: Sequential deep\nlearning for human action recognition. In: HBU (2011)\n3. Baradel, F., Wolf, C., Mille, J., Taylor, G.: Glimpse clouds: Human activity recognition from unstructured feature points. In: CVPR (2018)\n4. Battaglia, P.W., Pascanu, R., Lai, M., Rezende, D.J., Kavukcuoglu, K.: Interaction\nnetworks for learning about objects, relations and physics. In: NIPS (2016)\n5. Bolei, Z., Zhang, A.A., Torralba, A.: Temporal relational reasoning in videos. In:\nECCV (2018)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2220-6db4-11ee-af53-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-420", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "5. Bolei, Z., Zhang, A.A., Torralba, A.: Temporal relational reasoning in videos. In:\nECCV (2018)\n6. Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the\nkinetics dataset. In: CVPR (2017)\n7. Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E.,\nMoltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Scaling egocentric\nvision: The epic-kitchens dataset. In: ECCV (2018)\n8. Deng, Z., Vahdat, A., Hu, H., Mori, G.: Structure inference machines: Recurrent\nneural networks for analyzing relations in group activity recognition. In: CVPR\n(2016)\n9. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S.,\nSaenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual\nrecognition and description. In: CVPR (2015)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2221-6db4-11ee-bffb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-421", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual\nrecognition and description. In: CVPR (2015)\n10. Duke, B.: Lintel: Python video decoding. https://github.com/dukebw/lintel\n(2018)\n11. Fleuret, F., Li, T., Dubout, C., Wampler, E.K., Yantis, S., Geman, D.: Comparing\nmachines and humans on a visual categorization test. Proceedings of the National\nAcademy of Sciences of the United States of America 108 43 , 17621{5 (2011)\n12. Fouhey, D.F., Kuo, W., Efros, A.A., Malik, J.: From lifestyle vlogs to everyday\ninteractions. In: CVPR (2018)\n13. Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim,\nH., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau,\nC., Bax, I., Memisevic, R.: The \"something something\" video database for learning\nand evaluating visual common sense. In: ICCV (2017)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2222-6db4-11ee-85b5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-422", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "C., Bax, I., Memisevic, R.: The \"something something\" video database for learning\nand evaluating visual common sense. In: ICCV (2017)\n14. Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan,\nS., Toderici, G., Ricco, S., Sukthankar, R., Schmid, C., Malik, J.: Ava: A\nvideo dataset of spatio-temporally localized atomic visual actions. arXiv preprint\narXiv:1705.08421 (2017)\n15. He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)\n16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.\nIn: CVPR (2016)\n17. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation\n9(8), 1735{1780 (1997)\n18. Hudson, D., Manning, C.: Compositional attention networks for machine reasoning.\nIn: ICLR (2018)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2223-6db4-11ee-9055-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-423", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "9(8), 1735{1780 (1997)\n18. Hudson, D., Manning, C.: Compositional attention networks for machine reasoning.\nIn: ICLR (2018)\n19. Jain, A., Zamir, A.R., Savarese, S., Saxena, A.: Structural-RNN: Deep Learning\non Spatio-Temporal Graphs. In: CVPR (2016)\n20. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.: Largescale video classi\fcation with convolutional neural networks. In: CVPR (2014)\n18 Baradel et al.\n21. Kim, J., Ricci, M., Serre, T.: Not-so-CLEVR: Visual relations strain feedforward\nneural networks. arXiv preprint arXiv:1802.03390 (2018)\n22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICML (2015)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2224-6db4-11ee-ab85-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-424", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "22. Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: ICML (2015)\n23. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalanditis, Y., Li, L.J., Shamma, D.A., Bernstein, M., Fei-Fei, L.: Visual genome:\nConnecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV) 123, 32{73 (2017)\n24. Luc, P., , Neverova, N., Couprie, C., Verbeek, J., LeCun, Y.: Predicting deeper\ninto the future of semantic segmentation. In: ICCV (2017)\n25. Luvizon, D., Picard, D., Tabia, H.: 2d/3d pose estimation and action recognition\nusing multitask deep learning. In: CVPR (2018)\n26. Monfort, M., Zhou, B., Bargal, S.A., Andonian, A., Yan, T., Ramakrishnan,\nK., Brown, L., Fan, Q., Gutfruend, D., Vondrick, C., Oliva, A.: Moments\nin time dataset: one million videos for event understanding. arXiv preprint"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2225-6db4-11ee-94c5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-425", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "in time dataset: one million videos for event understanding. arXiv preprint\narXiv:1801.03150 (2018)\n27. Perez, E., Vries, H.D., Strub, F., Dumoulin, V., Courville, A.: Learning visual reasoning without strong priors. In: ICML Machine Learning in Speech and Language\nProcessing Workshop (2017)\n28. Pickup, L.C., Pan, Z., Wei, D., Shih, Y., Zhang, C., Zisserman, A., Scholkopf, B.,\nFreeman, W.T.: Seeing the arrow of time. In: CVPR (2014)\n29. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for\n3d classi\fcation and segmentation. In: CVPR (2017)\n30. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object\ndetection with region proposal networks. In: NIPS (2015)\n31. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2226-6db4-11ee-bf83-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-426", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "31. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,\nKarpathy, A., Khosla, A., Bernstein, M., Berg, A., Fei-Fei, L.: Imagenet large scale\nvisual recognition challenge. IJCV 115(3), 211{252 (2015)\n32. Santoro, A., Raposo, D., Barrett, D.G., Malinowski, M., Pascanu, R., Battaglia,\nP., Lillicrap, T.: A simple neural network module for relational reasoning. In: NIPS\n(2017)\n33. Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+D: A Large Scale Dataset\nfor 3D Human Activity Analysis. In: CVPR (2016)\n34. Sharma, S., Kiros, R., Salakhutdinov, R.: Action recognition using visual attention.\nIn: ICLR Workshop (2016)\n35. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2227-6db4-11ee-a843-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-427", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "35. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: NIPS (2014)\n36. Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An End-to-End Spatio-Temporal\nAttention Model for Human Action Recognition from Skeleton Data. In: AAAI\n(2016)\n37. Stabinger, S., Rodr\u0013 \u0010guez-S\u0013 anchez, A., Piater, J.: 25 years of CNNs: Can we compare\nto human abstraction capabilities? In: ICANN (2016)\n38. van Steenkiste, S., Chang, M., Gre\u000b, K., Schmidhuber, J.: Relational neural expectation maximization: Unsupervised discovery of objects and their interactions.\nIn: ICLR (2018)\n39. Sun, L., Jia, K., Chen, K., Yeung, D., Shi, B.E., Savarese, S.: Lattice long shortterm memory for human action recognition. In: ICCV (2017)\n40. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks. In: ICCV (2015)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2228-6db4-11ee-b068-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-428", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "41. Velikovi, P., Cucurull, G., Casanova, A., Romero, A., Li, P., Bengio, Y.: Graph\nattention networks. In: ICLR (2018)\nObject Level Visual Reasoning in Videos 19\n42. Wang, H., Kl\u007f aser, A., Schmid, C., Liu, C.L.: Action Recognition by Dense Trajectories. In: CVPR (2011)\n43. Watters, N., Zoran, D., Weber, T., Battaglia, P., Pascanu, R., Tacchetti, A.: Visual\ninteraction networks: Learning a physics simulator from video. In: NIPS (2017)\n44. Xie, S., Sun, C., Huang, J., Tu, Z., Murphy, K.: Rethinking spatiotemporal feature\nlearning for video understanding. arXiv prepring arxiv:1712.04851 (2017)\n45. Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., Fei-Fei, L.: Every moment counts: Dense detailed labeling of actions in complex videos. arXiv preprint\narXiv:1507.05738 (2015)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2229-6db4-11ee-af63-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1806.06157-429", "title": "Object Level Visual Reasoning in Videos", "source": "http://arxiv.org/pdf/1806.06157", "start_index": 0, "chroma:document": "arXiv:1507.05738 (2015)\n46. Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola,\nA.J.: Deep sets. In: NIPS. pp. 3391{3401 (2017), http://papers.nips.cc/paper/\n6931-deep-sets.pdf"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c222a-6db4-11ee-a151-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-0", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Massively Multilingual Neural Machine Translation\nRoee Aharoni\u0003\nBar Ilan University\nRamat-Gan\nIsrael\nroee.aharoni@gmail.comMelvin Johnson andOrhan Firat\nGoogle AI\nMountain View\nCalifornia\nmelvinp,orhanf@google.com\nAbstract\nMultilingual neural machine translation\n(NMT) enables training a single model that\nsupports translation from multiple source languages into multiple target languages. In this\npaper, we push the limits of multilingual NMT\nin terms of the number of languages being\nused. We perform extensive experiments in\ntraining massively multilingual NMT models,\ntranslating up to 102 languages to and from\nEnglish within a single model. We explore\ndifferent setups for training such models and\nanalyze the trade-offs between translation\nquality and various modeling decisions. We\nreport results on the publicly available TED\ntalks multilingual corpus where we show\nthat massively multilingual many-to-many\nmodels are effective in low resource settings,\noutperforming the previous state-of-the-art\nwhile supporting up to 59 languages. Our\nexperiments on a large-scale dataset with\n102 languages to and from English and up to\none million examples per direction also show\npromising results, surpassing strong bilingual\nbaselines and encouraging future work on\nmassively multilingual NMT.\n1 Introduction"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c222b-6db4-11ee-8cde-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-1", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "one million examples per direction also show\npromising results, surpassing strong bilingual\nbaselines and encouraging future work on\nmassively multilingual NMT.\n1 Introduction\nNeural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Bahdanau et al., 2014;\nSutskever et al., 2014) is the current state-ofthe-art approach for machine translation in both\nacademia (Bojar et al., 2016, 2017, 2018) and industry (Wu et al., 2016; Hassan et al., 2018). Recent works (Dong et al., 2015; Firat et al., 2016a;\nHa et al., 2016; Johnson et al., 2017) extended the\napproach to support multilingual translation, i.e.\ntraining a single model that is capable of translating between multiple language pairs.\nMultilingual models are appealing for several\nreasons. First, they are more ef\ufb01cient in terms\n\u0003Work carried out during an internship at Google AI.of the number of required models and model parameters, enabling simpler deployment. Another\nbene\ufb01t is transfer learning; when low-resource\nlanguage pairs are trained together with highresource ones, the translation quality may improve\n(Zoph et al., 2016; Nguyen and Chiang, 2017). An\nextreme case of such transfer learning is zero-shot"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c222c-6db4-11ee-aeff-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-2", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "(Zoph et al., 2016; Nguyen and Chiang, 2017). An\nextreme case of such transfer learning is zero-shot\ntranslation (Johnson et al., 2017), where multilingual models are able to translate between language\npairs that were never seen during training.\nWhile very promising, it is still unclear how far\none can scale multilingual NMT in terms of the\nnumber of languages involved. Previous works\non multilingual NMT typically trained models\nwith up to 7 languages (Dong et al., 2015; Firat et al., 2016b; Ha et al., 2016; Johnson et al.,\n2017; Gu et al., 2018) and up to 20 trained directions (Cettolo et al., 2017) simultaneously. One\nrecent exception is Neubig and Hu (2018) who\ntrained many-to-one models from 58 languages\ninto English. While utilizing signi\ufb01cantly more\nlanguages than previous works, their experiments\nwere restricted to many-to-one models in a lowresource setting with up to 214k examples per\nlanguage-pair and were evaluated only on four\ntranslation directions.\nIn this work, we take a step towards practical\n\u201cuniversal\u201d NMT \u2013 training massively multilingual models which support up to 102 languages\nand with up to one million examples per languagepair simultaneously. Speci\ufb01cally, we focus on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c222d-6db4-11ee-9f75-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-3", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "and with up to one million examples per languagepair simultaneously. Speci\ufb01cally, we focus on\ntraining \u201cEnglish-centric\u201d many-to-many models,\nin which the training data is composed of many\nlanguage pairs that contain English either on the\nsource side or the target side. This is a realistic\nsetting since English parallel data is widely available for many language pairs. We restrict our experiments to Transformer models (Vaswani et al.,\n2017) as they were shown to be very effective\nin recent benchmarks (Ott et al., 2018), also inarXiv:1903.00089v3  [cs.CL]  2 Jul 2019\nthe context of multilingual models (Lakew et al.,\n2018; Sachan and Neubig, 2018).\nWe evaluate the performance of such massively\nmultilingual models while varying factors like\nmodel capacity, the number of trained directions\n(tasks) and low-resource vs. high-resource settings. Our experiments on the publicly available\nTED talks dataset (Qi et al., 2018) show that massively multilingual many-to-many models with up\nto 58 languages to-and-from English are very effective in low resource settings, allowing to use\nhigh-capacity models while avoiding over\ufb01tting"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c222e-6db4-11ee-b42e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-4", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "to 58 languages to-and-from English are very effective in low resource settings, allowing to use\nhigh-capacity models while avoiding over\ufb01tting\nand achieving superior results to the current stateof-the-art on this dataset (Neubig and Hu, 2018;\nWang et al., 2019) when translating into English.\nWe then turn to experiment with models trained\non 103 languages in a high-resource setting. For\nthis purpose we compile an English-centric inhouse dataset, including 102 languages aligned\nto-and-from English with up to one million examples per language pair. We then train a single model on the resulting 204 translation directions and \ufb01nd that such models outperform strong\nbilingual baselines by more than 2 BLEU averaged across 10 diverse language pairs, both toand-from English. Finally, we analyze the tradeoffs between the number of involved languages\nand translation accuracy in such settings, showing\nthat massively multilingual models generalize better to zero-shot scenarios. We hope these results\nwill encourage future research on massively multilingual NMT.\n2 Low-Resource Setting: 59 Languages\n2.1 Experimental Setup\nThe main question we wish to answer in this work\nis how well a single NMT model can scale to\nsupport a very large number of language pairs.\nThe answer is not trivial: on the one hand, training multiple language pairs together may result"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c222f-6db4-11ee-9af0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-5", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "is how well a single NMT model can scale to\nsupport a very large number of language pairs.\nThe answer is not trivial: on the one hand, training multiple language pairs together may result\nin transfer learning (Zoph et al., 2016; Nguyen\nand Chiang, 2017). This may improve performance as we increase the number of language\npairs, since more information can be shared between the different translation tasks, allowing the\nmodel to learn which information to share. On the\nother hand, adding many language pairs may result in a bottleneck; the model has a limited capacity while it needs to handle this large number\nof translation tasks, and sharing all parameters between the different languages can be sub-optimal(Wang et al., 2018) especially if they are not from\nthe same typological language family (Sachan and\nNeubig, 2018).\nWe begin tackling this question by experimenting with the TED Talks parallel corpus compiled\nby Qi et al. (2018)1, which is unique in that it includes parallel data from 59 languages. For comparison, this is signi\ufb01cantly \u201cmore multilingual\u201d\nthan the data available from all previous WMT\nnews translation shared task evaluations throughout the years \u2013 the latest being Bojar et al. (2016,\n2017, 2018), which included 14 languages so far.2\nWe focus on the setting where we train"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2230-6db4-11ee-b083-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-6", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "2017, 2018), which included 14 languages so far.2\nWe focus on the setting where we train\n\u201cEnglish-centric\u201d models, i.e. training on all\nlanguage pairs that contain English in either the\nsource or the target, resulting in 116 translation\ndirections. This dataset is also highly imbalanced, with language pairs including between 3.3k\nto 214k sentence pairs for training. Table 9 in\nthe supplementary material details the languages\nand training set sizes for this dataset. Since the\ndataset is already tokenized we did not apply additional preprocessing other than applying joint\nsubword segmentation (Sennrich et al., 2016) with\n32k symbols.\nRegarding the languages we evaluate on, we begin with the same four languages as Neubig and\nHu (2018) \u2013 Azerbeijani (Az), Belarusian (Be),\nGalician (Gl) and Slovak (Sk). These languages\npresent an extreme low-resource case, with as few\nas 4.5k training examples for Belarusian-English.\nIn order to better understand the effect of training\nset size in these settings, we evaluate on four additional languages that have more than 167k training examples each \u2013 Arabic (Ar), German (De),\nHebrew (He) and Italian (It).\n2.2 Model Details\nUsing the same data, we trained three massively"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2231-6db4-11ee-8d27-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-7", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Hebrew (He) and Italian (It).\n2.2 Model Details\nUsing the same data, we trained three massively\nmultilingual models: a many-to-many model\nwhich we train using all 116 translation directions\nwith 58 languages to-and-from English, a one-tomany model from English into 58 languages, and\na many-to-one model from 58 languages into English. We follow the method of Ha et al. (2016);\nJohnson et al. (2017) and add a target-language\n1github.com/neulab/\nword-embeddings-for-nmt\n2Chinese, Czech, English, Estonian, Finnish, French,\nGerman, Hindi, Hungarian, Latvian, Romanian, Russian,\nSpanish, Turkish. According to http://www.statmt.\norg/wmtXX\npre\ufb01x token to each source sentence to enable\nmany-to-many translation. These different setups\nenable us to examine the effect of the number of\ntranslation tasks on the translation quality as measured in BLEU (Papineni et al., 2002). We also\ncompare our massively multilingual models to\nbilingual baselines and to two recently published\nresults on this dataset (Neubig and Hu (2018);\nWang et al. (2019))."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2232-6db4-11ee-94df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-8", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "bilingual baselines and to two recently published\nresults on this dataset (Neubig and Hu (2018);\nWang et al. (2019)).\nRegarding the models, we focused on the Transformer in the \u201cBase\u201d con\ufb01guration. We refer the\nreader to Vaswani et al. (2017) for more details\non the model architecture. Speci\ufb01cally, we use 6\nlayers in both the encoder and the decoder, with\nmodel dimension set at 512, hidden dimension\nsize of 2048 and 8 attention heads. We also applied dropout at a rate of 0.2 in the following components: on the sum of the input embeddings and\nthe positional embeddings, on the output of each\nsub-layer before added to the previous layer input\n(residual connection), on the inner layer output after the ReLU activation in each feed-forward sublayer, and to the attention weight in each attention\nsub-layer. This results in a model with approximately 93M trainable parameters. For all models we used the inverse square root learning rate\nschedule from Vaswani et al. (2017) with learningrate set at 3 and 40k warmup steps. All models are\nimplemented in Tensor\ufb02ow-Lingvo (Shen et al.,\n2019)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2233-6db4-11ee-b00c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-9", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "implemented in Tensor\ufb02ow-Lingvo (Shen et al.,\n2019).\nIn all cases we report test results for the checkpoint that performed best on the development set\nin terms of BLEU. For the multilingual models we\ncreate a development set that includes examples\nwe uniformly sample from a concatenation of all\nthe individual language pair development sets, resulting in 13k development examples per model.\nAnother important detail regarding multilingual\ntraining is the batching scheme. In all of our multilingual models we use heterogeneous batching,\nwhere each batch contains examples which are\nuniformly sampled from a concatenation of all the\nlanguage pairs the model is trained on. Speci\ufb01cally, we use batches of 64 examples for sequences\nshorter than 69 tokens and batches of 16 examples for longer sequences. We did not use oversampling as the dataset is relatively small.\n2.3 Results\nWe use tokenized BLEU in order to be comparable with Neubig and Hu (2018). Table 1 showsAz-En Be-En Gl-En Sk-En Avg.\n# of examples 5.9k 4.5k 10k 61k 20.3k\nNeubig & Hu 18\nbaselines 2.7 2.8 16.2 24 11.42"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2234-6db4-11ee-9a0c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-10", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Neubig & Hu 18\nbaselines 2.7 2.8 16.2 24 11.42\nmany-to-one 11.7 18.3 29.1 28.3 21.85\nWang et al. 18 11.82 18.71 30.3 28.77 22.4\nOurs\nmany-to-one 11.24 18.28 28.63 26.78 21.23\nmany-to-many 12.78 21.73 30.65 29.54 23.67\nTable 1: X !En test BLEU on the TED Talks corpus,\nfor the language pairs from Neubig and Hu (2018)\nAr-En De-En He-En It-En Avg.\n# of examples 213k 167k 211k 203k 198.5k\nbaselines 27.84 30.5 34.37 33.64 31.59\nmany-to-one 25.93 28.87 30.19 32.42 29.35\nmany-to-many 28.32 32.97 33.18 35.14 32.4\nTable 2: X !En test BLEU on the TED Talks corpus,\nfor language pairs with more than 167k examples\nupdates ar_many2one ro_many2one nl_many2one de_many2one it_many2one ar de it nl\n0 0"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2235-6db4-11ee-9b22-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-11", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "updates ar_many2one ro_many2one nl_many2one de_many2one it_many2one ar de it nl\n0 0\n49148 0.2839321792\n51751 0.2832087576\n53492 0.2823074162\n56111 0.2822623849\n57843 0.2835516334\n59595 0.2890901268\n61344 0.2845613658\n63938 0.2924402654\n65676 0.2828268111\n67401 0.2909407914\n70019 0.2906394601\n71770 0.283375144\n73479 0.2862167656\n75201 0.2850508094\n77780 0.2772497535\n79488 0.286601305\n81201 0.2964281738\n82895 0.2811564505\n84594 0.2892799973\n86301 0.2811136246\n88849 0.2873187661\n90551 0.287561506\n93101 0.2806210518\n95667 0.2852652967\n97386 0.286067158\n99069 0.285282582\n101647 0.2873030901"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2236-6db4-11ee-92aa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-12", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "95667 0.2852652967\n97386 0.286067158\n99069 0.285282582\n101647 0.2873030901\n103358 0.2812287509\n105932 0.2841886282\n107656 0.2752591372\n109350 0.2852451503\n111069 0.2802457809\n113631 0.2839408815\n115325 0.2698247731\n117038 0.2862476707\n119599 0.2838998437\n121302 0.2805997431\n123015 0.2826767564\nFigure 1: Development BLEU on\nfIt,Ro,Nl,De,Ar g!En vs. training BLEU for the\nmany-to-one and many-to-many models. Best viewed\nin color.\nthe results of our experiments when evaluating\non the same language pairs as they did. The results under \u201cNeubig & Hu 18\u201d are their bilingual baselines and their best many-to-one models.\nTheir many-to-one models use similar-languageregularization, i.e. \ufb01ne-tuning a pre-trained manyto-one model with data from the language pair of\ninterest together with data from a language pair"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2237-6db4-11ee-b99a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-13", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "interest together with data from a language pair\nthat has a typologically-similar source language\nand more training data (i.e. Russian and Belarusian, Turkish and Azerbaijani). The results under\n\u201cOurs\u201d are our many-to-one and many-to-many\nmodels we trained identically in terms of model\narchitecture and hyper-parameters.\nWe \ufb01rst note that our many-to-many model outperforms all other models when translating into\nEnglish, with 1.82 BLEU improvement (when averaged across the four language pairs) over the\nbest \ufb01ne-tuned many-to-one models of Neubig\nand Hu (2018) and 2.44 BLEU improvement over\nour many-to-one model when averaged across the\nfour low-resource language pairs (Table 1). This\nis surprising as it uses the same X !En data,\nmodel architecture and capacity as our many-toone model, while handling a heavier burden since\nit also supports 58 additional translation tasks\n(from English into58 languages). Our models also\noutperform the more complex models of Wang\net al. (2019) which use \u201dSoft Decoupled Encoding\u201d for the input tokens, while our models use a\nsimple subword segmentation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2238-6db4-11ee-b175-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-14", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "et al. (2019) which use \u201dSoft Decoupled Encoding\u201d for the input tokens, while our models use a\nsimple subword segmentation.\nOne possible explanation is that the many-toone model over\ufb01ts the English side of the corpus\nas it is multi-way-parallel: in such setting the English sentences are overlapping across the different language pairs, making it much easier for the\nmodel to memorize the training set instead of generalizing (when enough capacity is available). On\nthe other hand, the many-to-many model is trained\non additional target languages other than English,\nwhich can act as regularizers for the X !En tasks,\nreducing such over\ufb01tting.\nTo further illustrate this, Figure 1 tracks the\nBLEU scores on the individual development sets\nduring training for Italian (It), Romanian (Ro),\nDutch (Nl), German (De) and Arabic (Ar) into English (left), together with BLEU scores on a subset of the training set for each model. We can\nsee that while the many-to-one model degrades in\nperformance on the development set, the manyto-many model still improves. Note the large\ngap in the many-to-one model between the training set BLEU and the development set BLEU,\nwhich points on the generalization issue that is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2239-6db4-11ee-8065-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-15", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "gap in the many-to-one model between the training set BLEU and the development set BLEU,\nwhich points on the generalization issue that is\nnot present in the many-to-many setting. We also\nnote that our many-to-one model is on average\n0.75 BLEU behind the best many-to-one models\nin Neubig and Hu (2018). We attribute this to the\nfact that their models are \ufb01ne-tuned using similarlanguage-regularization while our model is not.\nWe \ufb01nd an additional difference between the\nresults on the resource-scarce languages (Table 1) and the higher-resource languages (Table\n2). Speci\ufb01cally, the bilingual baselines outperform the many-to-one models only in the higherresource setting. This makes sense as in the low-En-Az En-Be En-Gl En-Sk Avg.\n# of examples 5.9k 4.5k 10k 61k 20.3k\nbaselines 2.16 2.47 3.26 5.8 3.42\none-to-many 5.06 10.72 26.59 24.52 16.72\nmany-to-many 3.9 7.24 23.78 21.83 14.19\nEn-Ar En-De En-He En-It Avg."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c223a-6db4-11ee-b7a4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-16", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "many-to-many 3.9 7.24 23.78 21.83 14.19\nEn-Ar En-De En-He En-It Avg.\n# of examples 213k 167k 211k 203k 198.5k\nbaselines 12.95 23.31 23.66 30.33 22.56\none-to-many 16.67 30.54 27.62 35.89 27.68\nmany-to-many 14.25 27.95 24.16 33.26 24.9\nTable 3: En !X test BLEU on the TED Talks corpus\nresource setting the baselines have very few training examples to outperform the many-to-one models, while in the higher resource setting they have\naccess to more training data. This corroborates the\nresults of Gu et al. (2018) that showed the sensitivity of such models to similar low resource conditions and the improvements gained from using\nmany-to-one models (however with much fewer\nlanguage pairs).\nTable 3 shows the results of our massively\nmultilingual models and bilingual baselines when\nevaluated out-of-English. In this case we see an\nopposite trend: the many-to-many model performs\nworse than the one-to-many model by 2.53 BLEU\non average. While previous works (Wang et al.,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c223b-6db4-11ee-8ee6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-17", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "worse than the one-to-many model by 2.53 BLEU\non average. While previous works (Wang et al.,\n2018; Sachan and Neubig, 2018) discuss the phenomena of quality degradation in English-to-many\nsettings, this shows that increasing the number of\nsource languages also causes additional degradation in a many-to-many model. This degradation\nmay be due to the English-centric setting: since\nmost of the translation directions the model is\ntrained on are into English, this leaves less capacity for the other target languages (while still performing better than the bilingual baselines on all 8\nlanguage pairs). We also note that in this case the\nresults are consistent among the higher and lower\nresource pairs \u2013 the one-to-many model is better\nthan the many-to-many model, which outperforms\nthe bilingual baselines in all cases. This is unlike\nthe difference we saw in the X !En experiments\nsince here we do not have the multi-way-parallel\nover\ufb01tting issue.\n2.4 Discussion\nFrom the above experiments we learn that NMT\nmodels can scale to 59 languages in a lowresource, imbalanced, English-centric setting,\nwith the following observations: (1) massively\nmultilingual many-to-many models outperform"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c223c-6db4-11ee-8622-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-18", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "models can scale to 59 languages in a lowresource, imbalanced, English-centric setting,\nwith the following observations: (1) massively\nmultilingual many-to-many models outperform\nmany-to-one and bilingual models with similar capacity and identical training conditions when averaged over 8 language pairs into English. We\nattribute this improvement over the many-to-one\nmodels to the multiple target language pairs which\nmay act as regularizers, especially in this lowresource multi-way-parallel setting that is prone to\nmemorization. (2) many-to-many models are inferior in performance when going out-of-English\nin comparison to a one-to-many model. We attribute this to English being over-represented in\nthe English-centric many-to-many setting, where\nit appears as a target language in 58 out of 116\ntrained directions, which may harm the performance on the rest of the target languages as the\nmodel capacity is limited.3\nIt is important to stress the fact that we compared the different models under identical training\nconditions and did not perform extensive hyperparameter tuning for each setting separately. However, we believe that such tuning may improve\nperformance even further, as the diversity in each\ntraining batch is very different between the different settings. For example, while the baseline\nmodel batches include only one language in the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c223d-6db4-11ee-a0bf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-19", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "performance even further, as the diversity in each\ntraining batch is very different between the different settings. For example, while the baseline\nmodel batches include only one language in the\nsource and one language in the target, the manyto-many model includes 59 languages in each\nside with a strong bias towards English. These\ndifferences may require tailored hyper-parameter\nchoices for each settings (i.e. different batch sizes,\nlearning rate schedules, dropout rates etc.) which\nwould be interesting to explore in future work.\nIn the following experiments we investigate\nwhether these observations hold using (1) an even\nlarger set of languages, and (2) a much larger,\nbalanced training corpus that is not multi-wayparallel.\n3 High-Resource Setting: 103 Languages\n3.1 Experimental Setup\nIn this setting we scale the number of languages\nand examples per language pair further when\ntraining a single massively multilingual model.\nSince we are not aware of a publicly available resource for this purpose, we construct an in-house\ndataset. This dataset includes 102 language pairs\nwhich we \u201cmirror\u201d to-and-from English, with up\nto one million examples per language pair. This\nresults in 103 languages in total, and 204 translation directions which we train simultaneously."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c223e-6db4-11ee-b44f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-20", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "to one million examples per language pair. This\nresults in 103 languages in total, and 204 translation directions which we train simultaneously.\n3This issue may be alleviated by over-sampling the nonEnglish-target pairs, but we leave this for future work.More details about this dataset are available in Table 4, and Table 10 in the supplementary material\ndetails all the languages in the dataset.4\nSimilarly to our previous experiments, we compare the massively multilingual models to bilingual baselines trained on the same data. We tokenize the data using an in-house tokenizer and then\napply joint subword segmentation to achieve an\nopen-vocabulary. In this setting we used a vocabulary of 64k subwords rather than 32k. Since the\ndataset contains 24k unique characters, a 32k symbol vocabulary will consist of mostly characters,\nthereby increasing the average sequence length.\nRegarding the model, for these experiments we\nuse a larger Transformer model with 6 layers in\nboth the encoder and the decoder, model dimension set to 1024, hidden dimension size of 8192,\nand 16 attention heads. This results in a model\nwith approximately 473.7M parameters.5Since\nthe model and data are much larger in this case,\nwe used a dropout rate of 0.1 for our multilingual\nmodels and tuned it to 0.3 for our baseline models"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c223f-6db4-11ee-9288-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-21", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "the model and data are much larger in this case,\nwe used a dropout rate of 0.1 for our multilingual\nmodels and tuned it to 0.3 for our baseline models\nas it improved the translation quality on the development set.\nWe evaluate our models on 10 languages from\ndifferent typological families: Semitic \u2013 Arabic\n(Ar), Hebrew (He), Romance \u2013 Galician (Gl),\nItalian (It), Romanian (Ro), Germanic \u2013 German\n(De), Dutch (Nl), Slavic \u2013 Belarusian (Be), Slovak (Sk) and Turkic \u2013 Azerbaijani (Az) and Turkish (Tr). We evaluate both to-and-from English,\nwhere each language pair is trained on up to one\nmillion examples. As in the previous experiment,\nwe report test results from the model that performed best in terms of BLEU on the development\nset.\n4The average number of examples per language pair is\n940k, as for 13 out of the 102 pairs we had less than one\nmillion examples available.\n5This is larger than the Transformer \u201cBig\u201d con\ufb01guration,\nwhich includes approximately 213M trained parameters.\n# of language pairs 102\nexamples per pair\nmin 63,879\nmax 1,000,000\naverage 940,087\nstd. deviation 188,194"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2240-6db4-11ee-9791-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-22", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "# of language pairs 102\nexamples per pair\nmin 63,879\nmax 1,000,000\naverage 940,087\nstd. deviation 188,194\ntotal # of examples 95,888,938\nTable 4: Training set details for the 103 langauges corpus, X!En data.\nAr Az Be De He It Nl Ro Sk Tr Avg.\nbaselines 23.34 16.3 21.93 30.18 31.83 36.47 36.12 34.59 25.39 27.13 28.33\nmany-to-one 26.04 23.68 25.36 35.05 33.61 35.69 36.28 36.33 28.35 29.75 31.01\nmany-to-many 22.17 21.45 23.03 37.06 30.71 35.0 36.18 36.57 29.87 27.64 29.97\nTable 5: X !En test BLEU on the 103-language corpus\nAr Az Be De He It Nl Ro Sk Tr Avg.\nbaselines 10.57 8.07 15.3 23.24 19.47 31.42 28.68 27.92 11.08 15.54 19.13\none-to-many 12.08 9.92 15.6 31.39 20.01 33 31.06 28.43 17.67 17.68 21.68"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2241-6db4-11ee-bd59-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-23", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "one-to-many 12.08 9.92 15.6 31.39 20.01 33 31.06 28.43 17.67 17.68 21.68\nmany-to-many 10.57 9.84 14.3 28.48 17.91 30.39 29.67 26.23 18.15 15.58 20.11\nTable 6: En !X test BLEU on the 103-language corpus\n3.2 Results\nTable 5 describes the results when translating into\nEnglish. First, we can see that both multilingual\nmodels perform better than the baselines in terms\nof average BLEU. This shows that massively multilingual many-to-many models can work well in\nrealistic settings with millions of training examples, 102 languages and 204 jointly trained directions to-and-from English. Looking more closely,\nwe note several different behaviors in comparison\nto the low-resource experiments on the TED Talks\ncorpus. First, the many-to-one model here performs better than the many-to-many model. This\nshows that the previous result was indeed due to\nthe pathologies of the low-resource dataset; when\nthe training data is large enough and not multiway-parallel there is no over\ufb01tting in the many-toone model, and it outperforms the many-to-many\nmodel in most cases while they are trained identically."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2242-6db4-11ee-8481-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-24", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "model in most cases while they are trained identically.\nOne particular outlier in this case is German-toEnglish, where the many-to-one model is 2 BLEU\npoints below the many-to-many model. We examine the BLEU score of this language pair on its\ndedicated German-English development set during training in the many-to-one model and \ufb01nd\nthat it highly \ufb02uctuates. We then measure the\nperformance on the test set for this language pair\nby choosing the best checkpoint on the dedicated\nGerman-English development set (instead of on\nthe mixed multilingual development set) and \ufb01nd\nit to be 38.07, which is actually higher in 1 BLEU\nthan the best result of the many-to-many model.\nThis shows that while training many languages together, there is no \u201csilver bullet\u201d: some languages\nmay suffer from severe interference during training (i.e. a reduction of 3 BLEU in this case, from38.07 to 35.05) while other languages continue to\nimprove with more updates.\nTable 6 describes the results when translating\nout-of-English. Again, both of the massively multilingual models perform better than the baselines when averaged across the 10 evaluated language pairs, while handling up to 102 languages"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2243-6db4-11ee-8b1f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-25", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "out-of-English. Again, both of the massively multilingual models perform better than the baselines when averaged across the 10 evaluated language pairs, while handling up to 102 languages\nto-and-from English and 204 translation tasks simultaneously. In this case the results are similar to those we observed on the TED talks corpus, where the one-to-many model performs better\nthan the many-to-many model. Again, this advantage may be due to the one-to-many model handling a smaller number of tasks while not being\nbiased towards English in the target side like the\nmany-to-many model.\n4 Analysis\nThe above results show that massively multilingual NMT is indeed possible in large scale settings\nand can improve performance over strong bilingual baselines. However, it was shown in a somewhat extreme case with more than 100 languages\ntrained jointly, where we saw that in some cases\nthe joint training may harm the performance for\nsome language pairs (i.e. German-English above).\nIn the following analysis we would like to better understand the trade-off between the number\nof languages involved and the translation accuracy while keeping the model capacity and training con\ufb01guration \ufb01xed.\n4.1 Multilinguality & Supervised\nPerformance\nWe \ufb01rst study the effect of varying the number of languages on the translation accuracy in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2244-6db4-11ee-a791-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-26", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "4.1 Multilinguality & Supervised\nPerformance\nWe \ufb01rst study the effect of varying the number of languages on the translation accuracy in\na supervised setting, where we focus on manyAr-En En-Ar Fr-En En-Fr Ru-En En-Ru Uk-En En-Uk Avg.\n5-to-5 23.87 12.42 38.99 37.3 29.07 24.86 26.17 16.48 26.14\n25-to-25 23.43 11.77 38.87 36.79 29.36 23.24 25.81 17.17 25.8\n50-to-50 23.7 11.65 37.81 35.83 29.22 21.95 26.02 15.32 25.18\n75-to-75 22.23 10.69 37.97 34.35 28.55 20.7 25.89 14.59 24.37\n103-to-103 21.16 10.25 35.91 34.42 27.25 19.9 24.53 13.89 23.41\nTable 7: Supervised performance while varying the number of languages involved\nAr-Fr Fr-Ar Ru-Uk Uk-Ru Avg.\n5-to-5 1.66 4.49 3.7 3.02 3.21"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2245-6db4-11ee-9708-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-27", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Ar-Fr Fr-Ar Ru-Uk Uk-Ru Avg.\n5-to-5 1.66 4.49 3.7 3.02 3.21\n25-to-25 1.83 5.52 16.67 4.31 7.08\n50-to-50 4.34 4.72 15.14 20.23 11.1\n75-to-75 1.85 4.26 11.2 15.88 8.3\n103-to-103 2.87 3.05 12.3 18.49 9.17\nTable 8: Zero-Shot performance while varying the\nnumber of languages involved\nto-many models. We create four subsets of the\nin-house dataset by sub-sampling it to a different number of languages in each subset. In\nthis way we create four additional English-centric\ndatasets, containing 5, 25, 50 and 75 languages\neach to-and-from English. We make sure that\neach subset contains all the languages from the\nnext smaller subsets \u2013 i.e. the 25 language subset contains the 5 language subset, the 50 language subset contains the 25 language subset and\nso on. We train a similar-capacity large Transformer model (with 473.7M parameters) on each\nof these subsets and measure the performance for\neach model on the 8 supervised language pairs"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2246-6db4-11ee-9a49-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-28", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "of these subsets and measure the performance for\neach model on the 8 supervised language pairs\nfrom the smallest subset \u2013 fArabic, French, Russian, Ukrainian g$English. In this way we can\nanalyze to what extent adding more languages improves or harms translation quality while keeping\nthe model capacity \ufb01xed, testing the capacity vs.\naccuracy \u201csaturation point\u201d.\nTable 7 shows the results of this experiment,\nreporting the test results for the models that performed best on the multilingual development set.\nWe can see that in most cases the best results\nare obtained using the 5-to-5 model, showing that\nthere is indeed a trade off between the number\nof languages and translation accuracy when using a \ufb01xed model capacity and the same training setup. One may expect that the gaps between\nthe different models should become smaller and\neven close with more updates, as the models with\nmore languages see less examples per language\nin each batch, thus requiring more updates to improve in terms of BLEU. However, in our setting\nthese gaps did not close even after the models converged, leaving 2.73 average BLEU difference be-tween the 5-to-5 and the 103-to-103 model.\n4.2 Multilinguality & Zero-Shot\nPerformance\nWe then study the effect of the number of languages on zero-shot translation accuracy. Since"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2247-6db4-11ee-9ed7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-29", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "4.2 Multilinguality & Zero-Shot\nPerformance\nWe then study the effect of the number of languages on zero-shot translation accuracy. Since\nwe \ufb01nd zero-shot accuracy as an interesting measure for model generalization, we hypothesize that\nby adding more languages, the model is forced to\ncreate a more generalized representation to better utilize its capacity, which may improve zeroshot performance. We choose four language pairs\nfor this purpose: Arabic $French which are distant languages, and Ukrainian $Russian which are\nsimilar. Table 8 shows the results of our models\non these language pairs. For Arabic $French the\nBLEU scores are very low in all cases, with the\n50-to-50 and 25-to-25 models being slightly better than rest on Ar-Fr and Fr-Ar respectively. On\nRussian $Ukrainian we see clear improvements\nwhen increasing the number of languages to more\nthan \ufb01ve.\nFigure 2 further illustrates this, showing the better generalization performance of the massively\nmultilingual models under this zero-shot setting.\nWhile the zero-shot performance in this case is\nlow and unstable for the 5-to-5 and 25-to-25 modupdate 5-to-5 25-to-25 50-to-50 75-to-75 103-to-103\n0 0 0 0 0 0"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2248-6db4-11ee-85aa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-30", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "0 0 0 0 0 0\n25000 3.360397369 7.345648855 7.028211653 6.687645614 6.629930437\n50000 2.003555186 10.08476391 11.66040972 11.46485135 11.34905592\n100000 2.383616194 9.54657495 15.44517726 14.72926438 13.17522973\n150000 2.588021383 7.121089101 14.5408541 15.54533243 15.89359492\n200000 2.589718811 10.99432111 15.82096368 17.38970876 16.26121253\n250000 2.854427323 14.00393397 16.53215885 15.97282737 15.3215304\n300000 2.823847346 6.934611499 16.78672731 16.52613729 16.16107672\n350000 2.950227261 4.779103771 16.73545986 16.12752229 17.03165472\n400000 4.301280901 6.631205231 16.16190225 17.38892049 17.55904406"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2249-6db4-11ee-a85f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-31", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "400000 4.301280901 6.631205231 16.16190225 17.38892049 17.55904406\n450000 3.882381693 6.804813445 18.20554733 17.48778224 17.9339543\n500000 3.45445089 6.358428299 17.32598543 16.71108902 15.97367823\n550000 3.18 6.38 18.65 18.28 17.4\n600000 2.86 9.5 18.46 14.92 17.12\n650000 2.55 12.2 18.98 15.68 16.19\n700000 2.98 8.44 20.16 15.4 18.52\nFigure 2: Zero-shot BLEU during training for Ukranian to Russian\nels, it is much better for the 50-to-50, 75-to-75 and\n103-to-103 models. Given these results we can\nsay that the balance between capacity and generalization here favors the mid range 50-to-50 model,\neven when using models with more than 473M\ntrained parameters. This may hint at the necessity of even larger models for such settings, which\nis a challenging avenue for future work. We also\nnote that our 103 language corpus includes up to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c224a-6db4-11ee-898b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-32", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "trained parameters. This may hint at the necessity of even larger models for such settings, which\nis a challenging avenue for future work. We also\nnote that our 103 language corpus includes up to\none million examples per language pair \u2013 while in\nreal-world MT deployments, systems are trained\non much more examples per pair. This again emphasizes the need for better techniques for training\nsuch massively multilingual models as we may already be hitting the capacity barrier in our setting.\n5 Related Work\nDong et al. (2015) extended the NMT model of\nBahdanau et al. (2014) to one-to-many translation\n(from English into 4 languages) by adding a dedicated decoder per target language, showing improvements over strong single-pair baselines. Firat et al. (2016a,b) proposed many-to-many models (with up to 6 languages) by using separate encoders and decoders per language while sharing\nthe attention mechanism. They also introduced\nthe notion of zero-resource translation, where they\nuse synthetic training data generated through pivoting to train translation directions without available training data. Ha et al. (2016) and Johnson\net al. (2017) proposed to use a shared encoderdecoder-attention model for many-to-many translation (with up to 7 languages in the latter). In\norder to determine the target language in such"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c224b-6db4-11ee-915d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-33", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "order to determine the target language in such\nscenarios they proposed adding dedicated targetlanguage symbols to the source. This method enabled zero-shot translation, showing the ability of\nthe model to generalize to unseen pairs.\nRecent works propose different methods for parameter sharing between language pairs in multilingual NMT. Blackwood et al. (2018) propose\nsharing all parameters but the attention mechanism\nand show improvements over sharing all parameters. Sachan and Neubig (2018) explore sharing various components in self-attentional (Transformer) models. Lu et al. (2018) add a shared \u201cinterlingua\u201d layer while using separate encoders and\ndecoders. Zaremoodi et al. (2018) utilize recurrent\nunits with multiple blocks together with a trainable\nrouting network. Platanios et al. (2018) propose\nto share the entire network, while using a contex-tual parameter generator that learns to generate the\nparameters of the system given the desired source\nand target languages. Gu et al. (2018) propose\na \u201cUniversal Language Representation\u201d layer together with a Mixture-of-Language-Experts component to improve a many-to-one model from 5\nlanguages into English.\nWhile the mentioned studies provide valuable\ncontributions to improving multilingual models,\nthey apply their models on only up to 7 languages"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c224c-6db4-11ee-904a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-34", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "languages into English.\nWhile the mentioned studies provide valuable\ncontributions to improving multilingual models,\nthey apply their models on only up to 7 languages\n(Johnson et al., 2017) and 20 trained directions\n(Cettolo et al., 2017) in a single model, whereas\nwe focus on scaling NMT to much larger numbers of languages and trained directions. Regarding massively multilingual models, Neubig and\nHu (2018) explored methods for rapid adaptation\nof NMT to new languages by training multilingual models on the 59-language TED Talks corpus and \ufb01ne-tuning them using data from the new\nlanguages. While modeling signi\ufb01cantly more\nlanguages than previous studies, they only train\nmany-to-one models, which we show are inferior\nin comparison to our proposed massively multilingual many-to-many models when evaluated into\nEnglish on this dataset.\nTiedemann (2018) trained an English-centric\nmany-to-many model on translations of the bible\nincluding 927 languages. While this work pointed\nto an interesting phenomena in the latent space\nlearned by the model where it clusters representations of typologically-similar languages together, it did not include any evaluation of the\nproduced translations. Similarly, Malaviya et al.\n(2017) trained a many-to-English system including 1017 languages from bible translations, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c224d-6db4-11ee-846b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-35", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "produced translations. Similarly, Malaviya et al.\n(2017) trained a many-to-English system including 1017 languages from bible translations, and\nused it to infer typological features for the different languages (without evaluating the translation quality). In another relevant work, Artetxe\nand Schwenk (2018) trained an NMT model on\n93 languages and used the learned representations\nto perform cross-lingual transfer learning. Again,\nthey did not report the performance of the translation model learned in that massively multilingual\nsetting.\n6 Conclusions and Future Work\nWe showed that NMT models can successfully\nscale to 102 languages to-and-from English with\n204 trained directions and up to one million examples per direction. Such models improve the\ntranslation quality over similar single-pair baselines when evaluated to and from English by more\nthan 2 BLEU when averaged over 10 diverse language pairs in each case. We show a similar result on the low-resource TED Talks corpus with 59\nlanguages and 116 trained directions. We analyze\nthe trade-offs between translation quality and the\nnumber of languages involved, pointing on capacity bottlenecks even with very large models and\nshowing that massively multilingual models can\ngeneralize better to zero-shot settings.\nWe hope this work will encourage future research on massively multilingual NMT, enabling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c224e-6db4-11ee-842b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-36", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "showing that massively multilingual models can\ngeneralize better to zero-shot settings.\nWe hope this work will encourage future research on massively multilingual NMT, enabling\neasier support for systems that can serve more people around the globe. There are many possible avenues for future work, including semi-supervised\nlearning in such settings, exploring ways to reduce the performance degradation when increasing the number of languages, or using such models\nfor multilingual transfer learning (McCann et al.,\n2017; Eriguchi et al., 2018; Artetxe and Schwenk,\n2018). Understanding and improving zero-shot\nperformance in such scenarios is also a promising\ndirection for future work.\nAcknowledgments\nWe would like to thank the Google Brain and\nGoogle Translate teams for their useful inputs and\ndiscussions. We would also like to thank the entire\nLingvo development team for their foundational\ncontributions to this project.\nReferences\nMikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zeroshot cross-lingual transfer and beyond. arXiv\npreprint arXiv:1812.10464 .\nDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly\nlearning to align and translate. arXiv preprint"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c224f-6db4-11ee-915e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-37", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly\nlearning to align and translate. arXiv preprint\narXiv:1409.0473 .\nGraeme Blackwood, Miguel Ballesteros, and Todd\nWard. 2018. Multilingual neural machine translation with task-speci\ufb01c attention. In Proceedings of\nthe 27th International Conference on Computational\nLinguistics , Santa Fe, New Mexico, USA. Association for Computational Linguistics.\nOnd\u02c7rej Bojar, Rajen Chatterjee, Christian Federmann,\net al. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the\nSecond Conference on Machine Translation .Ond\u02c7rej Bojar, Rajen Chatterjee, Christian Federmann,\net al. 2016. Findings of the 2016 conference on machine translation. In ACL 2016 FIRST CONFERENCE ON MACHINE TRANSLATION (WMT16) ,\npages 131\u2013198.\nOnd\u02c7rej Bojar, Christian Federmann, Mark Fishel,\nYvette Graham, Barry Haddow, Philipp Koehn, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2250-6db4-11ee-ae99-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-38", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Ond\u02c7rej Bojar, Christian Federmann, Mark Fishel,\nYvette Graham, Barry Haddow, Philipp Koehn, and\nChristof Monz. 2018. Findings of the 2018 conference on machine translation (wmt18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers .\nMauro Cettolo, Federico Marcello, Bentivogli Luisa,\nNiehues Jan, St \u00a8uker Sebastian, Sudoh Katsuitho,\nYoshino Koichiro, and Federmann Christian. 2017.\nOverview of the iwslt 2017 evaluation campaign. In\nInternational Workshop on Spoken Language Translation .\nDaxiang Dong, Hua Wu, Wei He, Dianhai Yu, and\nHaifeng Wang. 2015. Multi-task learning for multiple language translation. In Proceedings of the\n53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint\nConference on Natural Language Processing .\nAkiko Eriguchi, Melvin Johnson, Orhan Firat, Hideto\nKazawa, and Wolfgang Macherey. 2018. Zeroshot cross-lingual classi\ufb01cation using multilingual neural machine translation. arXiv preprint\narXiv:1809.04686 ."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2251-6db4-11ee-9662-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-39", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "arXiv:1809.04686 .\nOrhan Firat, Kyunghyun Cho, and Yoshua Bengio.\n2016a. Multi-way, multilingual neural machine\ntranslation with a shared attention mechanism. In\nProceedings of the 2016 Conference of the North\nAmerican Chapter of the Association for Computational Linguistics: Human Language Technologies .\nOrhan Firat, Baskaran Sankaran, Yaser Al-Onaizan,\nFatos T Yarman Vural, and Kyunghyun Cho. 2016b.\nZero-resource translation with multi-lingual neural\nmachine translation. In Proceedings of the 2016\nConference on Empirical Methods in Natural Language Processing .\nJiatao Gu, Hany Hassan, Jacob Devlin, and Victor O.K.\nLi. 2018. Universal neural machine translation for\nextremely low resource languages. In Proceedings of the 2018 Conference of the North American\nChapter of the Association for Computational Linguistics: Human Language Technologies .\nThanh-Le Ha, Jan Niehues, and Alexander Waibel.\n2016. Toward multilingual neural machine translation with universal encoder and decoder. arXiv\npreprint arXiv:1611.04798 .\nHany Hassan, Anthony Aue, Chang Chen, et al."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2252-6db4-11ee-911a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-40", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "preprint arXiv:1611.04798 .\nHany Hassan, Anthony Aue, Chang Chen, et al.\n2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint\narXiv:1803.05567 .\nMelvin Johnson, Mike Schuster, Quoc V Le, et al.\n2017. Google\u2019s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association of Computational Linguistics.\nNal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings\nof the 2013 Conference on Empirical Methods in\nNatural Language Processing , Seattle, Washington,\nUSA.\nSurafel Melaku Lakew, Mauro Cettolo, and Marcello\nFederico. 2018. A comparison of transformer and\nrecurrent neural networks on multilingual neural\nmachine translation. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA.\nYichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, and Jason Sun. 2018. A neural interlingua for multilingual machine translation.\nInProceedings of the Third Conference on Machine\nTranslation: Research Papers , Belgium, Brussels."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2253-6db4-11ee-8a0f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-41", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "InProceedings of the Third Conference on Machine\nTranslation: Research Papers , Belgium, Brussels.\nChaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language\nProcessing .\nBryan McCann, James Bradbury, Caiming Xiong, and\nRichard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems .\nGraham Neubig and Junjie Hu. 2018. Rapid adaptation\nof neural machine translation to new languages. In\nProceedings of the 2018 Conference on Empirical\nMethods in Natural Language Processing .\nToan Q. Nguyen and David Chiang. 2017. Transfer\nlearning across low-resource, related languages for\nneural machine translation. In Proc. IJCNLP .\nMyle Ott, Sergey Edunov, David Grangier, and\nMichael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on\nMachine Translation: Research Papers .\nKishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of\n40th Annual Meeting of the Association for Computational Linguistics , Philadelphia, Pennsylvania,\nUSA."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2254-6db4-11ee-8beb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-42", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "40th Annual Meeting of the Association for Computational Linguistics , Philadelphia, Pennsylvania,\nUSA.\nEmmanouil Antonios Platanios, Mrinmaya Sachan,\nGraham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language\nProcessing .Ye Qi, Sachan Devendra, Felix Matthieu, Padmanabhan Sarguna, and Neubig Graham. 2018. When\nand why are pre-trained word embeddings useful for\nneural machine translation. In HLT-NAACL .\nDevendra Sachan and Graham Neubig. 2018. Parameter sharing methods for multilingual self-attentional\ntranslation models. In Proceedings of the Third\nConference on Machine Translation: Research Papers, Belgium, Brussels.\nRico Sennrich, Barry Haddow, and Alexandra Birch.\n2016. Neural machine translation of rare words with\nsubword units. In Proceedings of the 54th Annual\nMeeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pages\n1715\u20131725.\nJonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng\nChen, et al. 2019. Lingvo: a modular and scalable\nframework for sequence-to-sequence modeling."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2255-6db4-11ee-aa6b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-43", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng\nChen, et al. 2019. Lingvo: a modular and scalable\nframework for sequence-to-sequence modeling.\nIlya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.\nSequence to sequence learning with neural networks. In Advances in neural information processing systems , pages 3104\u20133112.\nJ\u00a8org Tiedemann. 2018. Emerging language spaces\nlearned from massively multilingual corpora. arXiv\npreprint arXiv:1802.00273 .\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob\nUszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz\nKaiser, and Illia Polosukhin. 2017. Attention is all\nyou need. In Advances in Neural Information Processing Systems , pages 5998\u20136008.\nXinyi Wang, Hieu Pham, Philip Arthur, and Graham\nNeubig. 2019. Multilingual neural machine translation with soft decoupled encoding. In International\nConference on Learning Representations .\nYining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2256-6db4-11ee-a8b7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-44", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Conference on Learning Representations .\nYining Wang, Jiajun Zhang, Feifei Zhai, Jingfang Xu,\nand Chengqing Zong. 2018. Three strategies to improve one-to-many multilingual translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , Brussels, Belgium.\nYonghui Wu, Mike Schuster, Zhifeng Chen, et al. 2016.\nGoogle\u2019s neural machine translation system: Bridging the gap between human and machine translation.\narXiv preprint arXiv:1609.08144 .\nPoorya Zaremoodi, Wray Buntine, and Gholamreza\nHaffari. 2018. Adaptive knowledge sharing in\nmulti-task learning: Improving low-resource neural\nmachine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational\nLinguistics , Melbourne, Australia.\nBarret Zoph, Deniz Yuret, Jonathan May, and Kevin\nKnight. 2016. Transfer learning for low-resource\nneural machine translation. In Proceedings of the\n2016 Conference on Empirical Methods in Natural\nLanguage Processing .\nA Supplementary Material\nLanguage Train set size\nArabic 214111\nHebrew 211819\nRussian 208458\nKorean 205640\nItalian 204503\nJapanese 204090\nChinese-Taiwan 202646"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2257-6db4-11ee-810f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-45", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Arabic 214111\nHebrew 211819\nRussian 208458\nKorean 205640\nItalian 204503\nJapanese 204090\nChinese-Taiwan 202646\nChinese-China 199855\nSpanish 196026\nFrench 192304\nPortuguese-Brazil 184755\nDutch 183767\nTurkish 182470\nRomanian 180484\nPolish 176169\nBulgarian 174444\nVietnamese 171995\nGerman 167888\nPersian 150965\nHungarian 147219\nSerbian 136898\nGreek 134327\nCroatian 122091\nUkrainian 108495\nCzech 103093\nThai 98064\nIndonesian 87406\nSlovak 61470\nSwedish 56647\nPortuguese 51785\nDanish 44940\nAlbanian 44525\nLithuanian 41919\nMacedonian 25335\nFinnish 24222\nBurmese 21497\nArmenian 21360\nFrench-Canadian 19870\nSlovenian 19831\nHindi 18798\nNorwegian 15825\nGeorgian 13193\nEstonian 10738\nKurdish 10371\nGalician 10017\nMarathi 9840\nMongolian 7607\nEsperanto 6535\nTamil 6224\nUrdu 5977\nAzerbaijani 5946"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2258-6db4-11ee-b734-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-46", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Marathi 9840\nMongolian 7607\nEsperanto 6535\nTamil 6224\nUrdu 5977\nAzerbaijani 5946\nBosnian 5664\nChinese 5534\nMalay 5220\nBasque 5182\nBengali 4649\nBelarusian 4509\nKazakh 3317\nTable 9: Language pairs in the TED talks dataset (58\nlanguages, paired with English) with the train-set size\nfor each pair.Languages\nAfrikaans Laothian\nAlbanian Latin\nAmharic Latvian\nArabic Lithuanian\nArmenian Luxembourgish*\nAzerbaijani Macedonian\nBasque Malagasy\nBelarusian Malay\nBengali Malayalam\nBosnian Maltese\nBulgarian Maori\nBurmese Marathi\nCatalan Mongolian\nCebuano Nepali\nChichewa* Norwegian\nChinese Pashto\nCorsican* Persian\nCroatian Polish\nCzech Portuguese\nDanish Punjabi\nDutch Romanian\nEsperanto Russian\nEstonian Samoan*\nFinnish Scots Gaelic*\nFrench Serbian\nFrisian Sesotho\nGalician Shona*\nGeorgian Sindhi*\nGerman Sinhalese\nGreek Slovak\nGujarati Slovenian\nHaitian Creole Somali"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2259-6db4-11ee-b794-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1903.00089-47", "title": "Massively Multilingual Neural Machine Translation", "source": "http://arxiv.org/pdf/1903.00089", "start_index": 0, "chroma:document": "Galician Shona*\nGeorgian Sindhi*\nGerman Sinhalese\nGreek Slovak\nGujarati Slovenian\nHaitian Creole Somali\nHausa* Spanish\nHawaiian* Sundanese\nHebrew Swahili\nHindi Swedish\nHmong* Tagalog\nHungarian Tajik*\nIcelandic Tamil\nIgbo Telugu\nIndonesian Thai\nIrish Turkish\nItalian Ukrainian\nJapanese Urdu\nJavanese Uzbek\nKannada Vietnamese\nKazakh Welsh\nKhmer Xhosa\nKorean Yiddish\nKurdish Yoruba*\nKyrgyz Zulu\nTable 10: Language pairs in the in-house dataset (102\nlanguages, paired with English). For languages marked\nwith * we had less than 1M examples, while for the rest\nwe used exactly 1M."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c225a-6db4-11ee-8a5a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-0", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "Attention Augmented Convolutional Networks\nIrwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens\nQuoc V . Le\nGoogle Brain\nfibello,barretzoph,avaswani,shlens,qvl g@google.com\nAbstract\nConvolutional networks have been the paradigm of\nchoice in many computer vision applications. The convolution operation however has a signi\ufb01cant weakness in that it\nonly operates on a local neighborhood, thus missing global\ninformation. Self-attention, on the other hand, has emerged\nas a recent advance to capture long range interactions, but\nhas mostly been applied to sequence modeling and generative modeling tasks. In this paper, we consider the use of\nself-attention for discriminative visual tasks as an alternative to convolutions. We introduce a novel two-dimensional\nrelative self-attention mechanism that proves competitive\nin replacing convolutions as a stand-alone computational\nprimitive for image classi\ufb01cation. We \ufb01nd in control experiments that the best results are obtained when combining\nboth convolutions and self-attention. We therefore propose\nto augment convolutional operators with this self-attention\nmechanism by concatenating convolutional feature maps\nwith a set of feature maps produced via self-attention. Extensive experiments show that Attention Augmentation leads"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c225b-6db4-11ee-9c03-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-1", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "mechanism by concatenating convolutional feature maps\nwith a set of feature maps produced via self-attention. Extensive experiments show that Attention Augmentation leads\nto consistent improvements in image classi\ufb01cation on ImageNet and object detection on COCO across many different models and scales, including ResNets and a stateof-the art mobile constrained network, while keeping the\nnumber of parameters similar. In particular, our method\nachieves a 1:3%top-1 accuracy improvement on ImageNet\nclassi\ufb01cation over a ResNet50 baseline and outperforms\nother attention mechanisms for images such as Squeezeand-Excitation [17]. It also achieves an improvement of\n1.4 mAP in COCO Object Detection on top of a RetinaNet\nbaseline.\n1. Introduction\nConvolutional Neural Networks have enjoyed tremendous success in many computer vision applications, especially in image classi\ufb01cation [24, 23]. The design of the\nconvolutional layer imposes 1) locality via a limited receptive \ufb01eld and 2) translation equivariance via weight sharing.\n20 30 40 50 60 70\nnumber of parameters (millions)72737475767778798081top-1 accuracy\n73.6%76.4%77.9%78.4%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c225c-6db4-11ee-908a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-2", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "number of parameters (millions)72737475767778798081top-1 accuracy\n73.6%76.4%77.9%78.4%\n74.3%77.5%78.4%78.9%\n74.7%77.7%78.7%79.1%\nResNet\nSE-ResNet\nAA-ResNet (ours)Figure 1. Attention Augmentation systematically improves image classi\ufb01cation across a large variety of networks of different\nscales. ImageNet classi\ufb01cation accuracy [9] versus the number of\nparameters for baseline models (ResNet) [14], models augmented\nwith channel-wise attention (SE-ResNet) [17] and our proposed\narchitecture (AA-ResNet).\nBoth these properties prove to be crucial inductive biases\nwhen designing models that operate over images. However,\nthe local nature of the convolutional kernel prevents it from\ncapturing global contexts in an image, often necessary for\nbetter recognition of objects in images [33].\nSelf-attention [43], on the other hand, has emerged as a\nrecent advance to capture long range interactions, but has\nmostly been applied to sequence modeling and generative\nmodeling tasks. The key idea behind self-attention is to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c225d-6db4-11ee-b94f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-3", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "recent advance to capture long range interactions, but has\nmostly been applied to sequence modeling and generative\nmodeling tasks. The key idea behind self-attention is to\nproduce a weighted average of values computed from hidden units. Unlike the pooling or the convolutional operator,\nthe weights used in the weighted average operation are produced dynamically via a similarity function between hidden units. As a result, the interaction between input signals\ndepends on the signals themselves rather than being predetermined by their relative location like in convolutions. In\nparticular, this allows self-attention to capture long rangearXiv:1904.09925v5  [cs.CV]  9 Sep 2020\nInput Attention maps Weighted average of the values\nStandard convolutionOutput\nW\nH\nNh=2\nValues\u00b2Head Head\u00b9\nFigure 2. Attention-augmented convolution : For each spatial location (h, w), Nhattention maps over the image are computed from\nqueries and keys. These attention maps are used to compute Nhweighted averages of the values V . The results are then concatenated,\nreshaped to match the original volume\u2019s spatial dimensions and mixed with a pointwise convolution. Multi-head attention is applied in\nparallel to a standard convolution operation and the outputs are concatenated.\ninteractions without increasing the number of parameters."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c225e-6db4-11ee-b7f5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-4", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "parallel to a standard convolution operation and the outputs are concatenated.\ninteractions without increasing the number of parameters.\nIn this paper, we consider the use of self-attention for\ndiscriminative visual tasks as an alternative to convolutions. We develop a novel two-dimensional relative selfattention mechanism [37] that maintains translation equivariance while being infused with relative position information, making it well suited for images. Our self-attention\nformulation proves competitive for replacing convolutions\nentirely, however we \ufb01nd in control experiments that the\nbest results are obtained when combining both. We therefore do not completely abandon the idea of convolutions,\nbut instead propose to augment convolutions with this selfattention mechanism. This is achieved by concatenating\nconvolutional feature maps, which enforce locality, to selfattentional feature maps capable of modeling longer range\ndependencies (see Figure 2).\nWe test our method on the CIFAR-100 and ImageNet\nclassi\ufb01cation [22, 9] and the COCO object detection [27]\ntasks, across a wide range of architectures at different computational budgets, including a state-of-the art resource\nconstrained architecture [42]. Attention Augmentation\nyields systematic improvements with minimal additional"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c225f-6db4-11ee-bf0e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-5", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "constrained architecture [42]. Attention Augmentation\nyields systematic improvements with minimal additional\ncomputational burden and notably outperforms the popular Squeeze-and-Excitation [17] channelwise attention approach in all experiments. In particular, Attention Augmentation achieves a 1.3% top-1 accuracy ImageNet on top of\na ResNet50 baseline and 1.4 mAP increase in COCO object detection on top of a RetinaNet baseline. Suprisingly,\nexperiments also reveal that fully self-attentional models,\na special case of Attention Augmentation, only perform\nslightly worse than their fully convolutional counterparts on\nImageNet, indicating that self-attention is a powerful standalone computational primitive for image classi\ufb01cation.2. Related Work\n2.1. Convolutional networks\nModern computer vision has been built on powerful image featurizers learned on image classi\ufb01cation tasks such\nas CIFAR-10 [22] and ImageNet [9]. These datasets have\nbeen used as benchmarks for delineating better image featurizations and network architectures across a broad range\nof tasks [21]. For example, improving the \u201cbackbone\u201d network typically leads to improvements in object detection\n[19] and image segmentation [6]. These observations have\ninspired the research and design of new architectures, which\nare typically derived from the composition of convolution"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2260-6db4-11ee-98d4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-6", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "[19] and image segmentation [6]. These observations have\ninspired the research and design of new architectures, which\nare typically derived from the composition of convolution\noperations across an array of spatial scales and skip connections [23, 41, 39, 40, 14, 47, 13]. Indeed, automated\nsearch strategies for designing architectures based on convolutional primitives result in state-of-the-art accuracy on\nlarge-scale image classi\ufb01cation tasks that translate across a\nrange of tasks [55, 21].\n2.2. Attention mechanisms in networks\nAttention has enjoyed widespread adoption as a computational module for modeling sequences because of its\nability to capture long distance interactions [2, 44, 4, 3].\nMost notably, Bahdanau et al. [2] \ufb01rst proposed to combine attention with a Recurrent Neural Network [15] for\nalignment in Machine Translation. Attention was further\nextended by Vaswani et al. [43], where the self-attentional\nTransformer architecture achieved state-of-the-art results in\nMachine Translation. Using self-attention in cooperation\nwith convolutions is a theme shared by recent work in Natural Language Processing [49] and Reinforcement Learning [52]. For example, the QANet [50] and Evolved Transformer [38] architectures alternate between self-attention"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2261-6db4-11ee-831b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-7", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "layers and convolution layers for Question Answering applications and Machine Translation respectively. Additionally, multiple attention mechanisms have been proposed for visual tasks to address the weaknesses of convolutions [17, 16, 7, 46, 45, 53]. For instance, Squeezeand-Excitation [17] and Gather-Excite [16] reweigh feature\nchannels using signals aggregated from entire feature maps,\nwhile BAM [31] and CBAM [46] re\ufb01ne convolutional features independently in the channel and spatial dimensions.\nIn non-local neural networks [45], improvements are shown\nin video classi\ufb01cation and object detection via the additive use of a few non-local residual blocks that employ\nself-attention in convolutional architectures. However, nonlocal blocks are only added to the architecture after ImageNet pretraining and are initialized in such a way that they\ndo not break pretraining.\nIn contrast, our attention augmented networks do not rely\non pretraining of their fully convolutional counterparts and\nemploy self-attention along the entire architecture. The use\nof multi-head attention allows the model to attend jointly"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2262-6db4-11ee-8fda-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-8", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "on pretraining of their fully convolutional counterparts and\nemploy self-attention along the entire architecture. The use\nof multi-head attention allows the model to attend jointly\nto both spatial and feature subspaces. Additionally, we enhance the representational power of self-attention over images by extending relative self-attention [37, 18] to two dimensional inputs allowing us to model translation equivariance in a principled way. Finally our method produces additional feature maps, rather than recalibrating convolutional\nfeatures via addition [45, 53] or gating [17, 16, 31, 46]. This\nproperty allows us to \ufb02exibly adjust the fraction of attentional channels and consider a spectrum of architectures,\nranging from fully convolutional to fully attentional models.\n3. Methods\nWe now formally describe our proposed Attention Augmentation method. We use the following naming conventions:H,WandFinrefer to the height, width and number\nof input \ufb01lters of an activation map. Nh,dvanddkrespectively refer the number of heads, the depth of values and the\ndepth of queries and keys in multihead-attention (MHA).\nWe further assume that Nhdividesdvanddkevenly and\ndenotedh\nvanddh\nkthe depth of values and queries/keys per\nattention head."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2263-6db4-11ee-b062-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-9", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "We further assume that Nhdividesdvanddkevenly and\ndenotedh\nvanddh\nkthe depth of values and queries/keys per\nattention head.\n3.1. Self-attention over images\nGiven an input tensor of shape (H;W;F in),1we \ufb02atten\nit to a matrix X2RHW\u0002Finand perform multihead attention as proposed in the Transformer architecture [43]. The\noutput of the self-attention mechanism for a single head h\n1We omit the batch dimension for simplicity.can be formulated as:\nOh=Softmax0\n@(XW q)(XW k)T\nq\ndh\nk1\nA(XW v)(1)\nwhereWq,Wk2RFin\u0002dh\nkandWv2RFin\u0002dh\nvare learned\nlinear transformations that map the input Xto queriesQ=\nXW q, keysK=XW kand valuesV=XW v. The outputs\nof all heads are then concatenated and projected again as\nfollows:\nMHA (X) =Concath\nO1;:::;O Nhi\nWO(2)\nwhereWO2Rdv\u0002dvis a learned linear transformation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2264-6db4-11ee-a055-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-10", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "O1;:::;O Nhi\nWO(2)\nwhereWO2Rdv\u0002dvis a learned linear transformation.\nMHA (X)is then reshaped into a tensor of shape (H;W;d v)\nto match the original spatial dimensions. We note that\nmulti-head attention incurs a complexity of O((HW)2dk)\nand a memory cost of O((HW)2Nh)as it requires to store\nattention maps for each head.\n3.1.1 Two-dimensional Positional Embeddings\nWithout explicit information about positions, self-attention\nispermutation equivariant :\nMHA (\u0019(X)) =\u0019(MHA (X))\nfor any permutation \u0019of the pixel locations, making it ineffective for modeling highly structured data such as images. Multiple positional encodings that augment activation\nmaps with explicit spatial information have been proposed\nto alleviate related issues. In particular, the Image Transformer [32] extends the sinusoidal waves \ufb01rst introduced in\nthe original Transformer [43] to 2 dimensional inputs and\nCoordConv [29] concatenates positional channels to an activation map."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2265-6db4-11ee-bf3f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-11", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "the original Transformer [43] to 2 dimensional inputs and\nCoordConv [29] concatenates positional channels to an activation map.\nHowever these encodings did not help in our experiments on image classi\ufb01cation and object detection (see Section 4.5). We hypothesize that this is because such positional encodings, while not permutation equivariant, do not\nsatisfy translation equivariance , which is a desirable property when dealing with images. As a solution, we propose\nto extend the use of relative position encodings [37] to two\ndimensions and present a memory ef\ufb01cient implementation\nbased on the Music Transformer [18].\nRelative positional embeddings: Introduced in [37] for\nthe purpose of language modeling, relative self-attention\naugments self-attention with relative position embeddings\nand enables translation equivariance while preventing permutation equivariance. We implement two-dimensional relative self-attention by independently adding relative height\ninformation and relative width information. The attention\nlogit for how much pixel i= (ix;iy)attends to pixel\nj= (jx;jy)is computed as:\nli;j=qT\niq\ndh\nk(kj+rW\njx\u0000ix+rH\njy\u0000iy) (3)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2266-6db4-11ee-80fc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-12", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "li;j=qT\niq\ndh\nk(kj+rW\njx\u0000ix+rH\njy\u0000iy) (3)\nwhereqiis the query vector for pixel i(the i-th row of Q),\nkjis the key vector for pixel j(the j-th row of K) andrW\njx\u0000ix\nandrH\njy\u0000iyare learned embeddings for relative width jx\u0000ix\nand relative height jy\u0000iy, respectively. The output of head\nhnow becomes:\nOh=Softmax0\n@QKT+Srel\nH+Srel\nWq\ndh\nk1\nAV (4)\nwhereSrel\nH;Srel\nW2RHW\u0002HWare matrices of relative position logits along height and width dimensions that satisfy\nSrel\nH[i;j] =qT\nirH\njy\u0000iyandSrel\nW[i;j] =qT\nirW\njx\u0000ix.\nThe relative attention algorithm in [37] explicitly\nstores all relative embeddings rijin a tensor of shape\n(HW;HW;dh\nk), thus incurring an additional memory cost\nofO((HW)2dh\nk). This compares to O((HW)2Nh)for the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2267-6db4-11ee-9d16-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-13", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "k), thus incurring an additional memory cost\nofO((HW)2dh\nk). This compares to O((HW)2Nh)for the\nposition-unaware version self-attention that does not use\nposition encodings. As we typically have Nh<dh\nk, such an\nimplementation can prove extremely prohibitive and restrict\nthe number of images that can \ufb01t in a minibatch. Instead, we\nextend the memory ef\ufb01cient relative masked attention algorithm presented in [18] to unmasked relative self-attention\nover 2 dimensional inputs. Our implementation has a memory cost ofO(HWdh\nk). We leave the Tensor\ufb02ow code of\nthe algorithm in the Appendix.\nThe relative positional embeeddings rHandrWare\nlearned and shared across heads but not layers. For each\nlayer, we add (2(H+W)\u00002)dh\nkparameters to model relative distances along height and width.\n3.2. Attention Augmented Convolution\nMultiple previously proposed attention mechanisms over\nimages [17, 16, 31, 46] suggest that the convolution operator is limited by its locality and lack of understanding\nof global contexts. These methods capture long-range dependencies by recalibrating convolutional feature maps. In"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2268-6db4-11ee-8239-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-14", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "of global contexts. These methods capture long-range dependencies by recalibrating convolutional feature maps. In\nparticular, Squeeze-and-Excitation (SE) [17] and GatherExcite (GE) [16] perform channelwise reweighing while\nBAM [31] and CBAM [46] reweigh both channels and\nspatial positions independently . In contrast to these approaches, we 1) use an attention mechanism that can attend\njointly to spatial and feature subspaces (each head corresponding to a feature subspace) and 2) introduce additional\nfeature maps rather than re\ufb01ning them. Figure 2 summarizes our proposed augmented convolution.Concatenating convolutional and attentional feature\nmaps: Formally, consider an original convolution operator with kernel size k,Fininput \ufb01lters and Foutoutput\n\ufb01lters. The corresponding attention augmented convolution\ncan be written as\nAAConv (X) =Concath\nConv (X);MHA (X)i\n:\nWe denote\u001d=dv\nFoutthe ratio of attentional channels to\nnumber of original output \ufb01lters and \u0014=dk\nFoutthe ratio of\nkey depth to number of original output \ufb01lters. Similarly to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c2269-6db4-11ee-bf29-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-15", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "number of original output \ufb01lters and \u0014=dk\nFoutthe ratio of\nkey depth to number of original output \ufb01lters. Similarly to\nthe convolution, the proposed attention augmented convolution 1) is equivariant to translation and 2) can readily operate on inputs of different spatial dimensions. We include\nTensor\ufb02ow code for the proposed attention augmented convolution in the Appendix A.3.\nEffect on number of parameters: Multihead attention\nintroduces a 1x1 convolution with Fininput \ufb01lters and\n(2dk+dv) =Fout(2\u0014+\u001d)output \ufb01lters to compute queries,\nkeys and values and an additional 1x1 convolution with\ndv=Fout\u001dinput and output \ufb01lters to mix the contribution of different heads. Considering the decrease in \ufb01lters\nin the convolutional part, this leads to the following change\nin parameters:\n\u0001params\u0018FinFout(2\u0014+ (1\u0000k2)\u001d+Fout\nFin\u001d2);(5)\nwhere we ignore the parameters introduced by relative position embeddings for simplicity as these are negligible. In\npractice, this causes a slight decrease in parameters when\nreplacing 3x3 convolutions and a slight increase in parameters when replacing 1x1 convolutions. Interestingly, we \ufb01nd"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1c226a-6db4-11ee-b519-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "1904.09925-16", "title": "Attention Augmented Convolutional Networks", "source": "http://arxiv.org/pdf/1904.09925", "start_index": 0, "chroma:document": "practice, this causes a slight decrease in parameters when\nreplacing 3x3 convolutions and a slight increase in parameters when replacing 1x1 convolutions. Interestingly, we \ufb01nd\nin experiments that attention augmented networks still signi\ufb01cantly outperform their fully convolutional counterparts\nwhile using less parameters.\nAttention Augmented Convolutional Architectures: In\nall our experiments, the augmented convolution is followed\nby a batch normalization [20] layer which can learn to scale\nthe contribution of the convolution feature maps and the attention feature maps. We apply our augmented convolution\nonce per residual block similarly to other visual attention\nmechanisms [17, 16, 31, 46] and along the entire architecture as memory permits (see Section 4 for more details).\nSince the memory cost O((Nh(HW)2)can be prohibitive for large spatial dimensions, we augment convolutions with attention starting from the last layer (with smallest spatial dimension) until we hit memory constraints. To\nreduce the memory footprint of augmented networks, we\ntypically resort to a smaller batch size and sometimes additionally downsample the inputs to self-attention in the layers with the largest spatial dimensions where it is applied.\nDownsampling is performed by applying 3x3 average pooling with stride 2 while the following upsampling (required\nfor the concatenation) is obtained via bilinear interpolation."}


