# embeddings_queue

created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6e7-6db4-11ee-94ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-183", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "educational, financial, cultural, and research center as well as one of\nnorthern Europe's major cities. Approximately 75% of foreign companies\nthat operate in Finland have settled in the Helsinki region. The nearby\nmunicipality of Vantaa is the location of Helsinki Airport, with frequent\nservice to various destinations in Europe and Asia.\nQ: what is the most populous municipality in Finland?\nA: Helsinki\nQ: how many people live there?\nA: 1.4 million in the metropolitan area\nQ: what percent of the foreign companies that operate in Finland are in\nHelsinki?\nA: 75%\nQ: what towns are a part of the metropolitan area?\nA:\nTarget Completion !Helsinki, Espoo, Vantaa, Kauniainen, and surrounding commuter towns\nFigure G.18: Formatted dataset example for CoQA\nContext!Please unscramble the letters into a word, and write that word:\nasinoc =\nTarget Completion !casino\nFigure G.19: Formatted dataset example for Cycled Letters\n55\nContext!Passage: Saint Jean de Br\u0013 ebeuf was a French Jesuit missionary who\ntravelled to New France in 1625. There he worked primarily with the Huron\nfor the rest of his life, except for a few years in France from 1629 to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6e8-6db4-11ee-a615-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-184", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "travelled to New France in 1625. There he worked primarily with the Huron\nfor the rest of his life, except for a few years in France from 1629 to\n1633. He learned their language and culture, writing extensively about\neach to aid other missionaries. In 1649, Br\u0013 ebeuf and another missionary\nwere captured when an Iroquois raid took over a Huron village . Together\nwith Huron captives, the missionaries were ritually tortured and killed\non March 16, 1649. Br\u0013 ebeuf was beatified in 1925 and among eight Jesuit\nmissionaries canonized as saints in the Roman Catholic Church in 1930.\nQuestion: How many years did Saint Jean de Br\u0013 ebeuf stay in New France\nbefore he went back to France for a few years?\nAnswer:\nTarget Completion !4\nFigure G.20: Formatted dataset example for DROP\nContext!Fill in blank:\nShe held the torch in front of her.\nShe caught her breath.\n\"Chris? There's a step.\"\n\"What?\"\n\"A step. Cut in the rock. About fifty feet ahead.\" She moved faster.\nThey both moved faster. \"In fact,\" she said, raising the torch higher,\n\"there's more than a . ->\nTarget Completion !step\nFigure G.21: Formatted dataset example for LAMBADA"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6e9-6db4-11ee-bb70-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-185", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "\"there's more than a . ->\nTarget Completion !step\nFigure G.21: Formatted dataset example for LAMBADA\nContext!Please unscramble the letters into a word, and write that word:\nskicts =\nTarget Completion !sticks\nFigure G.22: Formatted dataset example for Anagrams 1 (A1)\nContext!Please unscramble the letters into a word, and write that word:\nvolwskagen =\nTarget Completion !volkswagen\nFigure G.23: Formatted dataset example for Anagrams 2\nContext!Q: Who played tess on touched by an angel?\nA:\nTarget Completion !Delloreese Patricia Early (July 6, 1931 { November 19, 2017), known\nprofessionally as Della Reese\nFigure G.24: Formatted dataset example for Natural Questions\n56\nContext!TITLE: William Perry (American football) - Professional career\nPARAGRAPH: In 1985, he was selected in the first round of the 1985 NFL\nDraft by the Chicago Bears; he had been hand-picked by coach Mike Ditka.\nHowever, defensive coordinator Buddy Ryan, who had a highly acrimonious\nrelationship with Ditka, called Perry a \"wasted draft-pick\". Perry\nsoon became a pawn in the political power struggle between Ditka and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6ea-6db4-11ee-8021-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-186", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "relationship with Ditka, called Perry a \"wasted draft-pick\". Perry\nsoon became a pawn in the political power struggle between Ditka and\nRyan. Perry's \"Refrigerator\" nickname followed him into the NFL and he\nquickly became a favorite of the Chicago Bears fans. Teammates called\nhim \"Biscuit,\" as in \"one biscuit shy of 350 pounds.\" While Ryan refused\nto play Perry, Ditka decided to use Perry as a fullback when the team was\nnear the opponents' goal line or in fourth and short situations, either\nas a ball carrier or a lead blocker for star running back Walter Payton.\nDitka stated the inspiration for using Perry as a fullback came to him\nduring five-yard sprint exercises. During his rookie season, Perry\nrushed for two touchdowns and caught a pass for one. Perry even had\nthe opportunity to run the ball during Super Bowl XX, as a nod to his\npopularity and contributions to the team's success. The first time he\ngot the ball, he was tackled for a one-yard loss while attempting to throw\nhis first NFL pass on a halfback option play. The second time he got the\nball, he scored a touchdown (running over Patriots linebacker Larry McGrew\nin the process). About halfway through his rookie season, Ryan finally\nbegan to play Perry, who soon proved that he was a capable defensive"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6eb-6db4-11ee-997d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-187", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "in the process). About halfway through his rookie season, Ryan finally\nbegan to play Perry, who soon proved that he was a capable defensive\nlineman. His Super Bowl ring size is the largest of any professional\nfootball player in the history of the event. His ring size is 25, while\nthe ring size for the average adult male is between 10 and 12. Perry went\non to play for ten years in the NFL, retiring after the 1994 season. In\nhis ten years as a pro, he regularly struggled with his weight, which\nhampered his performance at times. He played in 138 games, recording\n29.5 sacks and five fumble recoveries, which he returned for a total of\n71 yards. In his offensive career he ran five yards for two touchdowns,\nand had one reception for another touchdown. Perry later attempted a\ncomeback, playing an unremarkable 1996 season with the London Monarchs of\nthe World League of American Football (later NFL Europa).\nQ: what team did he play for?\nA:\nTarget Completion !the Chicago Bears\nFigure G.25: Formatted dataset example for QuAC\nContext!Please unscramble the letters into a word, and write that word:\nr e!c.i p r o.c a/l =\nTarget Completion !reciprocal\nFigure G.26: Formatted dataset example for Symbol Insertion"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6ec-6db4-11ee-a37b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-188", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "r e!c.i p r o.c a/l =\nTarget Completion !reciprocal\nFigure G.26: Formatted dataset example for Symbol Insertion\nContext!Please unscramble the letters into a word, and write that word:\ntaefed =\nTarget Completion !defeat\nFigure G.27: Formatted dataset example for Reversed Words\n57\nContext!Title: The Blitz\nBackground: From the German point of view, March 1941 saw an improvement.\nThe Luftwaffe flew 4,000 sorties that month, including 12 major and\nthree heavy attacks. The electronic war intensified but the Luftwaffe\nflew major inland missions only on moonlit nights. Ports were easier to\nfind and made better targets. To confuse the British, radio silence was\nobserved until the bombs fell. X- and Y-Ger\u007f at beams were placed over\nfalse targets and switched only at the last minute. Rapid frequency\nchanges were introduced for X-Ger\u007f at, whose wider band of frequencies and\ngreater tactical flexibility ensured it remained effective at a time when\nBritish selective jamming was degrading the effectiveness of Y-Ger\u007f at.\nQ: How many sorties were flown in March 1941?\nA: 4,000\nQ: When did the Luftwaffe fly inland missions?\nA:\nTarget Completion !only on moonlit nights"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6ed-6db4-11ee-b940-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-189", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "A: 4,000\nQ: When did the Luftwaffe fly inland missions?\nA:\nTarget Completion !only on moonlit nights\nFigure G.28: Formatted dataset example for SQuADv2\nContext!Normal force -- In a simple case such as an object resting upon a table,\nthe normal force on the object is equal but in opposite direction to the\ngravitational force applied on the object (or the weight of the object),\nthat is, N = m g ( ndisplaystyle N=mg), where m is mass, and g is the\ngravitational field strength (about 9.81 m/s on Earth). The normal force\nhere represents the force applied by the table against the object that\nprevents it from sinking through the table and requires that the table is\nsturdy enough to deliver this normal force without breaking. However, it\nis easy to assume that the normal force and weight are action-reaction\nforce pairs (a common mistake). In this case, the normal force and\nweight need to be equal in magnitude to explain why there is no upward\nacceleration of the object. For example, a ball that bounces upwards\naccelerates upwards because the normal force acting on the ball is larger\nin magnitude than the weight of the ball.\nquestion: is the normal force equal to the force of gravity?\nanswer:\nTarget Completion !yes"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6ee-6db4-11ee-b3f6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-190", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "in magnitude than the weight of the ball.\nquestion: is the normal force equal to the force of gravity?\nanswer:\nTarget Completion !yes\nFigure G.29: Formatted dataset example for BoolQ\nContext!The trend toward lower rents may seem surprising given that some\ncommunities in New York are bemoaning the loss of favorite local\nbusinesses to high rents. But, despite the recent softening, for many\nof these retailers there's still been too big a jump from the rental rates\nof the late 1970s, when their leases were signed. Certainly, the recent\ndrop in prices doesn't mean Manhattan comes cheap.\nquestion: Manhattan comes cheap. true, false, or neither?\nanswer:\nTarget Completion !false\nFigure G.30: Formatted dataset example for CB\n58\nContext!The bet, which won him dinner for four, was regarding the existence and\nmass of the top quark, an elementary particle discovered in 1995.\nquestion: The Top Quark is the last of six flavors of quarks predicted by\nthe standard model theory of particle physics. True or False?\nanswer:\nTarget Completion !False\nFigure G.31: Formatted dataset example for RTE\nContext!An outfitter provided everything needed for the safari.\nBefore his first walking holiday, he went to a specialist outfitter to buy\nsome boots."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6ef-6db4-11ee-a5b7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-191", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Context!An outfitter provided everything needed for the safari.\nBefore his first walking holiday, he went to a specialist outfitter to buy\nsome boots.\nquestion: Is the word `outfitter' used in the same way in the two\nsentences above?\nanswer:\nTarget Completion !no\nFigure G.32: Formatted dataset example for WiC\nContext!Final Exam with Answer Key\nInstructions: Please carefully read the following passages. For each\npassage, you must identify which noun the pronoun marked in *bold* refers\nto.\n=====\nPassage: Mr. Moncrieff visited Chester's luxurious New York apartment,\nthinking that it belonged to his son Edward. The result was that Mr.\nMoncrieff has decided to cancel Edward's allowance on the ground that\nhe no longer requires *his* financial support.\nQuestion: In the passage above, what does the pronoun \"*his*\" refer to?\nAnswer:\nTarget Completion !mr. moncrieff\nFigure G.33: Formatted dataset example for WSC\nContext!Q: `Nude Descending A Staircase' is perhaps the most famous painting by\nwhich 20th century artist?\nA:\nTarget Completion !MARCEL DUCHAMP\nTarget Completion !r mutt\nTarget Completion !duchamp"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f0-6db4-11ee-973e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-192", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "which 20th century artist?\nA:\nTarget Completion !MARCEL DUCHAMP\nTarget Completion !r mutt\nTarget Completion !duchamp\nTarget Completion !marcel duchamp\nTarget Completion !R.Mutt\nTarget Completion !Marcel duChamp\nTarget Completion !Henri-Robert-Marcel Duchamp\nTarget Completion !Marcel du Champ\nTarget Completion !henri robert marcel duchamp\nTarget Completion !Duchampian\nTarget Completion !Duchamp\nTarget Completion !duchampian\nTarget Completion !marcel du champ\nTarget Completion !Marcel Duchamp\nTarget Completion !MARCEL DUCHAMP\nFigure G.34: Formatted dataset example for TriviaQA. TriviaQA allows for multiple valid completions.\n59\nContext!Q: What school did burne hogarth establish?\nA:\nTarget Completion !School of Visual Arts\nFigure G.35: Formatted dataset example for WebQA\nContext!Keinesfalls d\u007f urfen diese f\u007f ur den kommerziellen Gebrauch verwendet werden.\n=\nTarget Completion !In no case may they be used for commercial purposes."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f1-6db4-11ee-88d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-193", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "=\nTarget Completion !In no case may they be used for commercial purposes.\nFigure G.36: Formatted dataset example for De !En. This is the format for one- and few-shot learning, for this and\nother langauge tasks, the format for zero-shot learning is \u201cQ: What is the flanguagegtranslation offsentencegA:\nftranslationg.\u201d\nContext!In no case may they be used for commercial purposes. =\nTarget Completion !Keinesfalls d\u007f urfen diese f\u007f ur den kommerziellen Gebrauch verwendet werden.\nFigure G.37: Formatted dataset example for En !De\nContext!Analysis of instar distributions of larval I. verticalis collected from\na series of ponds also indicated that males were in more advanced instars\nthan females. =\nTarget Completion !L'analyse de la distribution de fr\u0013 equence des stades larvaires d'I.\nverticalis dans une s\u0013 erie d'\u0013 etangs a \u0013 egalement d\u0013 emontr\u0013 e que les larves\nm^ ales \u0013 etaient \u0012 a des stades plus avanc\u0013 es que les larves femelles.\nFigure G.38: Formatted dataset example for En !Fr"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f2-6db4-11ee-9c1a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-194", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Figure G.38: Formatted dataset example for En !Fr\nContext!L'analyse de la distribution de fr\u0013 equence des stades larvaires d'I.\nverticalis dans une s\u0013 erie d'\u0013 etangs a \u0013 egalement d\u0013 emontr\u0013 e que les larves\nm^ ales \u0013 etaient \u0012 a des stades plus avanc\u0013 es que les larves femelles. =\nTarget Completion !Analysis of instar distributions of larval I. verticalis collected from\na series of ponds also indicated that males were in more advanced instars\nthan females.\nFigure G.39: Formatted dataset example for Fr !En\nContext!The truth is that you want, at any price, and against the wishes of the\npeoples of Europe, to continue the negotiations for Turkey's accession\nto the European Union, despite Turkey's continuing refusal to recognise\nCyprus and despite the fact that the democratic reforms are at a\nstandstill. =\nTarget Completion !Adev\u0015 arul este c\u0015 a v\u0015 a dorit \u0018i, cu orice pret \u0018 \u0018 si ^ \u0010mpotriva dorint \u0018ei\neuropenilor, s\u0015 a continuat \u0018i negocierile de aderare a Turciei la Uniunea"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f3-6db4-11ee-b3c7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-195", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "europenilor, s\u0015 a continuat \u0018i negocierile de aderare a Turciei la Uniunea\nEuropean\u0015 a, ^ \u0010n ciuda refuzului continuu al Turciei de a recunoa\u0018 ste Ciprul\n\u0018 si ^ \u0010n ciuda faptului c\u0015 a reformele democratice au ajuns ^ \u0010ntr-un punct mort.\nFigure G.40: Formatted dataset example for En !Ro\n60\nContext!Adev\u0015 arul este c\u0015 a v\u0015 a dorit \u0018i, cu orice pret \u0018 \u0018 si ^ \u0010mpotriva dorint \u0018ei\neuropenilor, s\u0015 a continuat \u0018i negocierile de aderare a Turciei la Uniunea\nEuropean\u0015 a, ^ \u0010n ciuda refuzului continuu al Turciei de a recunoa\u0018 ste Ciprul\n\u0018 si ^ \u0010n ciuda faptului c\u0015 a reformele democratice au ajuns ^ \u0010ntr-un punct mort.\n=\nTarget Completion !The truth is that you want, at any price, and against the wishes of the\npeoples of Europe, to continue the negotiations for Turkey's accession"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f4-6db4-11ee-b49d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-196", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "=\nTarget Completion !The truth is that you want, at any price, and against the wishes of the\npeoples of Europe, to continue the negotiations for Turkey's accession\nto the European Union, despite Turkey's continuing refusal to recognise\nCyprus and despite the fact that the democratic reforms are at a\nstandstill.\nFigure G.41: Formatted dataset example for Ro !En\nContext!Q: What is (2 * 4) * 6?\nA:\nTarget Completion !48\nFigure G.42: Formatted dataset example for Arithmetic 1DC\nContext!Q: What is 17 minus 14?\nA:\nTarget Completion !3\nFigure G.43: Formatted dataset example for Arithmetic 2DContext!Q: What is 98 plus 45?\nA:\nTarget Completion !143\nFigure G.44: Formatted dataset example for Arithmetic 2D+\nContext!Q: What is 95 times 45?\nA:\nTarget Completion !4275\nFigure G.45: Formatted dataset example for Arithmetic 2Dx\nContext!Q: What is 509 minus 488?\nA:\nTarget Completion !21\nFigure G.46: Formatted dataset example for Arithmetic 3DContext!Q: What is 556 plus 497?\nA:\nTarget Completion !1053"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f5-6db4-11ee-8f93-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-197", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Figure G.46: Formatted dataset example for Arithmetic 3DContext!Q: What is 556 plus 497?\nA:\nTarget Completion !1053\nFigure G.47: Formatted dataset example for Arithmetic 3D+\nContext!Q: What is 6209 minus 3365?\nA:\nTarget Completion !2844\nFigure G.48: Formatted dataset example for Arithmetic 4D61\nContext!Q: What is 9923 plus 617?\nA:\nTarget Completion !10540\nFigure G.49: Formatted dataset example for Arithmetic 4D+\nContext!Q: What is 40649 minus 78746?\nA:\nTarget Completion !-38097\nFigure G.50: Formatted dataset example for Arithmetic 5D \u0000\nContext!Q: What is 65360 plus 16204?\nA:\nTarget Completion !81564\nFigure G.51: Formatted dataset example for Arithmetic 5D+\n62\nH Results on All Tasks for All Model Sizes\nZero-Shot One-Shot Few-Shot\nName Metric SplitFine-tune\nSOTA K Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B175B"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f6-6db4-11ee-b706-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-198", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "(test server)\nHellaSwag acc dev 85.6 20 33.7 43.6 51.0 54.7 62.8 67.4 70.9 78.9 33.0 42.9 50.5 53.5 61.9 66.5 70.0 78.1 33.5 43.1 51.3 54.9 62.9 67.3 71.3 79.3\nLAMBADA acc test 68.0 15 42.7 54.3 60.4 63.6 67.1 70.3 72.5 76.2 22.0 47.1 52.6 58.3 61.1 65.4 69.0 72.5 22.0 40.4 63.2 57.0 78.1 79.1 81.3 86.4\nLAMBADA ppl test 8.63 15 18.6 9.09 6.53 5.44 4.60 4.00 3.56 3.00 165.0 11.6 8.29 6.46 5.53 4.61 4.06 3.35 165.0 27.6 6.63 7.45 2.89 2.56 2.56 1.92"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f7-6db4-11ee-b5b1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-199", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "StoryCloze acc test 91.8 70 63.3 68.5 72.4 73.4 77.2 77.7 79.5 83.2 62.3 68.7 72.3 74.2 77.3 78.7 79.7 84.7 62.3 70.2 73.9 76.1 80.2 81.2 83.0 87.7\nNQs acc test 44.5 64 0.64 1.75 2.71 4.40 6.01 5.79 7.84 14.6 1.19 3.07 4.79 5.43 8.73 9.78 13.7 23.0 1.72 4.46 7.89 9.72 13.2 17.0 21.0 29.9\nTriviaQA acc dev 68.0 64 4.15 7.61 14.0 19.7 31.3 38.7 41.8 64.3 4.19 12.9 20.5 26.5 35.9 44.4 51.3 68.0 6.96 16.3 26.5 32.1 42.3 51.6 57.5 71.2 71.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f8-6db4-11ee-a342-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-200", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "WebQs acc test 45.5 64 1.77 3.20 4.33 4.63 7.92 7.73 8.22 14.4 2.56 6.20 8.51 9.15 14.5 15.1 19.0 25.3 5.46 12.6 15.9 19.6 24.8 27.7 33.5 41.5\nRo!En 16 BLEU-mb test 39.9 64 2.08 2.71 3.09 3.15 16.3 8.34 20.2 19.9 0.55 15.4 23.0 26.3 30.6 33.2 35.6 38.6 1.25 20.7 25.8 29.2 33.1 34.8 37.0 39.5\nRo!En 16 BLEU-sb test 64 2.39 3.08 3.49 3.56 16.8 8.75 20.8 20.9 0.65 15.9 23.6 26.8 31.3 34.2 36.7 40.0 1.40 21.3 26.6 30.1 34.3 36.2 38.4 41.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6f9-6db4-11ee-afbf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-201", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "En!Ro 16 BLEU-mb test 38.5 64 2.14 2.65 2.53 2.50 3.46 4.24 5.32 14.1 0.35 3.30 7.89 8.72 13.2 15.1 17.3 20.6 1.25 5.90 9.33 10.7 14.3 16.3 18.0 21.0\nEn!Ro 16 BLEU-sb test 64 2.61 3.11 3.07 3.09 4.26 5.31 6.43 18.0 0.55 3.90 9.15 10.3 15.7 18.2 20.8 24.9 1.64 7.40 10.9 12.9 17.2 19.6 21.8 25.8\nFr!En 14 BLEU-mb test 35.0 64 1.81 2.53 3.47 3.13 20.6 15.1 21.8 21.2 1.28 15.9 23.7 26.3 29.0 30.5 30.2 33.7 4.98 25.5 28.5 31.1 33.7 34.9 36.6 39.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6fa-6db4-11ee-a145-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-202", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Fr!En 14 BLEU-sb test 64 2.29 2.99 3.90 3.60 21.2 15.5 22.4 21.9 1.50 16.3 24.4 27.0 30.0 31.6 31.4 35.6 5.30 26.2 29.5 32.2 35.1 36.4 38.3 41.4\nEn!Fr 14 BLEU-mb test 45.6 64 1.74 2.16 2.73 2.15 15.1 8.82 12.0 25.2 0.49 8.00 14.8 15.9 20.3 23.3 24.9 28.3 4.08 14.5 19.3 21.5 24.9 27.3 29.5 32.6\nEn!Fr 14 BLEU-sb test 45.9 64 2.44 2.75 3.54 2.82 19.3 11.4 15.3 31.3 0.81 10.0 18.2 19.3 24.7 28.3 30.1 34.1 5.31 18.0 23.6 26.1 30.3 33.3 35.5 39.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6fb-6db4-11ee-859c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-203", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "De!En 16 BLEU-mb test 40.2 64 2.06 2.87 3.41 3.63 21.5 17.3 23.0 27.2 0.83 16.2 22.5 24.7 28.2 30.7 33.0 30.4 3.25 22.7 26.2 29.2 32.7 34.8 37.3 40.6\nDe!En 16 BLEU-sb test 64 2.39 3.27 3.85 4.04 22.5 18.2 24.4 28.6 0.93 17.1 23.4 25.8 29.2 31.9 34.5 32.1 3.60 23.8 27.5 30.5 34.1 36.5 39.1 43.0\nEn!De 16 BLEU-mb test 41.2 64 1.70 2.27 2.31 2.43 12.9 8.66 10.4 24.6 0.50 7.00 12.9 13.1 18.3 20.9 22.5 26.2 3.42 12.3 15.4 17.1 20.9 23.0 26.6 29.7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6fc-6db4-11ee-9c82-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-204", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "En!De 16 BLEU-sb test 41.2 64 2.09 2.65 2.75 2.92 13.7 9.36 11.0 25.3 0.54 7.40 13.4 13.4 18.8 21.7 23.3 27.3 3.78 12.9 16.1 17.7 21.7 24.1 27.7 30.9\nWinograd acc test 93.8 7 66.3 72.9 74.7 76.9 82.4 85.7 87.9 88.3 63.4 68.5 72.9 76.9 82.4 84.6 86.1 89.7 63.4 67.4 73.6 76.9 84.3 85.4 82.4 88.6\nWinogrande acc dev 84.6 50 52.0 52.1 57.4 58.7 62.3 64.5 67.9 70.2 51.3 53.0 58.3 59.1 61.7 65.8 66.9 73.2 51.3 52.6 57.5 59.1 62.6 67.4 70.0 77.7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6fd-6db4-11ee-9ca8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-205", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "PIQA acc dev 77.1 50 64.6 70.2 72.9 75.1 75.6 78.0 78.5 81.0 64.3 69.3 71.8 74.4 74.3 76.3 77.8 80.5 64.3 69.4 72.0 74.3 75.4 77.8 79.9 82.3 82.8\nARC (Challenge) acc test 78.5 50 26.6 29.5 31.8 35.5 38.0 41.4 43.7 51.4 25.5 30.2 31.6 36.4 38.4 41.5 43.1 53.2 25.5 28.4 32.3 36.7 39.5 43.7 44.8 51.5\nARC (Easy) acc test 92.0 50 43.6 46.5 53.0 53.8 58.2 60.2 63.8 68.8 42.7 48.2 54.6 55.9 60.3 62.6 66.8 71.2 42.7 51.0 58.1 59.1 62.1 65.8 69.1 70.1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6fe-6db4-11ee-947a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-206", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "OpenBookQA acc test 87.2 100 35.6 43.2 45.2 46.8 53.0 50.4 55.6 57.6 37.0 39.8 46.2 46.4 53.4 53.0 55.8 58.8 37.0 43.6 48.0 50.6 55.6 55.2 60.8 65.4\nQuac f1 dev 74.4 5 21.2 26.8 31.0 30.1 34.7 36.1 38.4 41.5 21.1 26.9 31.9 32.3 37.4 39.0 40.6 43.4 21.6 27.6 32.9 34.2 38.2 39.9 40.9 44.3\nRACE-h acc test 90.0 10 35.2 37.9 40.1 40.9 42.4 44.1 44.6 45.5 34.3 37.7 40.0 42.0 43.8 44.3 44.6 45.9 34.3 37.0 40.4 41.4 42.3 44.7 45.1 46.8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc6ff-6db4-11ee-a322-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-207", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "RACE-m acc test 93.1 10 42.1 47.2 52.1 52.3 54.7 54.4 56.7 58.4 42.3 47.3 51.7 55.2 56.1 54.7 56.9 57.4 42.3 47.0 52.7 53.0 55.6 55.4 58.1 58.1\nSQuADv2 em dev 90.7 16 22.6 32.8 33.9 43.1 43.6 45.4 49.0 52.6 25.1 37.5 37.9 47.9 47.9 51.1 56.0 60.1 27.5 40.5 39.2 53.5 50.0 56.6 62.6 64.9\nSQuADv2 f1 dev 93.0 16 28.3 40.2 41.4 50.3 51.0 52.7 56.3 59.5 30.1 43.6 44.1 54.0 54.1 57.1 61.8 65.4 32.1 45.5 44.9 58.7 55.9 62.1 67.7 69.8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc700-6db4-11ee-9593-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-208", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "CoQA f1 dev 90.7 5 34.5 55.0 61.8 65.3 71.1 72.8 76.3 81.5 30.6 52.1 61.6 66.1 71.8 75.1 77.9 84.0 31.1 52.0 62.7 66.8 73.2 77.3 79.9 85.0\nDROP f1 dev 89.1 20 9.40 13.6 14.4 16.4 19.7 17.0 24.0 23.6 11.7 18.1 20.9 23.0 26.4 27.3 29.2 34.3 12.9 18.7 24.0 25.6 29.7 29.7 32.3 36.5\nBoolQ acc dev 91.0 32 49.7 60.3 58.9 62.4 67.1 65.4 66.2 60.5 52.6 61.7 60.4 63.7 68.4 68.7 69.0 76.7 43.1 60.6 62.0 64.1 70.3 70.0 70.2 77.5 76.4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc701-6db4-11ee-8559-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-209", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "CB acc dev 96.9 32 0.00 32.1 8.93 19.6 19.6 28.6 19.6 46.4 55.4 53.6 53.6 48.2 57.1 33.9 55.4 64.3 42.9 58.9 53.6 69.6 67.9 60.7 66.1 82.1 75.6\nCB f1 dev 93.9 32 0.00 29.3 11.4 17.4 22.4 25.1 20.3 42.8 60.1 39.8 45.6 37.5 45.7 28.5 44.6 52.5 26.1 40.4 32.6 48.3 45.7 44.6 46.0 57.2 52.0\nCopa acc dev 94.8 32 66.0 68.0 73.0 77.0 76.0 80.0 84.0 91.0 62.0 64.0 66.0 74.0 76.0 82.0 86.0 87.0 67.0 64.0 72.0 77.0 83.0 83.0 86.0 92.0 92.0"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc702-6db4-11ee-835e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-210", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "RTE acc dev 92.5 32 47.7 49.8 48.4 56.0 46.6 55.2 62.8 63.5 53.1 47.3 49.5 49.5 54.9 54.9 56.3 70.4 52.3 48.4 46.9 50.9 56.3 49.5 60.6 72.9 69.0\nWiC acc dev 76.1 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50.0 50.3 50.3 49.2 49.4 50.3 50.0 48.6 49.8 55.0 53.0 53.0 51.6 53.1 51.1 55.3 49.4\nWSC acc dev 93.8 32 59.6 56.7 65.4 61.5 66.3 60.6 64.4 65.4 58.7 58.7 60.6 62.5 66.3 60.6 66.3 69.2 58.7 60.6 54.8 49.0 62.5 67.3 75.0 75.0 80.1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc703-6db4-11ee-a4de-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-211", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "MultiRC acc dev 62.3 32 4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6 4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6 6.09 11.8 16.8 20.8 24.7 23.8 25.0 32.5 30.5\nMultiRC f1a dev 88.2 32 57.0 59.7 60.4 59.9 60.0 64.5 71.4 72.9 57.0 59.7 60.4 59.9 60.0 64.5 71.4 72.9 45.0 55.9 64.2 65.4 69.5 66.4 69.3 74.8 75.4\nReCoRD acc dev 92.5 32 70.8 78.5 82.1 84.1 86.2 88.6 89.0 90.2 69.8 77.0 80.7 83.0 85.9 88.0 88.8 90.2 69.8 77.2 81.3 83.1 86.6 87.9 88.9 89.0 90.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc704-6db4-11ee-bbea-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-212", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "ReCoRD f1 dev 93.3 32 71.9 79.2 82.8 85.2 87.3 89.5 90.4 91.0 70.7 77.8 81.6 83.9 86.8 88.8 89.7 91.2 70.7 77.9 82.1 84.0 87.5 88.8 89.8 90.1 91.1\nSuperGLUE average dev 89.0 40.6 47.4 46.8 49.6 50.1 52.3 54.4 58.2 54.4 55.1 56.7 57.8 61.2 59.7 64.3 68.9 50.2 56.2 56.8 60.0 64.3 63.6 66.9 73.2 71.8\nANLI R1 acc test 73.8 50 33.4 34.2 33.4 33.4 34.2 32.3 33.2 34.6 32.1 31.6 31.9 34.6 30.6 31.6 32.7 32.0 32.1 32.5 30.9 32.5 33.5 33.1 33.3 36.8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc705-6db4-11ee-b31c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-213", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "ANLI R2 acc test 50.7 50 33.2 31.9 33.3 33.3 33.8 33.5 33.5 35.4 35.7 33.7 33.2 32.7 32.7 33.9 33.9 33.9 35.7 33.8 32.1 31.4 32.6 33.3 32.6 34.0\nANLI R3 acc test 48.3 50 33.6 34.0 33.8 33.4 35.3 34.8 34.4 34.5 35.0 32.6 33.0 33.9 34.1 33.1 32.5 35.1 35.0 34.4 35.1 36.0 32.7 33.9 34.5 40.2\n2D+ acc n/a 50 0.70 0.65 0.70 0.85 1.10 2.54 15.4 76.9 2.00 0.55 3.15 4.00 12.1 19.6 73.0 99.6 2.00 4.10 3.50 4.50 8.90 11.9 55.5 100.0"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc706-6db4-11ee-9806-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-214", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "2D- acc n/a 50 1.25 1.25 1.25 1.25 1.60 7.60 12.6 58.0 1.15 0.95 1.45 1.95 3.85 11.5 44.6 86.4 1.15 1.45 2.25 2.70 7.35 13.6 52.4 98.9\n3D+ acc n/a 50 0.10 0.10 0.05 0.10 0.10 0.25 1.40 34.2 0.15 0.00 0.10 0.30 0.45 0.95 15.4 65.5 0.15 0.45 0.30 0.55 0.75 0.90 8.40 80.4\n3D- acc n/a 50 0.05 0.05 0.05 0.05 0.05 0.45 1.35 48.3 0.05 0.15 0.25 0.30 0.55 1.60 6.15 78.7 0.05 0.10 0.15 0.35 0.65 1.05 9.20 94.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc707-6db4-11ee-a1ca-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-215", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "4D+ acc n/a 50 0.05 0.05 0.00 0.00 0.05 0.05 0.15 4.00 0.00 0.00 0.10 0.00 0.00 0.10 0.80 14.0 0.00 0.05 0.05 0.00 0.15 0.15 0.40 25.5\n4D- acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.10 7.50 0.00 0.00 0.00 0.00 0.05 0.00 0.50 14.0 0.00 0.05 0.00 0.00 0.10 0.05 0.40 26.8\n5D+ acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.65 0.00 0.00 0.00 0.00 0.00 0.00 0.05 3.45 0.00 0.00 0.00 0.00 0.00 0.00 0.05 9.30"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc708-6db4-11ee-8da3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-216", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "5D- acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.05 3.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.90\n2Dx acc n/a 50 2.20 2.25 2.65 2.10 2.55 5.80 6.15 19.8 1.35 2.35 3.35 2.35 4.75 9.15 11.0 27.4 1.35 2.90 2.70 2.85 4.25 6.10 7.05 29.2\n1DC acc n/a 50 1.25 2.95 2.75 0.05 0.30 2.35 0.75 9.75 1.90 2.80 2.85 3.65 6.45 9.15 8.20 14.3 1.70 2.15 3.90 5.75 6.20 7.60 9.95 21.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc709-6db4-11ee-b833-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-217", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Cycled Letters acc n/a 100 0.62 0.71 2.85 0.00 0.63 1.35 2.58 3.66 1.67 4.36 5.68 6.46 6.25 9.41 15.1 21.7 4.63 9.27 10.7 14.5 16.7 21.9 27.7 37.9\nAnagrams 1 acc n/a 100 0.10 0.14 0.40 0.00 0.27 0.69 1.16 2.28 0.21 0.61 1.12 1.27 1.60 2.72 3.72 8.62 0.50 1.27 2.13 3.05 3.81 5.49 8.38 15.1\nAnagrams 2 acc n/a 100 0.81 1.21 2.69 0.01 1.71 3.75 4.53 8.91 1.19 2.62 4.70 4.77 6.97 10.2 14.6 25.9 1.94 4.80 7.59 9.87 12.6 18.9 25.6 39.7"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc70a-6db4-11ee-bc3f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-218", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Symbol Insertion acc n/a 100 0.00 0.00 0.10 0.00 0.05 0.42 0.89 8.26 0.03 0.05 0.57 1.18 1.67 3.46 6.62 45.4 0.11 0.28 2.19 4.18 6.61 11.0 27.3 67.2\nReversed Words acc n/a 100 0.00 0.01 0.01 0.01 0.02 0.03 0.03 0.09 0.02 0.01 0.01 0.00 0.05 0.07 0.11 0.48 0.00 0.05 0.00 0.17 0.24 0.30 0.42 0.44\nSAT Analogies acc n/a 20 35.6 39.0 45.2 44.1 50.0 49.2 52.7 53.7 30.5 41.2 43.1 46.5 55.1 54.3 53.5 59.1 30.5 40.4 42.8 40.6 48.4 51.9 53.5 65.2\nTable H.1: Scores for every task, setting and model that we investigate in this paper.\n63\nFigure H.1: All results for all SuperGLUE tasks.\nFigure H.2: Results for SAT task."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc70b-6db4-11ee-ba0c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-219", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "63\nFigure H.1: All results for all SuperGLUE tasks.\nFigure H.2: Results for SAT task.\n Figure H.3: All results for all Winograd tasks.\n64\nFigure H.4: All results for all Arithmetic tasks.\nFigure H.5: All results for all Cloze and Completion tasks.\n65\nFigure H.6: All results for all Common Sense Reasoning tasks.\nFigure H.7: All results for all QA tasks.\nFigure H.8: All results for all Reading Comprehension tasks.\nFigure H.9: All results for all ANLI rounds.\n66\nFigure H.10: All results for all Scramble tasks.\nFigure H.11: All results for all Translation tasks.\n67\nReferences\n[ADG+16]Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul,\nBrendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent.\nInAdvances in neural information processing systems , pages 3981\u20133989, 2016.\n[AI19] WeChat AI. Tr-mt (ensemble), December 2019.\n[AJF19] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc70c-6db4-11ee-bc4f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-220", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[AJF19] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In\nProceedings of the 2019 Conference of the North American Chapter of the Association for Computational\nLinguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.\n[BBDIW20] Su Lin Blodgett, Solon Barocas, Hal Daum \u00b4e III, and Hanna Wallach. Language (technology) is power:\nA critical survey of \u201cbias\u201d in nlp. arXiv preprint arXiv:2005.14050 , 2020.\n[BCFL13] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from\nquestion-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language\nprocessing , pages 1533\u20131544, 2013.\n[BDD+09]Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The \ufb01fth\nPASCAL recognizing textual entailment challenge. 2009.\n[BES10] Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Sentiwordnet 3.0: an enhanced lexical"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc70d-6db4-11ee-81f5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-221", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[BES10] Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Sentiwordnet 3.0: an enhanced lexical\nresource for sentiment analysis and opinion mining. In Lrec, volume 10, pages 2200\u20132204, 2010.\n[BHDD+06]Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan\nSzpektor. The second PASCAL recognising textual entailment challenge. 2006.\n[BHT+20]Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella\nLapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language.\narXiv preprint arXiv:2004.10151 , 2020.\n[BLC13] Yoshua Bengio, Nicholas L \u00b4eonard, and Aaron C. Courville. Estimating or propagating gradients through\nstochastic neurons for conditional computation. Arxiv , 2013.\n[BZB+19]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc70e-6db4-11ee-be50-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-222", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "physical commonsense in natural language. arXiv preprint arXiv:1911.11641 , 2019.\n[Car97] Rich Caruana. Multitask learning. Machine learning , 28(1), 1997.\n[CB78] Susan Carey and Elsa Bartlett. Acquiring a single new word. Proceedings of the Stanford Child Language\nConference , 1978.\n[CCE+18]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and\nOyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv ,\nabs/1803.05457, 2018.\n[CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse\ntransformers, 2019.\n[CHI+18]Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke\nZettlemoyer. Quac : Question answering in context. Arxiv , 2018.\n[CLC+19]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc70f-6db4-11ee-963f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-223", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[CLC+19]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina\nToutanova. BoolQ: Exploring the surprising dif\ufb01culty of natural yes/no questions. arXiv preprint\narXiv:1905.10044 , 2019.\n[CLY+19]Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and\nJingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 ,\n2019.\n[Cra17] Kate Crawford. The trouble with bias. NIPS 2017 Keynote , 2017.\n[DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep\nbidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.\n68\n[DGM06] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment\nchallenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classi\ufb01cation,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc710-6db4-11ee-9445-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-224", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classi\ufb01cation,\nand recognising textual entailment , pages 177\u2013190. Springer, 2006.\n[DGV+18]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal\ntransformers. Arxiv , 2018.\n[DHKH14] Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Hea\ufb01eld. Edinburgh\u2019s phrase-based machine\ntranslation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation ,\npages 97\u2013104, 2014.\n[DL15] Andrew M. Dai and Quoc V . Le. Semi-supervised sequence learning. In Advances in neural information\nprocessing systems , 2015.\n[DMST19] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung\n23. Data can be found at https://github.com/mcdm/CommitmentBank/."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc711-6db4-11ee-80ae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-225", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "23. Data can be found at https://github.com/mcdm/CommitmentBank/.\n[DSC+16]Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast\nreinforcement learning via slow reinforcement learning. ArXiv , abs/1611.02779, 2016.\n[DWD+19]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.\nDrop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint\narXiv:1903.00161 , 2019.\n[DYY+19]Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V . Le, and Ruslan Salakhutdinov.\nTransformer-xl: Attentive language models beyond a \ufb01xed-length context. Arxiv , 2019.\n[EOAG18] Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale.\narXiv preprint arXiv:1808.09381 , 2018."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc712-6db4-11ee-9a58-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-226", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "arXiv preprint arXiv:1808.09381 , 2018.\n[FAL17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of\ndeep networks. ArXiv , abs/1703.03400, 2017.\n[Fyo00] Yaroslav Fyodorov. A natural logic inference system, 2000.\n[GG19] Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases\nin word embeddings but do not remove them. arXiv preprint arXiv:1903.03862 , 2019.\n[GLT+20]Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909 , 2020.\n[GMDD07] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing\ntextual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and\nparaphrasing , pages 1\u20139. Association for Computational Linguistics, 2007."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc713-6db4-11ee-a3fb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-227", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "paraphrasing , pages 1\u20139. Association for Computational Linguistics, 2007.\n[Gra16] Alex Graves. Adaptive computation time for recurrent neural networks. Arxiv , 2016.\n[GSL+18]Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A\nSmith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324 , 2018.\n[GSR19] Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv: 1906.04043 , 2019.\n[GWC+18]Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li. Meta-learning for low-resource\nneural machine translation. arXiv preprint arXiv:1808.08437 , 2018.\n[HB20] Daniel Hernandez and Tom Brown. Ai and ef\ufb01ciency, May 2020.\n[HBFC19] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.\nCoRR , abs/1904.09751, 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc714-6db4-11ee-82d6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-228", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "CoRR , abs/1904.09751, 2019.\n[HLW+20]Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song.\nPretrained transformers improve out of distribution robustness. arXiv preprint arXiv:2004.06100 , 2020.\n69\n[HNA+17]Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md.\nMostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.\narXiv preprint arXiv:1712.00409 , 2017.\n[HR18] Jeremy Howard and Sebastian Ruder. Universal language model \ufb01ne-tuning for text classi\ufb01cation. arXiv\npreprint arXiv:1801.06146 , 2018.\n[HVD15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv\npreprint arXiv:1503.02531 , 2015.\n[HYC01] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to Learn Using Gradient Descent."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc715-6db4-11ee-b835-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-229", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[HYC01] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to Learn Using Gradient Descent.\nInInternational Conference on Arti\ufb01cial Neural Networks , pages 87\u201394. Springer, 2001.\n[HZJ+19]Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini,\nDani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual\nevaluation. arXiv preprint arXiv:1911.03064 , 2019.\n[IBGC+14]Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daum \u00b4e III. A neural\nnetwork for factoid question answering over paragraphs. In Empirical Methods in Natural Language\nProcessing , 2014.\n[IDCBE19] Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of\ngenerated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650 , 2019.\n[JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc716-6db4-11ee-9190-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-230", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly\nsupervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 , 2017.\n[JN20] Zheng Junyuan and Gamma Lab NYC. Numeric transformer - albert, March 2020.\n[JVS+16]Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits\nof language modeling. arXiv preprint arXiv:1602.02410 , 2016.\n[JYS+19]Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.\nTinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351 , 2019.\n[JZC+19]Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. Technical report on\nconversational question answering. arXiv preprint arXiv:1909.10772 , 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc717-6db4-11ee-ae79-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-231", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "conversational question answering. arXiv preprint arXiv:1909.10772 , 2019.\n[KCR+18]Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond\nthe surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of North\nAmerican Chapter of the Association for Computational Linguistics (NAACL) , 2018.\n[KKS+20]Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.\nUni\ufb01edqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700 , 2020.\n[KMB20] Sarah E. Kreps, Miles McCain, and Miles Brundage. All the news that\u2019s \ufb01t to fabricate: Ai-generated\ntext as a tool of media misinformation, 2020.\n[KMH+20]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott\nGray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc718-6db4-11ee-aaa4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-232", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.\n[KPR+19]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red\ufb01eld, Michael Collins, Ankur Parikh, Chris Alberti,\nDanielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova,\nLlion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational\nLinguistics , 2019.\n[KR16] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. Arxiv , 2016.\n[LB02] Edward Loper and Steven Bird. Nltk: The natural language toolkit, 2002.\n[LC19] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint\narXiv:1901.07291 , 2019.\n70"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc719-6db4-11ee-ad37-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-233", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "arXiv:1901.07291 , 2019.\n70\n[LCG+19]Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint\narXiv:1909.11942 , 2019.\n[LCH+20]Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao.\nAdversarial training for large neural language models. arXiv preprint arXiv:2004.08994 , 2020.\n[LDL19] Zhongyang Li, Xiao Ding, and Ting Liu. Story ending prediction by transferable bert. arXiv preprint\narXiv:1905.07504 , 2019.\n[LDM12] Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth\nInternational Conference on the Principles of Knowledge Representation and Reasoning , 2012.\n[LGG+20]Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc71a-6db4-11ee-a701-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-234", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint\narXiv:2001.08210 , 2020.\n[LGH+15]Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. Representation\nlearning using multi-task deep neural networks for semantic classi\ufb01cation and information retrieval. In\nProceedings of the 2015 Conference of the North American Chapter of the Association for Computational\nLinguistics: Human Language Technologies , 2015.\n[LH17] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint\narXiv:1711.05101 , 2017.\n[LHCG19a] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural\nnetworks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482 ,\n2019.\n[LHCG19b] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for\nnatural language understanding. arXiv preprint arXiv:1901.11504 , 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc71b-6db4-11ee-9694-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-235", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "natural language understanding. arXiv preprint arXiv:1901.11504 , 2019.\n[Lin20] Tal Linzen. How can we accelerate progress towards human-like linguistic generalization? arXiv preprint\narXiv:2005.00955 , 2020.\n[LLG+19]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,\nVes Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural\nlanguage generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.\n[LM17] Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441 , 2017.\n[LOG+19]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,\nLuke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach.\narXiv preprint arXiv:1907.11692 , 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc71c-6db4-11ee-8796-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-236", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "arXiv preprint arXiv:1907.11692 , 2019.\n[LPP+20]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,\nHeinrich K \u00a8uttler, Mike Lewis, Wen-tau Yih, Tim Rockt \u00a8aschel, Sebastian Riedel, and Kiela Douwe.\nRetrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401 ,\n2020.\n[LSP+18]Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam\nShazeer. Generating Wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198 , 2018.\n[LWS+20]Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez.\nTrain large, then compress: Rethinking model size for ef\ufb01cient training and inference of transformers,\n2020.\n[LXL+17]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc71d-6db4-11ee-9192-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-237", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[LXL+17]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading\ncomprehension dataset from examinations. arXiv preprint arXiv:1704.04683 , 2017.\n[LYN+20]Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy\nLin. Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380 , 2020.\n[Mac92] David. MacKay. Information-based objective functions for active data selection. Neural Computation ,\n1992.\n71\n[MBXS17] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems , pages 6294\u20136305,\n2017.\n[MCCD13] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Ef\ufb01cient estimation of word representations\nin vector space. arXiv preprint arXiv:1301.3781 , 2013."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc71e-6db4-11ee-bca3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-238", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "in vector space. arXiv preprint arXiv:1301.3781 , 2013.\n[MCH+16]Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende,\nPushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of\ncommonsense stories. arXiv preprint arXiv:1604.01696 , 2016.\n[MCKS18] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity?\na new dataset for open book question answering. ArXiv , abs/1809.02789, 2018.\n[MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of\nlarge-batch training, 2018.\n[MKM+94]Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson,\nKaren Katz, and Britta Schasberger. The penn treebank: annotating predicate argument structure.\nInProceedings of the workshop on Human Language Technology , pages 114\u2013119. Association for\nComputational Linguistics, 1994."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc71f-6db4-11ee-a368-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-239", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "InProceedings of the workshop on Human Language Technology , pages 114\u2013119. Association for\nComputational Linguistics, 1994.\n[MKXS18] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language\ndecathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730 , 2018.\n[MPL19] R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic\nheuristics in natural language inference. arXiv preprint arXiv:1902.01007 , 2019.\n[MWZ+18]Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson,\nElena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting, 2018.\n[NBR20] Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained\nlanguage models. arXiv preprint arXiv:2004.09456 , 2020.\n[NK19] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc720-6db4-11ee-b609-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-240", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[NK19] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments.\narXiv preprint arXiv:1907.07355 , 2019.\n[Nor09] Peter Norvig. Natural language corpus data, 2009.\n[NvNvdG19] Malvina Nissim, Rik van Noord, and Rob van der Goot. Fair is better than sensational: Man is to doctor\nas woman is to doctor. arXiv preprint arXiv:1905.09866 , 2019.\n[NWD+19]Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial\nnli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599 , 2019.\n[oR16] University of Regensburg. Fascha, 2016.\n[PCC18] Mohammad Taher Pilehvar and Jose Camacho-Collados. WIC: 10,000 example pairs for evaluating\ncontext-sensitive representations. arXiv preprint arXiv:1808.09121 , 2018.\n[PFB18] Jason Phang, Thibault F \u00b4evry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc721-6db4-11ee-9471-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-241", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[PFB18] Jason Phang, Thibault F \u00b4evry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary\ntraining on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 , 2018.\n[PHR+18]Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and\nBenjamin Van Durme. Collecting diverse natural language inference problems for sentence representation\nevaluation. In Proceedings of EMNLP , 2018.\n[PKL+16]Denis Paperno, Germ \u00b4an Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro\nPezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern \u00b4andez. The lambada dataset: Word prediction\nrequiring a broad discourse context. arXiv preprint arXiv:1606.06031 , 2016.\n[PNZtY18] Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen tau Yih. Dissecting contextual word\nembeddings: Architecture and representation, 2018."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc722-6db4-11ee-ad3b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-242", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "embeddings: Architecture and representation, 2018.\n[Pos18] Matt Post. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771 , 2018.\n72\n[PSM14] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word\nrepresentation. In Proceedings of the 2014 conference on empirical methods in natural language\nprocessing (EMNLP) , 2014.\n[QIA20] QIANXIN. Sa-net on albert (ensemble), April 2020.\n[QMZH19] Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. Reducing gender bias in word-level language\nmodels with a gender-equalizing loss function. arXiv preprint arXiv:1905.12801 , 2019.\n[RBG11] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An\nevaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series , 2011.\n[RCM19] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering\nchallenge. Transactions of the Association for Computational Linguistics , 7:249\u2013266, 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc723-6db4-11ee-b1cd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-243", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "challenge. Transactions of the Association for Computational Linguistics , 7:249\u2013266, 2019.\n[RCP+17]Scott Reed, Yutian Chen, Thomas Paine, A \u00a8aron van den Oord, SM Eslami, Danilo Rezende, Oriol\nVinyals, and Nando de Freitas. Few-shot autoregressive density estimation: Towards learning to learn\ndistributions. arXiv preprint arXiv:1710.10304 , 2017.\n[RJL18] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don\u2019t know: Unanswerable questions for\nsquad. arXiv preprint arXiv:1806.03822 , 2018.\n[RL16] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR 2017 (oral) ,\n2016.\n[RLL+19]Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. NumNet: Machine reading comprehension\nwith numerical reasoning. In Proceedings of EMNLP , 2019.\n[RNLVD18] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc724-6db4-11ee-9f0b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-244", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[RNLVD18] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in\ncoreference resolution. arXiv preprint arXiv:1804.09301 , 2018.\n[RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding\nby generative pre-training, 2018.\n[Ros12] R.S. Ross. Guide for conducting risk assessments. NIST Special Publication , 2012.\n[RRBS19] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of\nthe generalization error across scales, 2019.\n[RRS20] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters\nof a language model? arXiv preprint arXiv:2002.08910 , 2020.\n[RSR+19]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi\nZhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a uni\ufb01ed text-to-text\ntransformer, 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc725-6db4-11ee-b206-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-245", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a uni\ufb01ed text-to-text\ntransformer, 2019.\n[RWC+19]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language\nmodels are unsupervised multitask learners, 2019.\n[SBBC19] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial\nwinograd schema challenge at scale, 2019.\n[SBC+19]Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-V oss, Jeff Wu, Alec Radford,\nGretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris\nMcGuf\ufb01e, and Jasmine Wang. Release strategies and the social impacts of language models, 2019.\n[SCNP19] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a\nbabysitter: On biases in language generation. arXiv preprint arXiv:1909.01326 , 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc726-6db4-11ee-b253-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-246", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326 , 2019.\n[SDCW19] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of\nBERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.\n[SDSE19] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AI. CoRR , abs/1907.10597, 2019.\n[SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with\nmonolingual data. arXiv preprint arXiv:1511.06709 , 2015.\n73\n[SMM+17]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff\nDean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint\narXiv:1701.06538 , 2017."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc727-6db4-11ee-8efa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-247", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "arXiv:1701.06538 , 2017.\n[SPP+19]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.\nMegatron-lm: Training multi-billion parameter language models using model parallelism, 2019.\n[SS20] Timo Schick and Hinrich Sch \u00a8utze. Exploiting cloze questions for few-shot text classi\ufb01cation and natural\nlanguage inference. arXiv preprint arXiv:2001.07676 , 2020.\n[STQ+19]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence\npre-training for language generation. arXiv preprint arXiv:1905.02450 , 2019.\n[TFR+17]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain\nrandomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ\ninternational conference on intelligent robots and systems (IROS) , pages 23\u201330. IEEE, 2017."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc728-6db4-11ee-a647-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-248", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "international conference on intelligent robots and systems (IROS) , pages 23\u201330. IEEE, 2017.\n[TL05] Peter D. Turney and Michael L. Littman. Corpus-based learning of analogies and semantic relations.\nCoRR , abs/cs/0508103, 2005.\n[TL18] Trieu H. Trinh and Quoc V . Le. A simple method for commonsense reasoning. arXiv preprint\narXiv:1806.02847 , 2018.\n[TLBS03] Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. Combining independent\nmodules to solve multiple-choice synonym and analogy problems. CoRR , cs.CL/0309035, 2003.\n[Tur20] Project Turing. Microsoft research blog, Feb 2020.\n[VBL+16]Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching Networks for One\nShot Learning. In Advances in neural information processing systems , pages 3630\u20133638, 2016.\n[VSP+17]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, \u0141ukasz"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc729-6db4-11ee-81c6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-249", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing\nsystems , 2017.\n[WPN+19]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer\nLevy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems , pages 3261\u20133275, 2019.\n[WXH+18]Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Multi-agent\ndual learning. ICLR 2019 , 2018.\n[XDH+19]Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V . Le. Unsupervised data\naugmentation for consistency training, 2019.\n[YdC+19]Dani Yogatama, Cyprien de Masson d\u2019Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski,\nLingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc72a-6db4-11ee-9c7f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-250", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating\ngeneral linguistic intelligence. arXiv preprint arXiv:1901.11373 , 2019.\n[YDY+19]Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V . Le. XLNet:\nGeneralized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 ,\n2019.\n[ZHB+19]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine\nreally \ufb01nish your sentence? arXiv preprint arXiv:1905.07830 , 2019.\n[ZHR+19]Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin\nChoi. Defending against neural fake news. arXiv preprint arXiv:1905.12616 , 2019.\n[ZLL+18]Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc72b-6db4-11ee-9334-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2005.14165-251", "title": "Language Models are Few-Shot Learners", "source": "http://arxiv.org/pdf/2005.14165", "start_index": 0, "chroma:document": "[ZLL+18]Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.\nReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv\npreprint arXiv:1810.12885 , 2018.\n[ZSW+19a] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul\nChristiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.\n74\n[ZSW+19b] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv , abs/1909.08593,\n2019.\n75"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc72c-6db4-11ee-bd72-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-0", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Uncertainty-Aware Weakly Supervised Action\nDetection from Untrimmed Videos\nAnurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia Schmid\nGoogle Research\nfaarnab, chensun, anagrani, cordelias g@google.com\nAbstract. Despite the recent advances in video classi\fcation, progress\nin spatio-temporal action recognition has lagged behind. A major contributing factor has been the prohibitive cost of annotating videos frameby-frame. In this paper, we present a spatio-temporal action recognition\nmodel that is trained with only video-level labels, which are signi\fcantly\neasier to annotate. Our method leverages per-frame person detectors\nwhich have been trained on large image datasets within a Multiple Instance Learning framework. We show how we can apply our method in\ncases where the standard Multiple Instance Learning assumption, that\neach bag contains at least one instance with the speci\fed label, is invalid using a novel probabilistic variant of MIL where we estimate the\nuncertainty of each prediction. Furthermore, we report the \frst weaklysupervised results on the AVA dataset and state-of-the-art results among\nweakly-supervised methods on UCF101-24.\nKeywords: spatio-temporal action recognition, weak supervision, video"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc72d-6db4-11ee-87f6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-1", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "weakly-supervised methods on UCF101-24.\nKeywords: spatio-temporal action recognition, weak supervision, video\nunderstanding, mulitple instance learning\n1 Introduction\nVideo classi\fcation has witnessed great advances recently due to large datasets\nsuch as Kinetics [ 20] and Moments in Time [ 30] which have enabled training\nof specialised neural network architectures for video [ 5,9]. However, progress in\nother video understanding tasks, such as spatio-temporal action detection, has\nlagged behind in comparison. There are fewer datasets for action recognition,\nwhich are also signi\fcantly smaller than their video-classi\fcation counterparts. A\nreason for this is the exorbitant cost of annotating videos with spatio-temporal\nlabels { each frame of an action has to be manually labelled with a bounding\nbox. Moreover, annotating temporal boundaries of actions is not only arduous,\nbut often ambiguous with annotators failing to reach consensus about the start\nand end times of an action [6,39].\nIn this paper, we propose a method to train spatio-temporal action detectors\nusing only weak, video-level annotations as shown in Fig. 1. To achieve this, we\nleverage image-based person detectors which have been trained on large image\ndatasets such as Microsoft COCO [ 26] and are accurate across large variations"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc72e-6db4-11ee-ac61-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-2", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "leverage image-based person detectors which have been trained on large image\ndatasets such as Microsoft COCO [ 26] and are accurate across large variations\nin appearance, scene and pose. We adopt a Multiple Instance Learning (MIL)arXiv:2007.10703v1  [cs.CV]  21 Jul 2020\n2 A. Arnab et al.\nTraining dataPrediction\nLabels: \nsit, talk to (people), watch (a person), carry/hold (an object), listen to (a person)\n sit, carry/hold (an object), listen to (a person), talk to (people), watch (a person)\nsit, talk to (people), watch (a person)\nsit, listen to (a person), watch (a person)\ntime\nFig. 1. We propose a method to train a spatio-temporal action detector using only\nweak, video-level labels on challenging, real-world datasets. Note that the video-level\nlabels that we have may apply to multiple people in the video, and that these labels\nmay only be active for an unannotated time interval of the input clip.\nframework, where a person tubelet is an instance, and all person tubelets in the\nvideo form a bag. An important consideration in our approach is the presence of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc72f-6db4-11ee-8beb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-3", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "framework, where a person tubelet is an instance, and all person tubelets in the\nvideo form a bag. An important consideration in our approach is the presence of\nlabel noise: this is introduced from using o\u000b-the-shelf person detectors which have\nnot been trained on the video-dataset of interest, and also the fact that we have\nto sample tubelets from large bags in long videos due to memory constraints. In\nboth of these scenarios, the standard Multiple Instance Learning assumption [ 7],\nthat each bag contains at least one instance with the bag-level label, may be\nviolated. We are not aware of previous work that has explicitly addressed this\nproblem, and we do so with a probabilistic variant of MIL where we estimate the\nuncertainty of an instance-level prediction.\nUsing our approach, we obtain state-of-the-art results among weakly-supervised\nmethods on the UCF101-24 dataset. Furthermore, we report, to our knowledge,\nthe \frst weakly-supervised results on the AVA dataset (the only large-scale\ndataset for spatio-temporal action recognition), where we also show the accuracy\ntrade-o\u000bs when annotating video-clips for time intervals of varying durations.\n2 Related Work\nMost prior work on spatio-temporal action recognition has been fully-supervised."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc730-6db4-11ee-bb15-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-4", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "2 Related Work\nMost prior work on spatio-temporal action recognition has been fully-supervised.\nInitial approaches in the area used 3D sliding window detectors in conjunction\nwith handcrafted, volumetric features [ 21,25]. Current state-of-the-art approaches\nare temporal extensions of object detection architectures [ 19,41,52,35] such as\nFaster-RCNN [ 37] and SSD [ 27]. These approaches predict bounding boxes around\nthe action in a frame, using as input either a single frame along with optical\nUncertainty-Aware Weakly Supervised Action Detection 3\n\row to capture temporal information [ 41,38] or multiple frames at the input to\nprovide temporal context [ 19]. The predicted bounding boxes are then linked\nover time using an online, greedy algorithm or dynamic programming to create\nspatio-temporal tracks. Our work builds on these methods by also utilising a\ndetection architecture and spatio-temporal linking. However, these approaches\nall require bounding box annotations at each frame in the video whilst we only\nuse video-level labels which are signi\fcantly cheaper to acquire.\nWeakly supervised approaches to spatio-temporal action recognition have\nalso been explored before as they enable a signi\fcant reduction in annotation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc731-6db4-11ee-868e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-5", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Weakly supervised approaches to spatio-temporal action recognition have\nalso been explored before as they enable a signi\fcant reduction in annotation\ntime and cost. Relevant to our approach is the work of [ 6]. Che\u0013 ron et al. [6] also\nuse person detections, and infer their action labels using a formulation based on\ndiscriminative clustering [ 2]. Although their approach allows them to incorporate\ndi\u000berent types of supervision, it e\u000bectively learns a linear classi\fer on top of\npretrained, deep features. Our method in contrast is learned fully end-to-end.\nMettes et al. [29] also employed Multiple Instance Learning (MIL), but used\naction proposals [ 47] instead of the human detections used by our work and [ 6].\nHowever, [ 29], rely on additional cheap \\point\" annotations (a single spatial point\nannotated for a subset of the frames which constitute the action) which also\nensures that the standard MIL assumption is not violated. In follow-up work [ 28],\nthe authors removed the need for \\point\" annotations by incorporating biases\n(i.e.the presence of objects in the video, a bias that actions typically occur in the\ncentre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc732-6db4-11ee-b2dd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-6", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "centre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple\nInstance Learning framework in conjunction with human detections. The authors,\nhowever, assumed that sparse spatial supervision was present ( i.e.bounding\nboxes for a small subset of frames in the action tube), unlike our method which\nrequires video-level labels alone.\nWe also note that many approaches have addressed temporal action detection\n(localising actions in time but not space) with only video-level tags as supervision [ 48,42,31,34]. UntrimmedNets [ 48] uses a network with two branches, a\nclassi\fcation module to perform action classi\fcation and a selection module to\nselect relevant frames.Hide-and-Seek [ 42] obtains more precise temporal boundaries by forcing the network to attend to more discriminative frames by randomly\nhiding parts of videos. However, these methods are trained and evaluated on\ndatasets such as ActivityNet [ 4] and THUMOS14 [ 18], which contain mostly one\naction per video, and are thus signi\fcantly less challenging than datasets such as\nAVA [14] which we evaluate on.\nFinally, we note that another approach to combat the e\u000bort of dataset\nannotation has been various forms of self-supervised learning, where discriminative"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc733-6db4-11ee-977b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-7", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Finally, we note that another approach to combat the e\u000bort of dataset\nannotation has been various forms of self-supervised learning, where discriminative\nfeature representations can be learned with unlabelled data. Examples in video\ninclude cross-modal self-supervised learning by learning correspondences between\nthe audio and image streams readily available in videos [ 1,33,51], transcribed\nspeech [ 44] or using meta-data such as hashtags [ 10] as a form of weak labelling.\nSelf-supervised approaches, however, are complementary to our approach, as they\nstill require a limited amount of fully-labelled data for the \fnal task of interest.\n4 A. Arnab et al.\nIn our weakly-supervised action detection scenario, we never have access to full,\nspatio-temporal ground-truth annotations for a single training example.\n3 Proposed Approach\nAs shown in Fig. 1, given a set of video clips, with only clip-level annotations of\nthe actions taking place, our goal is to learn a model to recognise and localise\nthese actions in space and time. Our method is based on Multiple Instance\nLearning (MIL) which we brie\ry review in Sec. 3.1. Thereafter, we show how we"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc734-6db4-11ee-8c09-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-8", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Learning (MIL) which we brie\ry review in Sec. 3.1. Thereafter, we show how we\nuse it for weakly-supervised spatio-temporal action recognition in Sec. 3.2. We\nthen describe how the standard MIL assumption, is often violated in our scenario\nand describe a method to mitigate this by leveraging uncertainty estimates by\nour network in Sec. 3.3. Finally, we discuss implementation details of our network\nin Sec. 3.4.\n3.1 Multiple Instance Learning\nIn the standard Multiple Instance Learning (MIL) [ 7] formulation, one is given a\nbag ofNinstances, denoted as x=fx1;x2;:::;xNg. The class labels for each\nof the instances is unknown, but the label for the entire bag, x, is known. The\nstandard MIL assumption is that a bag is assigned a class label if at least one\ninstance in the bag is associated with this label. More formally, we consider the\nmulti-label classi\fcation case, where the label vector for the bag is y2RC, and\nyl= 1 if there is at least one instance with the lthlabel is present in the bag,\nandyl= 0 otherwise. Note that each bag can be labelled with multiple of the C\nclass labels."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc735-6db4-11ee-8b96-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-9", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "andyl= 0 otherwise. Note that each bag can be labelled with multiple of the C\nclass labels.\nOur goal is to train an instance-level classi\fer (parameterised as a neural\nnetwork), that predicts p(yl= 1jxj), or the label probabilities for the jthinstance.\nHowever, as we only have the labels for the entire bag, and not each instance,\nMIL methods aggregate the set of instance-level probabilities, fpijgfor a bag\ni, to bag-level probabilities, pi, using an aggregation function, g(\u0001), where the\nprobabilities are obtained from a suitable activation function (sigmoid or softmax)\non the logits output by the neural network:\np(yil= 1jx1;x2;:::;xN) =g(pi1;pi2;:::;piN): (1)\nOnce we have bag-level predictions, we can apply a standard classi\fcation\nloss between the bag-level probabilities and bag-level ground truth, and train a\nneural network with stochastic gradient descent. Since we consider the multi-label\nclassi\fcation case, we use the binary cross-entropy:\nLce(x;y) =\u0000NbX\niCX"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc736-6db4-11ee-bc08-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-10", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "classi\fcation case, we use the binary cross-entropy:\nLce(x;y) =\u0000NbX\niCX\nlyillogpil+ (1\u0000yil) log(1\u0000pil) (2)\nNote that we de\fned pilas the bag-level probability of the ithbag taking the\nlthlabel, which is obtained using Eq. 1, and Nbis the number of bags in the\nmini-batch.\nUncertainty-Aware Weakly Supervised Action Detection 5\nAggregation The aggregation function, g(\u0001), can naturally be implemented in\nneural networks as a global pooling function over all outputs of the network.\nCommon, permutation-invariant pooling functions include, max-pooling, generalised mean-pooling and log-sum-exponential (LSE) pooling [ 3] (a smooth and\nconvex approximation of the maximum function) respectively:\ng(fpjg) = max\njpj (3)\ng(fpjg) =0\n@1\njjjX\njpr\nj1\nA1\nr\n(4)\ng(fpjg) =1\nrlog0\n@1\njjjX\njer\u0001pj1\nA (5)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc737-6db4-11ee-9839-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-11", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "A1\nr\n(4)\ng(fpjg) =1\nrlog0\n@1\njjjX\njer\u0001pj1\nA (5)\nMax-pooling only considers the top-scoring instance in the bag, and thus naturally\ncaptures the MIL assumption that at least one instance in the bag has the\nspeci\fed, bag-level label. Moreover, it can also be more robust to instances in\nthe bag that do not have the bag-level label. However, mean and LSE pooling\nhave been employed in applications such as weakly-supervised segmentation\n[36], object recognition [ 45] and medical imaging [ 24] where multiple instances\nin the bag do typically have the bag-level label. Note that higher values of\ntherhyperparameter for both these functions increases their \\peakiness\" and\napproximates the maximum value. For our scenario, detailed in the next section,\nwe found max-pooling to be the most appropriate.\n3.2 Weakly-supervised spatio-temporal action recognition as\nmultiple instance learning\nOur goal is to learn a model to recognise and localise actions in space and\ntime given only video-level annotations. To facilitate this, we leverage a person\ndetector that has been trained on a large image dataset, i.e.Microsoft COCO [ 26]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc738-6db4-11ee-9848-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-12", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "time given only video-level annotations. To facilitate this, we leverage a person\ndetector that has been trained on a large image dataset, i.e.Microsoft COCO [ 26].\nConcretely, we run a person detector on our training videos, and create person\ntubelets which are person detections over Kconsecutive frames in the video.\nOur bag for multiple instance learning thus consists of all the tubelets within a\nvideo, and is annotated with the video-level labels that we have as supervision,\nas illustrated in Fig. 2. Note that the size of the bag varies for every video clip, as\nthe bag size is determined by the length of the video and the number of detected\npeople.\nAs shown in Fig. 2, our network architecture for this task is a Fast-RCNN [ 12]\nstyle detector that has been extended temporally. Given a video clip of Kframes,\nand proposals which in our case are person detections, the network classi\fes the\naction(s) taking place at the centre frame of each proposal, given the temporal\ncontext of the K\u00001 frames around it.\nNote that the spatio-temporal localisation task is e\u000bectively factorised: the\nspatial localisation capability of the model depends on the quality of the person\n6 A. Arnab et al."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc739-6db4-11ee-99dd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-13", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "spatial localisation capability of the model depends on the quality of the person\n6 A. Arnab et al.\nCNN feature extractorROI AlignClassi\ufb01er\nUncertainty-aware MILLoss \u2112Multiple Instance Learning Bag\nProbabilities, pUncertainties,  \u03c3\nBag-level label\n\u2026\nstand, listen to (a person), watch (a person), \nwalk, carry/hold (an object) \u2026\nFig. 2. Overview of our approach for training an action detector in a weakly-supervised\nmanner using multiple instance learning: Each bag consists of all the tubelets that have\nbeen extracted from the video-clip. These tubelets are obtained using an o\u000b-the-shelf\nperson detector which has not been trained on the dataset of interest. These tubelets\nact as proposals for a Fast-RCNN style detector operating on a sequence of rgb images.\nThe predictions for each of the tubelets in the bag are then aggregated together, and\ncompared to the bag-level label. Uncertainty estimates produced by the network are\nused to compensate for noise in the bag-level labels during training.\ndetections. Temporal localisation, on the other hand, is performed by linking\nperson tubelets through the video as commonly done in the literature [ 19,41,52,6],"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc73a-6db4-11ee-84f2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-14", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "detections. Temporal localisation, on the other hand, is performed by linking\nperson tubelets through the video as commonly done in the literature [ 19,41,52,6],\nsince this method can scale to arbitrarily long videos. We use the same algorithm\nas Kalogeiton et al. [19] which links together detections within a small temporal\nwindow greedily based on the spatial intersection over union (IoU) between\nbounding boxes on consecutive frames.\nFinally, note that for a video consisting of Tframes, the bag could consist of\nT\u0000K+1 person tubelets if a person is detected on each frame of the video, and a\ntubelet is started from each frame. Due to memory limitations, it is infeasible to \ft\nan entire bag onto a GPU for training. As a result, we uniformly sample instances\nfrom each bag during training, whilst still retaining the original bag-level label.\nThis introduces additional noise into the problem, as detailed next.\n3.3 Label noise and violation of the standard MIL assumption\nThe standard MIL assumption, that at least one instance in the bag is assigned\nthe bag-level label is often violated in our scenario. There are two primary factors\nfor this: Firstly, due to computational constraints, we cannot process a whole bag\nat a time, but must instead sample instances from a bag. It is therefore possible"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc73b-6db4-11ee-bf88-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-15", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "for this: Firstly, due to computational constraints, we cannot process a whole bag\nat a time, but must instead sample instances from a bag. It is therefore possible\nto sample a bag that does not contain any tubelets with the labelled action. The\nUncertainty-Aware Weakly Supervised Action Detection 7\nPredicted \nuncertainty\n, \n\u0000\n\u0000\nPredicted probability\n, \n\u0000\n\u0000\nb\na\nc\nWrong prediction. Low \nuncertainty. Highest loss\nCorrect prediction. Low \nuncertainty. Lowest loss\nIncorrect prediction. High uncertainty. Moderate loss\nFig. 3. The loss surface of our uncertainty-based loss (Eq. 6). The ground truth binary\nlabel in this example is 1. Hence, the loss is minimised when the network predicts\na high probability and a low uncertainty (point \\a\"). However, making an incorrect\nprediction with a high uncertainty is not penalised as much (point \\b\"), and is suitable\nfor cases when the input bags are noisy and the bag-level label is not present in any of\nthe tubelets. Finally, predicting the incorrect label with a low uncertainty is penalised\nthe most (point \\c\"). Best viewed in colour.\nlikelihood of this occurring is inversely proportional to the ratio of the duration"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc73c-6db4-11ee-9816-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-16", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "the most (point \\c\"). Best viewed in colour.\nlikelihood of this occurring is inversely proportional to the ratio of the duration\nof the labelled action to the total video length. Secondly, in a weakly-supervised\nscenario, we use person detectors that are not trained on the video dataset of\ninterest. Consequently, there can be failures in the detector, especially when\nthere is a large domain gap between the detector's training distribution and the\nvideo dataset. False negatives (missing detections for people in the scene) are a\nparticular issue because it is possible that we do not have a single person tubelet\nin the bag that corresponds to the labelled action.\nTherefore, there are cases when there is no tubelet which actually has the baglevel label. To handle these cases, inspired by [ 22,32], we modify the network to\nadditionally predict the uncertainty \u001b2RCfor each binary label for all tubelets\nin the bag. Intuitively, to minimise the training error, the network can predict the\nbag-level label with low uncertainty or it can predict a high uncertainty to avoid\nbeing penalised heavily for noisy bags where the bag-level label is not present in\nany of the tubelets. The \fnal loss, in conjunction with the original cross entropy,\nis de\fned as:\nL(x;y;\u001b ) =1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc73d-6db4-11ee-8216-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-17", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "any of the tubelets. The \fnal loss, in conjunction with the original cross entropy,\nis de\fned as:\nL(x;y;\u001b ) =1\n\u001b2Lce(x;y) + log\u001b2(6)\nAs shown by [ 23], this corresponds to assuming a Boltzmann distribution on the\noutput of the network with a temperature of \u001b2, and approximately minimising\nits log-likelihood.\n8 A. Arnab et al.\nThe loss surface of this probabilistic loss is visualised in Fig. 3. Note how the\nloss is the lowest when the predicted label is correct and there is low uncertainty.\nHowever, the loss is not excessive if the incorrect label is predicted with a\nhigh uncertainty. This is in contrast with the standard cross-entropy loss which\npenalises incorrect predictions heavily.\n3.4 Network architecture and implementation\nOur action detector is similar to Fast-RCNN [ 12] using the SlowFast [ 9] video\nnetwork architecture based on the ResNet-50 backbone [ 16] pretrained on Kinetics [ 20]. As described in Sec. 3.2, we use region proposals obtained from a\nFaster-RCNN detection model trained with Detectron [ 13]. Region-of-interest\nfeatures [ 12] are extracted from the last feature map of \\res5\" using RoIAlign [ 15]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc73e-6db4-11ee-a1cb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-18", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "features [ 12] are extracted from the last feature map of \\res5\" using RoIAlign [ 15].\nOur choice for this architecture is motivated by the fact that it is simple and has\nachieved state-of-the-art results on the AVA dataset [ 14] in a fully-supervised\nsetting [ 9]. Note that our network does not use additional optical \row inputs\n(which can be considered as an additional source of supervision) as common in\nother video architectures [5,19,41,6].\nWe predict the uncertainty, \u001b2RCfor each of the Cbinary labels de\fned\nby the dataset for each tubelet. As we use max-pooling to aggregate the tubelet\npredictions, we select the uncertainty prediction corresponding to the selected\ntubelet for computing the loss. For numerical stability, we predict v:=log\u001b2\nwith our network, using the \\softplus\", f(x) =log(1 + exp(\u0000x)), activation\nfunction to ensure positivity. We then compute1\n\u001b2=exp(\u0000v), and avoid the\npossibility of dividing by 0 which could be the case if we predicted \u001b2directly\nwith the network.\nWe train our network with synchronous stochastic gradient descent (SGD),\nusing 8 GPUs and a batch size of 4 on each GPU. In our case, each element of a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc73f-6db4-11ee-9266-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-19", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "We train our network with synchronous stochastic gradient descent (SGD),\nusing 8 GPUs and a batch size of 4 on each GPU. In our case, each element of a\nbatch is of a bag from Multiple Instance Learning. Each bag samples a maximum\nof 4 tubelets. Each tubelet itself consists of 16 frames.\n4 Experiments\n4.1 Experimental set-up\nWe evaluate our method on UCF101-24 and AVA, described in more detail below.\nNote that other video datasets such as THUMOS [ 18] and ActivityNet [ 4] are not\nsuitable for spatiotemporal localisation, as they lack bounding box annotations.\nUCF101-24: UCF101-24 is a subset of the UCF101 [ 43] dataset, consisting of 24\naction classes with spatio-temporal localisation annotation, released as bounding\nbox annotations of humans.Although each video contains only a single action\nclass, it may contain multiple individuals performing the action with di\u000berent\nspatial and temporal boundaries. Moreover, there may also be people present in\nthe video who are not performing any labelled action. Following standard practice,\nUncertainty-Aware Weakly Supervised Action Detection 9\nTable 1. Ablation study of di\u000berent variants of our method on the UCF101-24 validation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc740-6db4-11ee-afb8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-20", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Uncertainty-Aware Weakly Supervised Action Detection 9\nTable 1. Ablation study of di\u000berent variants of our method on the UCF101-24 validation\nset. We report the Video mAP at IoU thresholds of 0.2 and 0.5 respectively.\nVideo AP\n0.2 0.5\nWeakly supervised baseline 54.3 29.7\nMIL - LSE pooling 60.1 33.1\nMIL - mean pooling 60.3 33.0\nMIL - max pooling 60.7 33.5\nMIL - max pooling, uncertainty 61.7 35.0\nFully supervised 69.3 43.6\nwe use the corrected annotations of [ 41] and report the mean average precision\nat a video level (Video AP) for the \frst split of the dataset. For evaluating the\nVideo AP, we link tubelets together using the algorithm of [19].\nAVA [ 14]:This dataset consists of 430, 15-minute video clips obtained from\nmovies. 80 atomic visual actions are annotated exhaustively for all people in\nthe video, where one person is often simultaneously performing multiple actions.\nThe dataset annotates keyframes at every second in the video. Following standard practice, we report the Frame AP at an IoU threshold of 0.5 using v2.2\nannotations."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc741-6db4-11ee-aac0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-21", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "The dataset annotates keyframes at every second in the video. Following standard practice, we report the Frame AP at an IoU threshold of 0.5 using v2.2\nannotations.\n4.2 Experiments on UCF101-24\nWe \frst conduct ablation studies of our model on the UCF101-24 dataset. We\ndiscard the spatio-temporal annotations for the whole untrimmed video, and so\nour bag in multiple instance learning contains tubelets from the whole video.\nAblation study Table 1 ablates di\u000berent variants of our method: The most na\u007f \u0010ve\nbaseline is to not perform any multiple instance learning, and to simply train in\na fully-supervised fashion assuming that the label of a tubelet is the video-level\nlabel. As shown in the \frst row of Tab. 1, this method performs the worst as\nthe assumed tubelet-level labels are often incorrect. The use of multiple instance\nlearning improves results, with the various aggregation functions performing\nsimilarly. Max-pooling, however, performs the best, and we believe this is because\nthe max operation is the most suitable for dealing with the noise present in our\ntubelets as described in Sec. 3.3. Note that for mean and LSE-pooling, we set\nr= 1. Finally, introducing our uncertainty-based loss function improves results"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc742-6db4-11ee-a7d4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-22", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "tubelets as described in Sec. 3.3. Note that for mean and LSE-pooling, we set\nr= 1. Finally, introducing our uncertainty-based loss function improves results\neven further, obtaining a Video mAP of 35.0 at a threshold of 0.5. This is 80%\nof the performance achieved by our fully-supervised baseline.\nPerson detections on UCF101-24 Note that for our weakly-supervised experiments, the person tubelets for training are obtained from a Faster-RCNN [ 37]\nperson detector that has only been trained on Microsoft COCO [ 26]. There is a\n10 A. Arnab et al.\nsigni\fcant domain gap between COCO and UCF, and the annotation protocol of\nperson boxes on UCF is also not consistent (for example, the bounding box for a\nperson riding a horse often includes the horse in UCF) with that of COCO. These\ndiscrepancies are re\rected by the fact that our person detections used during\ntraining only have a recall of 46.9% compared to the ground truth person boxes,\nwhen using an IoU threshold of 0.5 to signify a correct match. Furthermore,\nthe precision of our person tubelets on the training set is only 21.1%. A major\ncontributing factor to this is that UCF action annotations are not exhaustive"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc743-6db4-11ee-a2b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-23", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "the precision of our person tubelets on the training set is only 21.1%. A major\ncontributing factor to this is that UCF action annotations are not exhaustive\n{ there may be people in the video who are not labelled at all as they are not\nperforming an annotated action. These people will, however, still be detected by\na COCO-trained detector and considered as false positives during this evaluation.\nThe fact that we are able to train our model with these annotations demonstrates the ability of our multiple instance learning method to handle label noise\nin the training set. The inconsistencies in the UCF101-24 dataset labelling are\ndetailed further in the supplementary, and has also been noted previously by\nCh\u0013 eron et al. [6].\nNoise in the person detections are not a problem for the training of our\nfully-supervised baseline, as it is trained with ground-truth boxes in addition\nto predicted boxes. As we have box-level supervision in this case, predicted\ndetections which have an IoU of more than 0.5 with a ground-truth detection\nare assigned the label of the ground-truth box, or the negative label otherwise,\nduring fully-supervised training.\nAs the goal of this paper is not to develop a better human detector or tracker\nfor building the person tubelets, we use the Faster-RCNN detector released"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc744-6db4-11ee-bfff-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-24", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "As the goal of this paper is not to develop a better human detector or tracker\nfor building the person tubelets, we use the Faster-RCNN detector released\npublicly by Ch\u0013 eron et al. [6] for all our evaluations on the UCF101-24 validation\nset. This detector was originally trained on COCO and then \fnetuned on the\nUCF101-24 training set using Detectron [13].\nThe e\u000bect of tubelet sampling For the tubelets of length K= 16 that we use,\nthere is a mean of 33.1 tubelets per video in the UCF101-24 dataset. In computing\nthis, we only consider tubelets which have a spatio-temporal IoU of less than 0.5\nwith each other. More tubelets would be obtained if we counted one from each\nframe of the video.\nAs we can \ft a maximum of 16 tubelets onto a 16GB Nvidia V100 GPU, it\nis clear that it is necessary to sample the tubelets in each bag. Note that UCF\nvideos often have a high number of tubelets, as there are often many people\nin the video who are not labelled as performing an action. As described in the\nprevious subsection, this is also a signi\fcant source of noise."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc745-6db4-11ee-a2f8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-25", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "in the video who are not labelled as performing an action. As described in the\nprevious subsection, this is also a signi\fcant source of noise.\nTable 2 shows the e\u000bect of changing the batch size (number of bags), and the\nnumber of tubelets sampled per bag, such that GPU memory usage is maximised.\nWe can see that the uncertainty loss helps in all cases and that accuracy decreases\nwith low batch sizes. We believe this is due to batch normalisation statistics\nbeing too correlated when more tubes are from the same video.\nComparison to state-of-the-art Table 3 compares our results to the state-of-the-art.\nThe bottom-half of the table shows that we outperform previous weakly-supervised\nUncertainty-Aware Weakly Supervised Action Detection 11\nTable 2. The e\u000bect of the number of bags in each training batch on accuracy (Video\nAP at 0.5). The uncertainty loss improves accuracy in all scenarios. Although fewer, but\nlarger, bags can reduce the noise due to sampling, they also cause batch normalisation\nstatistics to be too correlated, reducing accuracy.\nNumber of bags in batch Tubelets sampled per bag Video AP without uncertainty Video AP with uncertainty\n4 4 33.5 35.0\n3 5 33.6 34.1\n2 8 33.3 34.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc746-6db4-11ee-b7b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-26", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "4 4 33.5 35.0\n3 5 33.6 34.1\n2 8 33.3 34.2\n1 16 25.8 26.2\nTable 3. Comparison to state-of-the-art methods on the UCF101-24 dataset in both\nfully- and weakly-supervised scenarios.\nVideo AP at 0.2 Video AP at 0.5\nFully supervised\nPeng et al. [35] 42.3 35.9\nHou et al. [17] 47.1 {\nWeinzaepfel et al. [50] 58.9 {\nSaha et al. [38] 63.1 33.1\nSingh et al. [41] 73.5 46.3\nZhao et al. [52] 78.5 50.3\nSingh et al. [40] 79.0 50.9\nKalogeiton et al. [19] 77.2 51.4\nOurs 69.3 43.6\nWeakly supervised\nEscorcia et al. [8] 45.5 {\nCh\u0013 eron et al. [6] 43.9 17.7\nOurs 61.7 35.0\nmethods by a large margin. The top-half shows that our fully-supervised baseline"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc747-6db4-11ee-b004-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-27", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Ours 61.7 35.0\nmethods by a large margin. The top-half shows that our fully-supervised baseline\nis also competitive with the fully-supervised state-of-the-art, although that is not\nthe main goal of this work. The fully-supervised methods which outperform our\nmethod are based on action detectors which directly predict the person proposals\nwith the network, and are thus able to handle the person annotation peculiarities\nof the UCF101-24 dataset more e\u000bectively. We do not observe any issues with\nperson detections for our experiments on AVA in the next section.\nQualitative Results Figure 4 presents qualitative results of our method. The\n\frst two rows show success cases of our method where the tubelet detection\nand linking have performed well. The third row shows a failure case, since the\nbasketball player represented by the green track is not actually performing the\n\\Basketball Dunk\" action. According to the UCF101-24 annotations, only the\nplayer represented with the blue track is performing this action. This video clip\nis thus an example of a video where there are many people not performing the\naction annotated for the video, and is especially challenging for our weaklysupervised method. The fourth row shows a di\u000berent failure case as an error by\n12 A. Arnab et al."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc748-6db4-11ee-b805-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-28", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "12 A. Arnab et al.\nFig. 4. Qualitative examples on UCF101-24. Note that the bounding boxes are coloured\naccording to the identity of the track. The action label, and tube score are labelled\nfrom the top-left of the bounding box. Further discussion is included in the text.\nthe online tubelet linking algorithm (we used the same method as [ 19]) has made\nthe identities of the two cyclists change after they occluded each other.\n4.3 Experiments on AVA\nIn this section, we report what to our knowledge are the \frst weakly-supervised\naction detection experiments on AVA [ 14]. The AVA dataset labels keyframes\nin a 15-minute video clip, where each keyframe is sampled every second ( i.e.at\n1 Hz). The evaluation protocol of the AVA dataset measures the ability of an\naction detection model to classify the actions occuring in a keyframe given the\ntemporal context around it.\nWe control the di\u000eculty of the weakly-supervised action recognition problem by combining the annotations from Nconsecutive keyframes into a single,\nclip-level annotation. This e\u000bectively means that we are obtaining clip-level"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc749-6db4-11ee-949f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-29", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "clip-level annotation. This e\u000bectively means that we are obtaining clip-level\nannotations for sub-clips of Nseconds from the original AVA video. The weaklysupervised problem gets more di\u000ecult as Nincreases, as the sub-clips get longer\nand the number of observed labels within each sub-clip increases. Note that when\nN= 1, only the spatial localisation ability of the model is being tested, as during\ntraining, it is unknown which of the subclip-level labels correspond to each person\ntubelet in the MIL bag. When N > 1, the subclip-level labels can correspond\nUncertainty-Aware Weakly Supervised Action Detection 13\nTable 4. Results of our method on the AVA dataset in terms of the Frame mAP at\nan IoU threshold of 0.5. We vary the length of the sub-clips from which we extract\nclip-level annotations to control the di\u000eculty of the weakly supervised problems. FS\ndenotes a fully-supervised baseline representing the upper bound on performance. A\nsub-clip of 900 seconds is an entire AVA video clip.\nSub-clip duration (seconds)\nFS 1 5 10 30 60 900\nFrame AP 24.9 22.4 18.0 15.8 11.4 9.1 4.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc74a-6db4-11ee-a2e1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-30", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Sub-clip duration (seconds)\nFS 1 5 10 30 60 900\nFrame AP 24.9 22.4 18.0 15.8 11.4 9.1 4.2\nTable 5. State-of-the-art fully-supervised methods on the AVA dataset.\nMethod Frame AP\nAVA (with optical \row) [14] 15.6\nARCN (with optical \row) [46] 17.4\nAction Transformer [11] 25.0\nSlowFast (ResNet 101) [9] 26.8\nSlowFast (ResNet 50, Ours) 24.9\nto zero, one or many of the person tubelets at di\u000berent keyframes in the clip,\nand it is thus a more di\u000ecult task. As an AVA video clip consists of 900 seconds,\nN= 900 represents the most extreme case when spatio-temporal annotations are\ndiscarded for the entire 15-minute video.\nTable 4 shows the results of our model in this setting. As expected, the\nperformance of our method improves the shorter the sub-clip. For N= 1 and\nN= 5, our method obtains 90% and 72% of fully-supervised performance\nrespectively, suggesting that bounding-box level annotations are not required for\ntraining action recognition models if the video clips are annotated over short"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc74b-6db4-11ee-8e7d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-31", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "respectively, suggesting that bounding-box level annotations are not required for\ntraining action recognition models if the video clips are annotated over short\ntemporal intervals. Understandably, the results from N= 900, where we use the\nvideo-level annotations over the whole 15-minute clip are the worst as it is the\nmost di\u000ecult setting.\nFigure 5 further analyses the per-class results for the di\u000berent levels of\nsupervision presented in Tab. 4. As expected, stronger levels of supervision\n(shorter sub-clip durations) result in better per-class accuracy. However, some\naction classes are a\u000bected more than others by weaker labels (longer sub-clips).\nExamples of this include \\sing to\" and \\listen to\" which show a larger di\u000berence\nto the fully-supervised baseline than other classes. Moreover, some classes such\nas \\watch (a person)\", \\get up\", \\close (e.g., a door, a box)\" and \\hand clap\"\nperform reasonably when trained with sub-clips ( N\u001410), but much more poorly\nwhen trained with longer sub-clips.\nFinally, we compare our fully-supervised baseline to the state-of-the-art in\nTab. 5. Note that our weakly-supervised result from sub-clips of 10 seconds"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc74c-6db4-11ee-8e3a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-32", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Tab. 5. Note that our weakly-supervised result from sub-clips of 10 seconds\n(Tab. 4) outperforms the original fully-supervised baseline using introduced by\nthe AVA dataset [ 14] that uses both RGB and optical \row as inputs. Our model,\non the other hand, only uses RGB as its input modality. Our SlowFast model\n14 A. Arnab et al.\nFig. 5. Per-class results, in terms of the Frame AP, on the AVA dataset under di\u000berent\nlevels of supervision (the longer the sub-clip duration, the weaker the supervision). For\nclarity, the \frst 30 classes, ranked using the fully-supervised model, are shown. As\nexpected, action classes bene\ft from stronger supervision, while some classes, such as\n\\watch\", \\get up\" and \\close\" are very di\u000ecult to learn from long sub-clips.\nperforms similarly to the published results of the original authors [ 9]. Note that\nwe have not used Non-local [ 49], test-time augmentation or ensembling which\nare all complementary methods to improve performance [ 9]. We can see that\nin contrast to the UCF dataset in the previous section, our person detector is\naccurate on AVA, and so a Fast-RCNN-style detector using person tubelets as"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc74d-6db4-11ee-ad05-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-33", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "in contrast to the UCF dataset in the previous section, our person detector is\naccurate on AVA, and so a Fast-RCNN-style detector using person tubelets as\nproposals can achieve state-of-the-art results.\n5 Conclusion and Future Work\nWe have proposed a weakly supervised spatio-temporal action detection method\nbased on Multiple Instance Learning (MIL). Our approach incorporates uncertainty predictions made by the network such that it can better handle noise\nin our bags and violations of the standard MIL assumption by predicting a\nhigh uncertainty for noisy bags which cannot be classi\fed correctly. We achieve\nstate-of-the-art results among weakly supervised methods on the UCF101-24\ndataset, and also report the \frst weakly-supervised results on AVA, which is the\nonly large-scale action recognition dataset. Our analysis of the accuracy trade-o\u000bs\nas the time intervals for which sub-clips of the video are annotated will also aid\nfuture dataset annotation e\u000borts.\nFuture work is to incorporate additional sources of noisy, weakly-labelled\ndata, such as data which can be scraped o\u000b internet search engines.\nUncertainty-Aware Weakly Supervised Action Detection 15\nReferences"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc74e-6db4-11ee-810e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-34", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "data, such as data which can be scraped o\u000b internet search engines.\nUncertainty-Aware Weakly Supervised Action Detection 15\nReferences\n1.Arandjelovic, R., Zisserman, A.: Look, listen and learn. In: Proceedings of the IEEE\nInternational Conference on Computer Vision. pp. 609{617 (2017)\n2.Bach, F.R., Harchaoui, Z.: Di\u000brac: a discriminative and \rexible framework for\nclustering. In: Advances in Neural Information Processing Systems. pp. 49{56\n(2008)\n3.Boyd, S., Boyd, S.P., Vandenberghe, L.: Convex optimization. Cambridge university\npress (2004)\n4.Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A\nlarge-scale video benchmark for human activity understanding. In: Proceedings of\nthe ieee conference on computer vision and pattern recognition. pp. 961{970 (2015)\n5.Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the\nkinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and\nPattern Recognition. pp. 6299{6308 (2017)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc74f-6db4-11ee-a020-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-35", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and\nPattern Recognition. pp. 6299{6308 (2017)\n6.Ch\u0013 eron, G., Alayrac, J.B., Laptev, I., Schmid, C.: A \rexible model for training action\nlocalization with varying levels of supervision. In: Advances in Neural Information\nProcessing Systems. pp. 942{953 (2018)\n7.Dietterich, T.G., Lathrop, R.H., Lozano-P\u0013 erez, T.: Solving the multiple instance\nproblem with axis-parallel rectangles. Arti\fcial intelligence 89(1-2), 31{71 (1997)\n8.Escorcia, V., Dao, C.D., Jain, M., Ghanem, B., Snoek, C.: Guess where?\nactor-supervision for spatiotemporal action localization. In: arXiv preprint\narXiv:1804.01824 (2018)\n9.Feichtenhofer, C., Fan, H., Malik, J., He, K.: Slowfast networks for video recognition.\nIn: Proceedings of the IEEE International Conference on Computer Vision. pp.\n6202{6211 (2019)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc750-6db4-11ee-8ea2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-36", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "In: Proceedings of the IEEE International Conference on Computer Vision. pp.\n6202{6211 (2019)\n10.Ghadiyaram, D., Tran, D., Mahajan, D.: Large-scale weakly-supervised pre-training\nfor video action recognition. In: Proceedings of the IEEE Conference on Computer\nVision and Pattern Recognition. pp. 12046{12055 (2019)\n11.Girdhar, R., Carreira, J., Doersch, C., Zisserman, A.: Video action transformer\nnetwork. In: Proceedings of the IEEE Conference on Computer Vision and Pattern\nRecognition. pp. 244{253 (2019)\n12. Girshick, R.: Fast r-cnn. In: ICCV (2015)\n13.Girshick, R., Radosavovic, I., Gkioxari, G., Doll\u0013 ar, P., He, K.: Detectron. https:\n//github.com/facebookresearch/detectron (2018)\n14.Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc751-6db4-11ee-9216-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-37", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "14.Gu, C., Sun, C., Ross, D.A., Vondrick, C., Pantofaru, C., Li, Y., Vijayanarasimhan,\nS., Toderici, G., Ricco, S., Sukthankar, R., et al.: Ava: A video dataset of spatiotemporally localized atomic visual actions. In: Proceedings of the IEEE Conference\non Computer Vision and Pattern Recognition. pp. 6047{6056 (2018)\n15. He, K., Gkioxari, G., Doll\u0013 ar, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)\n16.He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.\nIn: CVPR (2016)\n17.Hou, R., Chen, C., Shah, M.: Tube convolutional neural network (t-cnn) for action\ndetection in videos. In: International Conference on Computer Vision. pp. 5822{5831\n(2017)\n18.Jiang, Y.G., Liu, J., Zamir, A.R., Toderici, G., Laptev, I., Shah, M., Sukthankar,\nR.: Thumos challenge: Action recognition with a large number of classes (2014)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc752-6db4-11ee-92ce-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-38", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "R.: Thumos challenge: Action recognition with a large number of classes (2014)\n19.Kalogeiton, V., Weinzaepfel, P., Ferrari, V., Schmid, C.: Joint learning of object\nand action detectors. In: Proceedings of the IEEE International Conference on\nComputer Vision. pp. 4163{4172 (2017)\n16 A. Arnab et al.\n20.Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S.,\nViola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video\ndataset. arXiv preprint arXiv:1705.06950 (2017)\n21.Ke, Y., Sukthankar, R., Hebert, M.: E\u000ecient visual event detection using volumetric\nfeatures. In: International Conference on Computer Vision. vol. 1, pp. 166{173.\nIEEE (2005)\n22.Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning\nfor computer vision? In: Advances in neural information processing systems. pp.\n5574{5584 (2017)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc753-6db4-11ee-87e7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-39", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "for computer vision? In: Advances in neural information processing systems. pp.\n5574{5584 (2017)\n23.Kendall, A., Gal, Y., Cipolla, R.: Multi-task learning using uncertainty to weigh\nlosses for scene geometry and semantics. In: Computer Vision and Pattern Recognition. pp. 7482{7491 (2018)\n24.Kraus, O.Z., Ba, L.J., Frey, B.: Classifying and segmenting microscopy images\nusing convolutional multiple instance learning. In: arXiv preprint arXiv:1511.05286\n(2015)\n25.Laptev, I., P\u0013 erez, P.: Retrieving actions in movies. In: International Conference on\nComputer Vision. pp. 1{8. IEEE (2007)\n26.Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll\u0013 ar, P.,\nZitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV (2014)\n27.Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc754-6db4-11ee-a4ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-40", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Single shot multibox detector. In: European conference on computer vision. pp.\n21{37. Springer (2016)\n28.Mettes, P., Snoek, C.G., Chang, S.F.: Localizing actions from video labels and\npseudo-annotations. In: British Machine Vision Conference (BMVC) (2017)\n29.Mettes, P., Van Gemert, J.C., Snoek, C.G.: Spot on: Action localization from\npointly-supervised proposals. In: European conference on computer vision. pp.\n437{453. Springer (2016)\n30.Monfort, M., Andonian, A., Zhou, B., Ramakrishnan, K., Bargal, S.A., Yan, T.,\nBrown, L., Fan, Q., Gutfruend, D., Vondrick, C., et al.: Moments in time dataset:\none million videos for event understanding. arXiv preprint arXiv:1801.03150 (2018)\n31.Nguyen, P., Liu, T., Prasad, G., Han, B.: Weakly supervised action localization\nby sparse temporal pooling network. In: Proceedings of the IEEE Conference on\nComputer Vision and Pattern Recognition. pp. 6752{6761 (2018)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc755-6db4-11ee-b0bd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-41", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "by sparse temporal pooling network. In: Proceedings of the IEEE Conference on\nComputer Vision and Pattern Recognition. pp. 6752{6761 (2018)\n32.Novotny, D., Albanie, S., Larlus, D., Vedaldi, A.: Self-supervised learning of geometrically stable features through probabilistic introspection. In: Computer Vision\nand Pattern Recognition. pp. 3637{3645 (2018)\n33.Owens, A., Wu, J., McDermott, J.H., Freeman, W.T., Torralba, A.: Ambient sound\nprovides supervision for visual learning. In: European conference on computer vision.\npp. 801{816. Springer (2016)\n34.Paul, S., Roy, S., Roy-Chowdhury, A.K.: W-talc: Weakly-supervised temporal\nactivity localization and classi\fcation. In: Proceedings of the European Conference\non Computer Vision (ECCV). pp. 563{579 (2018)\n35.Peng, X., Schmid, C.: Multi-region two-stream r-cnn for action detection. In:\nEuropean conference on computer vision. pp. 744{759. Springer (2016)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc756-6db4-11ee-8527-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-42", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "European conference on computer vision. pp. 744{759. Springer (2016)\n36.Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: Computer Vision and Pattern Recognition. pp. 1713{1721\n(2015)\n37.Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing\nsystems. pp. 91{99 (2015)\nUncertainty-Aware Weakly Supervised Action Detection 17\n38.Saha, S., Singh, G., Sapienza, M., Torr, P.H., Cuzzolin, F.: Deep learning for\ndetecting multiple space-time action tubes in videos. In: BMVC (2016)\n39.Sigurdsson, G.A., Russakovsky, O., Gupta, A.: What actions are needed for understanding human actions in videos? In: Proceedings of the IEEE International\nConference on Computer Vision. pp. 2137{2146 (2017)\n40.Singh, G., Saha, S., Cuzzolin, F.: Tramnet-transition matrix network for e\u000ecient"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc757-6db4-11ee-b514-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-43", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "40.Singh, G., Saha, S., Cuzzolin, F.: Tramnet-transition matrix network for e\u000ecient\naction tube proposals. In: Asian Conference on Computer Vision. pp. 420{437.\nSpringer (2018)\n41.Singh, G., Saha, S., Sapienza, M., Torr, P.H., Cuzzolin, F.: Online real-time multiple\nspatiotemporal action localisation and prediction. In: International Conference on\nComputer Vision. pp. 3637{3646 (2017)\n42.Singh, K.K., Lee, Y.J.: Hide-and-seek: Forcing a network to be meticulous for\nweakly-supervised object and action localization. In: 2017 IEEE international\nconference on computer vision (ICCV). pp. 3544{3553. IEEE (2017)\n43.Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes\nfrom videos in the wild. arXiv preprint arXiv:1212.0402 (2012)\n44.Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model\nfor video and language representation learning. In: International Conference on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc758-6db4-11ee-b71e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-44", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "for video and language representation learning. In: International Conference on\nComputer Vision. pp. 7464{7473 (2019)\n45.Sun, C., Paluri, M., Collobert, R., Nevatia, R., Bourdev, L.: Pronet: Learning to\npropose object-speci\fc boxes for cascaded neural networks. In: The IEEE Conference\non Computer Vision and Pattern Recognition (CVPR) (June 2016)\n46.Sun, C., Shrivastava, A., Vondrick, C., Murphy, K., Sukthankar, R., Schmid, C.:\nActor-centric relation network. In: Proceedings of the European Conference on\nComputer Vision (ECCV). pp. 318{334 (2018)\n47.Van Gemert, J.C., Jain, M., Gati, E., Snoek, C.G., et al.: Apt: Action localization\nproposals from dense trajectories. In: BMVC. vol. 2, p. 4 (2015)\n48.Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: Proceedings of the IEEE conference on\nComputer Vision and Pattern Recognition. pp. 4325{4334 (2017)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc759-6db4-11ee-8ff2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2007.10703-45", "title": "Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos", "source": "http://arxiv.org/pdf/2007.10703", "start_index": 0, "chroma:document": "Computer Vision and Pattern Recognition. pp. 4325{4334 (2017)\n49.Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp.\n7794{7803 (2018)\n50.Weinzaepfel, P., Martin, X., Schmid, C.: Towards weakly-supervised action localization. arXiv preprint arXiv:1605.05197 2(2016)\n51.Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J., Torralba, A.:\nThe sound of pixels. In: Proceedings of the European Conference on Computer\nVision (ECCV). pp. 570{586 (2018)\n52.Zhao, J., Snoek, C.G.: Dance with \row: Two-in-one stream action detection. In:\nProceedings of the IEEE Conference on Computer Vision and Pattern Recognition.\npp. 9935{9944 (2019)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc75a-6db4-11ee-98ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-0", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Is Space-Time Attention All You Need for Video Understanding?\nGedas Bertasius1Heng Wang1Lorenzo Torresani1 2\nAbstract\nWe present a convolution-free approach to video\nclassi\ufb01cation built exclusively on self-attention\nover space and time. Our method, named \u201cTimeSformer,\u201d adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of framelevel patches. Our experimental study compares different self-attention schemes and suggests that \u201cdivided attention,\u201d where temporal\nattention and spatial attention are separately applied within each block, leads to the best video\nclassi\ufb01cation accuracy among the design choices\nconsidered. Despite the radically new design,\nTimeSformer achieves state-of-the-art results on\nseveral action recognition benchmarks, including the best reported accuracy on Kinetics-400\nand Kinetics-600. Finally, compared to 3D\nconvolutional networks, our model is faster to\ntrain, it can achieve dramatically higher test\nef\ufb01ciency (at a small drop in accuracy), and\nit can also be applied to much longer video\nclips (over one minute long). Code and models are available at: https://github.com/\nfacebookresearch/TimeSformer .\n1. Introduction"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc75b-6db4-11ee-8077-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-1", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "clips (over one minute long). Code and models are available at: https://github.com/\nfacebookresearch/TimeSformer .\n1. Introduction\nOver the last few years, the \ufb01eld of natural language processing (NLP) has been revolutionized by the emergence of\nmethods based on self-attention (Vaswani et al., 2017a). Because of their excellent capabilities at capturing long-range\ndependencies among words as well as their training scalability, self-attention architectures, such as the Transformer\nmodel, represent the current state-of-the-art across a wide\nrange of language tasks, including machine translation (Ott\net al., 2018; Chen et al., 2018a), question answering (Devlin et al., 2019; Dai et al., 2019), and autoregressive word\ngeneration (Radford et al., 2019; Brown et al., 2020).\nVideo understanding shares several high-level similarities\n1Facebook AI2Dartmouth College. Correspondence to: Gedas\nBertasius <gberta@seas.upenn.edu>.with NLP. First of all, videos and sentences are both sequential. Furthermore, precisely as the meaning of a word can\noften be understood only by relating it to the other words in\nthe sentence, it may be argued that atomic actions in shortterm segments need to be contextualized with the rest of the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc75c-6db4-11ee-a8ed-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-2", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "often be understood only by relating it to the other words in\nthe sentence, it may be argued that atomic actions in shortterm segments need to be contextualized with the rest of the\nvideo in order to be fully disambiguated. Thus, one would\nexpect the long-range self-attention models from NLP to be\nhighly effective for video modeling as well. However, in the\nvideo domain, 2D or 3D convolutions still represent the core\noperators for spatiotemporal feature learning across different video tasks (Feichtenhofer et al., 2019a; Teed & Deng,\n2020; Bertasius & Torresani, 2020). While self-attention\nhas shown bene\ufb01ts when applied on top of convolutional\nlayers (Wang et al., 2018a), to the best of our knowledge, no\nattempt to use self-attention as the exclusive building block\nfor video recognition models has been reported.\nIn this work we pose the question of whether it may be\npossible to build a performant convolution-free video architecture by replacing altogether the convolution operator with\nself-attention. We argue that such a design has the potential to overcome a few inherent limitations of convolutional\nmodels for video analysis. First, while their strong inductive\nbiases (e.g., local connectivity and translation equivariance)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc75d-6db4-11ee-8d10-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-3", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "models for video analysis. First, while their strong inductive\nbiases (e.g., local connectivity and translation equivariance)\nare undoubtedly bene\ufb01cial on small training sets, they may\nexcessively limit the expressivity of the model in settings\nwhere there is ample availability of data and \u201call\u201d can be\nlearned from examples. Compared to CNNs, Transformers\nimpose less restrictive inductive biases. This broadens the\nfamily of functions they can represent (Cordonnier et al.,\n2020; Zhao et al., 2020), and renders them better suited\nto modern big-data regimes where there is less need for\nstrong inductive priors. Second, while convolutional kernels\nare speci\ufb01cally designed to capture short-range spatiotemporal information, they cannot model dependencies that\nextend beyond the receptive \ufb01eld. While deep stacks of\nconvolutions (Simonyan & Zisserman, 2015; Szegedy et al.,\n2015; Carreira & Zisserman, 2017) naturally extend the\nreceptive \ufb01eld, these strategies are inherently limited in capturing long-range dependencies by means of aggregation\nof shorter-range information. Conversely, the self-attention\nmechanism can be applied to capture both local as well as"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc75e-6db4-11ee-b0ca-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-4", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "of shorter-range information. Conversely, the self-attention\nmechanism can be applied to capture both local as well as\nglobal long-range dependencies by directly comparing feature activations at all space-time locations, much beyond the\nreceptive \ufb01eld of traditional convolutional \ufb01lters. Finally,arXiv:2102.05095v4  [cs.CV]  9 Jun 2021\nIs Space-Time Attention All You Need for Video Understanding?\ndespite the advances in GPU hardware acceleration, training\ndeep CNNs remains very costly, especially when applied to\nhigh-resolution and long videos. Recent work in the stillimage domain (Dosovitskiy et al., 2020; Carion et al., 2020;\nZhao et al., 2020) has demonstrated that Transformers enjoy\nfaster training and inference compared to CNNs, making it\npossible to construct models with larger learning capacity\nfor the same computational budget.\nMotivated by these observations, we propose a video architecture built exclusively on self-attention. We adapt the\nimage model \u201cVision Transformer\u201d (ViT) (Dosovitskiy et al.,\n2020) to video by extending the self-attention mechanism\nfrom the image space to the space-time 3D volume. Our\nproposed model, named \u201cTimeSformer\u201d (from Time-Space"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc75f-6db4-11ee-a77b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-5", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "from the image space to the space-time 3D volume. Our\nproposed model, named \u201cTimeSformer\u201d (from Time-Space\nTransformer), views the video as a sequence of patches extracted from the individual frames. As in ViT, each patch is\nlinearly mapped into an embedding and augmented with positional information. This makes it possible to interpret the\nresulting sequence of vectors as token embeddings which\ncan be fed to a Transformer encoder, analogously to the\ntoken features computed from words in NLP.\nOne downside of self-attention in standard Transformer is\nthat it requires computing a similarity measure for all pairs\nof tokens. In our setting, this is computationally costly due\nto the large number of patches in the video. To address these\nchallenges, we propose several scalable self-attention designs over the space-time volume and empirically evaluate\nthem over large-scale action classi\ufb01cation datasets. Among\nthe proposed schemes, we found that the best design is represented by a \u201cdivided attention\u201d architecture which separately\napplies temporal attention and spatial attention within each\nblock of the network. Compared to the established paradigm\nof convolution-based video architecture, TimeSformer follows a radically different design. Yet, it achieves accuracy"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc760-6db4-11ee-b625-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-6", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "block of the network. Compared to the established paradigm\nof convolution-based video architecture, TimeSformer follows a radically different design. Yet, it achieves accuracy\ncomparable, and in some cases superior, to the state-of-theart in this \ufb01eld. We also show that our model can be used\nfor long-range modeling of videos spanning many minutes.\n2. Related Work\nOur approach is in\ufb02uenced by recent works that use selfattention for image classi\ufb01cation, either in combination with\nthe convolution operator or even as a full replacement for it.\nWithin the former class, Non-Local Networks (Wang et al.,\n2018b) employ a non-local mean that effectively generalizes\nthe self-attention function of Transformers (Vaswani et al.,\n2017b). Bello et al. (Bello et al., 2019) propose a 2D selfattention mechanism that is competitive as a replacement of\n2D convolution but gives even stronger results when used to\naugment convolutional features with self-attention features.\nBeyond image categorization, Relation Networks (Hu et al.,\n2018) and DETR (Carion et al., 2020) use self-attention on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc761-6db4-11ee-b3d3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-7", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Beyond image categorization, Relation Networks (Hu et al.,\n2018) and DETR (Carion et al., 2020) use self-attention on\ntop of convolutional feature maps for object detection.Our method is more closely related to image networks leveraging self-attention as a substitute for convolution (Parmar\net al., 2018; Ramachandran et al., 2019; Cordonnier et al.,\n2020; Zhao et al., 2020). Since these works use individual\npixels as queries, in order to maintain a manageable computational cost and a small memory consumption, they must\nrestrict the scope of self-attention to local neighborhoods or\nuse global self-attention on heavily downsized versions of\nthe image. Alternative strategies for scalability to full images include sparse key-value sampling (Child et al., 2019)\nor constraining the self-attention to be calculated along the\nspatial axes (Ho et al., 2019; Huang et al., 2019; Wang\net al., 2020b). A few of the self-attention operators considered in our experiments adopt similar sparse and axial\ncomputation, although generalized to the spatiotemporal\nvolume. However, the ef\ufb01ciency of our approach stems\nmainly from decomposing the video into a sequence of\nframe-level patches and then feeding linear embeddings of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc762-6db4-11ee-9ac0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-8", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "mainly from decomposing the video into a sequence of\nframe-level patches and then feeding linear embeddings of\nthese patches as input token embeddings to a Transformer.\nThis strategy was recently introduced in Vision Transformers (ViT) (Dosovitskiy et al., 2020) which were shown to\ndeliver impressive performance on image categorization. In\nthis work, we build on the ViT design, and extend it to video\nby proposing and empirically comparing several scalable\nschemes for space-time self-attention over videos.\nWhile Transformers have been recently used for video generation (Weissenborn et al., 2020), we are not aware of prior\nvideo recognition architectures using self-attention as the\nexclusive building block. However, we note that Transformers have been adopted on top of convolutional feature\nmaps for action localization and recognition (Girdhar et al.,\n2019), video classi\ufb01cation (Wang et al., 2018b; Chen et al.,\n2018b), and group activity recognition (Gavrilyuk et al.,\n2020). We also note that there is a wide literature based on\nthe use of text Transformers combined with video CNNs\nto address various video-language tasks, such as captioning (Zhou et al., 2018), question-answering (Yang et al.,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc763-6db4-11ee-a69c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-9", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "to address various video-language tasks, such as captioning (Zhou et al., 2018), question-answering (Yang et al.,\n2020) and dialog (Le et al., 2019). Finally, multimodal\nvideo-text transformers (Sun et al., 2019; Li et al., 2020a)\nhave also been trained or pretrained in unsupervised fashion\nby adopting masked-token pretext tasks adapted from the\nlanguage domain (Devlin et al., 2018; Radford et al., 2018).\n3. The TimeSformer Model\nInput clip. The TimeSformer takes as input a clip X2\nRH\u0002W\u00023\u0002Fconsisting of FRGB frames of size H\u0002W\nsampled from the original video.\nDecomposition into patches. Following the ViT (Dosovitskiy et al., 2020), we decompose each frame into N\nnon-overlapping patches, each of size P\u0002P, such that\ntheNpatches span the entire frame, i.e., N=HW=P2.\nWe \ufb02atten these patches into vectors x(p;t)2R3P2with\nIs Space-Time Attention All You Need for Video Understanding?"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc764-6db4-11ee-a8e1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-10", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc765-6db4-11ee-b0f8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc766-6db4-11ee-ab43-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc767-6db4-11ee-93c8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc768-6db4-11ee-ad2e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc769-6db4-11ee-ab1d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc76a-6db4-11ee-930e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc76b-6db4-11ee-92dd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc76c-6db4-11ee-8030-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc76d-6db4-11ee-8ead-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-19", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Att.Space Att.\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc76e-6db4-11ee-b3ba-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc76f-6db4-11ee-992f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc770-6db4-11ee-8505-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc771-6db4-11ee-9934-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc772-6db4-11ee-9e91-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc773-6db4-11ee-b98d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc774-6db4-11ee-9954-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cc775-6db4-11ee-8468-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced85-6db4-11ee-b6b8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced86-6db4-11ee-bd8c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced87-6db4-11ee-99bd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced88-6db4-11ee-9157-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced89-6db4-11ee-95c5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced8a-6db4-11ee-89fa-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced8b-6db4-11ee-9fa8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced8c-6db4-11ee-9ee6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced8d-6db4-11ee-94ef-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced8e-6db4-11ee-bd31-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced8f-6db4-11ee-856f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced90-6db4-11ee-b752-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced91-6db4-11ee-a2b3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced92-6db4-11ee-b0ee-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced93-6db4-11ee-9b9b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced94-6db4-11ee-bbc3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced95-6db4-11ee-8087-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-44", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Att.MLP\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced96-6db4-11ee-95f1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced97-6db4-11ee-aa29-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced98-6db4-11ee-8370-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced99-6db4-11ee-a0fc-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced9a-6db4-11ee-b4b9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced9b-6db4-11ee-aafb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced9c-6db4-11ee-a7f3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced9d-6db4-11ee-a727-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced9e-6db4-11ee-b692-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ced9f-6db4-11ee-a933-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda0-6db4-11ee-a269-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda1-6db4-11ee-9e02-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda2-6db4-11ee-a39b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda3-6db4-11ee-9502-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda4-6db4-11ee-8f78-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda5-6db4-11ee-8778-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda6-6db4-11ee-b893-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-61", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Space-Time Att.\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda7-6db4-11ee-96fc-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda8-6db4-11ee-a1d8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceda9-6db4-11ee-b92c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedaa-6db4-11ee-a8c2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedab-6db4-11ee-8d51-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedac-6db4-11ee-b222-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedad-6db4-11ee-a23d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedae-6db4-11ee-9aeb-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedaf-6db4-11ee-a2ab-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb0-6db4-11ee-bb94-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb1-6db4-11ee-bd3c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb2-6db4-11ee-802a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb3-6db4-11ee-8ee6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb4-6db4-11ee-817f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb5-6db4-11ee-aca8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb6-6db4-11ee-a0e1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb7-6db4-11ee-89c1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb8-6db4-11ee-88c1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedb9-6db4-11ee-9301-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedba-6db4-11ee-8034-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedbb-6db4-11ee-b89a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedbc-6db4-11ee-b1fd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedbd-6db4-11ee-b4cc-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedbe-6db4-11ee-ba54-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedbf-6db4-11ee-b8c3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc0-6db4-11ee-9373-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc1-6db4-11ee-b281-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc2-6db4-11ee-9f22-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc3-6db4-11ee-94b1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc4-6db4-11ee-89a0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc5-6db4-11ee-be47-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc6-6db4-11ee-bded-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc7-6db4-11ee-851e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-94", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Space Attention (S)  Joint Space-Time      Attention (ST)Divided Space-Time     Attention (T+S)\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc8-6db4-11ee-8bff-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedc9-6db4-11ee-b669-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedca-6db4-11ee-8b56-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedcb-6db4-11ee-a1a1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedcc-6db4-11ee-9dc6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedcd-6db4-11ee-ac69-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedce-6db4-11ee-af3a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedcf-6db4-11ee-98c8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd0-6db4-11ee-91d3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-103", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Att.Width Att.\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd1-6db4-11ee-b08f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd2-6db4-11ee-87d0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd3-6db4-11ee-8d95-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd4-6db4-11ee-9403-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd5-6db4-11ee-996e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd6-6db4-11ee-ae95-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd7-6db4-11ee-a193-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd8-6db4-11ee-a840-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedd9-6db4-11ee-b5c1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-112", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "MLP\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedda-6db4-11ee-aee0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceddb-6db4-11ee-ad5a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceddc-6db4-11ee-90b2-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceddd-6db4-11ee-8681-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedde-6db4-11ee-b0d5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceddf-6db4-11ee-b976-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede0-6db4-11ee-b94f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede1-6db4-11ee-9267-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede2-6db4-11ee-8c3d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-121", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Att.\u0000<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede3-6db4-11ee-a122-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede4-6db4-11ee-9286-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede5-6db4-11ee-ae02-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede6-6db4-11ee-b5c4-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede7-6db4-11ee-841d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede8-6db4-11ee-9c08-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cede9-6db4-11ee-af44-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedea-6db4-11ee-a1b3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedeb-6db4-11ee-bf70-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-130", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Axial Attention      (T+W+H)Sparse Local Global     Attention (L+G)MLPLocal Att.Global Att.MLPz(`)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedec-6db4-11ee-b28d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-131", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1ceded-6db4-11ee-91bd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedee-6db4-11ee-b18f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedef-6db4-11ee-ad51-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf0-6db4-11ee-b2b8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf1-6db4-11ee-9cee-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf2-6db4-11ee-abe9-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf3-6db4-11ee-b88e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf4-6db4-11ee-b02c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf5-6db4-11ee-904b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-140", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf6-6db4-11ee-a88a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf7-6db4-11ee-848c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf8-6db4-11ee-b836-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedf9-6db4-11ee-99ab-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedfa-6db4-11ee-9797-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedfb-6db4-11ee-994e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedfc-6db4-11ee-8306-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedfd-6db4-11ee-8545-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedfe-6db4-11ee-a43f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-149", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "z(`)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cedff-6db4-11ee-841b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-150", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee00-6db4-11ee-8347-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee01-6db4-11ee-bf28-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee02-6db4-11ee-9e91-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee03-6db4-11ee-b1ca-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee04-6db4-11ee-ab62-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee05-6db4-11ee-bb14-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee06-6db4-11ee-b27e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee07-6db4-11ee-841c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee08-6db4-11ee-85f1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-159", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee09-6db4-11ee-9d6e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee0a-6db4-11ee-ba23-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee0b-6db4-11ee-8c05-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee0c-6db4-11ee-92c6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee0d-6db4-11ee-8e69-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee0e-6db4-11ee-ba14-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee0f-6db4-11ee-887c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee10-6db4-11ee-9e16-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee11-6db4-11ee-b0b6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-168", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee12-6db4-11ee-9117-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee13-6db4-11ee-bb2c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee14-6db4-11ee-a3b0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee15-6db4-11ee-ad90-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee16-6db4-11ee-bb42-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee17-6db4-11ee-a0ac-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee18-6db4-11ee-b885-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee19-6db4-11ee-aab6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee1a-6db4-11ee-b5f2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-177", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee1b-6db4-11ee-8992-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee1c-6db4-11ee-bd77-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee1d-6db4-11ee-8467-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee1e-6db4-11ee-9023-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee1f-6db4-11ee-8880-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee20-6db4-11ee-8fd7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee21-6db4-11ee-9448-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee22-6db4-11ee-aab6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee23-6db4-11ee-98e6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-186", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee24-6db4-11ee-9708-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee25-6db4-11ee-bd87-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee26-6db4-11ee-bb7d-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee27-6db4-11ee-aa17-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee28-6db4-11ee-a2ba-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee29-6db4-11ee-a46e-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee2a-6db4-11ee-98f7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee2b-6db4-11ee-b9ec-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee2c-6db4-11ee-850f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-195", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee2d-6db4-11ee-b237-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee2e-6db4-11ee-b309-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee2f-6db4-11ee-8828-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee30-6db4-11ee-9799-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee31-6db4-11ee-8e08-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee32-6db4-11ee-8409-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee33-6db4-11ee-8a6a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee34-6db4-11ee-a8dd-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee35-6db4-11ee-b0e4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-204", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee36-6db4-11ee-8bc3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee37-6db4-11ee-a037-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee38-6db4-11ee-b3ee-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee39-6db4-11ee-a848-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee3a-6db4-11ee-8544-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee3b-6db4-11ee-8fb5-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee3c-6db4-11ee-9914-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee3d-6db4-11ee-a236-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee3e-6db4-11ee-b040-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-213", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee3f-6db4-11ee-b261-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee40-6db4-11ee-86c1-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee41-6db4-11ee-a2d3-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee42-6db4-11ee-b00a-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee43-6db4-11ee-af7b-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee44-6db4-11ee-9dbf-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee45-6db4-11ee-8512-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee46-6db4-11ee-a876-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee47-6db4-11ee-9df8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-222", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Figure 1. The video self-attention blocks that we investigate in this work. Each attention layer implements self-attention (Vaswani et al.,\n2017b) on a speci\ufb01ed spatiotemporal neighborhood of frame-level patches (see Figure 2 for a visualization of the neighborhoods). We use\nresidual connections to aggregate information from different attention layers within each block. A 1-hidden-layer MLP is applied at the\nend of each block. The \ufb01nal model is constructed by repeatedly stacking these blocks on top of each other.\np= 1;:::;N denoting spatial locations and t= 1;:::;F\ndepicting an index over frames.\nLinear embedding. We linearly map each patch x(p;t)into\nan embedding vector z(0)\n(p;t)2RDby means of a learnable\nmatrixE2RD\u00023P2:\nz(0)\n(p;t)=Ex(p;t)+epos\n(p;t)(1)\nwhere epos\n(p;t)2RDrepresents a learnable positional embedding added to encode the spatiotemporal position of each\npatch. The resulting sequence of embedding vectors z(0)\n(p;t)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee48-6db4-11ee-8a2b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-223", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "patch. The resulting sequence of embedding vectors z(0)\n(p;t)\nforp= 1;:::;N , andt= 1;:::;F represents the input to\nthe Transformer, and plays a role similar to the sequences of\nembedded words that are fed to text Transformers in NLP.\nAs in the original BERT Transformer (Devlin et al., 2018),\nwe add in the \ufb01rst position of the sequence a special learnable vector z(0)\n(0;0)2RDrepresenting the embedding of the\nclassi\ufb01cation token.\nQuery-Key-Value computation. Our Transformer consists\nofLencoding blocks. At each block `, a query/key/value\nvector is computed for each patch from the representation\nz(`\u00001)\n(p;t)encoded by the preceding block:\nq(`;a)\n(p;t)=W(`;a)\nQLN\u0010\nz(`\u00001)\n(p;t)\u0011\n2RDh(2)\nk(`;a)\n(p;t)=W(`;a)\nKLN\u0010\nz(`\u00001)\n(p;t)\u0011\n2RDh(3)\nv(`;a)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee49-6db4-11ee-941d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-224", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "KLN\u0010\nz(`\u00001)\n(p;t)\u0011\n2RDh(3)\nv(`;a)\n(p;t)=W(`;a)\nVLN\u0010\nz(`\u00001)\n(p;t)\u0011\n2RDh(4)where LN() denotes LayerNorm (Ba et al., 2016), a=\n1;:::;Ais an index over multiple attention heads and A\ndenotes the total number of attention heads. The latent\ndimensionality for each attention head is set to Dh=D=A.\nSelf-attention computation. Self-attention weights are\ncomputed via dot-product. The self-attention weights\n\u000b\u000b\u000b(`;a)\n(p;t)2RNF+1for query patch (p;t)are given by:\n\u000b\u000b\u000b(`;a)\n(p;t)= SM0\n@q(`;a)\n(p;t)pDh>\n\u0001\"\nk(`;a)\n(0;0)n\nk(`;a)\n(p0;t0)o\np0=1;:::;N\nt0=1;:::;F#1\nA\n(5)\nwhere SMdenotes the softmax activation function. Note"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee4a-6db4-11ee-928c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-225", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "p0=1;:::;N\nt0=1;:::;F#1\nA\n(5)\nwhere SMdenotes the softmax activation function. Note\nthat when attention is computed over one dimension only\n(e.g., spatial-only or temporal-only), the computation is\nsigni\ufb01cantly reduced. For example, in the case of spatial\nattention, only N+ 1 query-key comparisons are made,\nusing exclusively keys from the same frame as the query:\n\u000b\u000b\u000b(`;a)space\n(p;t)= SM0\n@q(`;a)\n(p;t)pDh>\n\u0001\u0014\nk(`;a)\n(0;0)n\nk(`;a)\n(p0;t)o\np0=1;:::;N\u00151\nA:\n(6)\nEncoding. The encoding z(`)\n(p;t)at block`is obtained by\n\ufb01rst computing the weighted sum of value vectors using\nself-attention coef\ufb01cients from each attention head:\nIs Space-Time Attention All You Need for Video Understanding?\nframe tframe t - \u0000"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee4b-6db4-11ee-9c4c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-226", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee4c-6db4-11ee-aaa8-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee4d-6db4-11ee-9c33-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee4e-6db4-11ee-997f-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee4f-6db4-11ee-a781-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee50-6db4-11ee-934c-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee51-6db4-11ee-80ab-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee52-6db4-11ee-bd04-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee53-6db4-11ee-bdb0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee54-6db4-11ee-948d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-235", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "t + \u0000"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee55-6db4-11ee-a8c2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-236", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "<latexit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee56-6db4-11ee-a518-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee57-6db4-11ee-84ab-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee58-6db4-11ee-a854-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee59-6db4-11ee-8736-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee5a-6db4-11ee-bfb0-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee5b-6db4-11ee-b9df-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee5c-6db4-11ee-9ad7-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee5d-6db4-11ee-a4c6-00155d3fea2c
encoding: FLOAT32
metadata:


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee5e-6db4-11ee-be8d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-245", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Attention (S)   Joint Space-Time       Attention (ST)Divided Space-Time      Attention (T+S)  Sparse Local Global        Attention (L+G)  Axial Attention        (T+W+H)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee5f-6db4-11ee-9c9c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-246", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Figure 2. Visualization of the \ufb01ve space-time self-attention schemes studied in this work. Each video clip is viewed as a sequence of\nframe-level patches with a size of 16\u000216pixels. For illustration, we denote in blue the query patch and show in non-blue colors its\nself-attention space-time neighborhood under each scheme. Patches without color are not used for the self-attention computation of the\nblue patch. Multiple colors within a scheme denote attentions separately applied along different dimensions (e.g., space and time for\n(T+S)) or over different neighborhoods (e.g., for (L+G)). Note that self-attention is computed for every single patch in the video clip, i.e.,\nevery patch serves as a query. We also note that although the attention pattern is shown for only two adjacent frames, it extends in the\nsame fashion to all frames of the clip.\ns(`;a)\n(p;t)=\u000b(`;a)\n(p;t);(0;0)v(`;a)\n(0;0)+NX\np0=1FX\nt0=1\u000b(`;a)\n(p;t);(p0;t0)v(`;a)\n(p0;t0):\n(7)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee60-6db4-11ee-aab7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-247", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "(p;t);(p0;t0)v(`;a)\n(p0;t0):\n(7)\nThen, the concatenation of these vectors from all heads\nis projected and passed through an MLP, using residual\nconnections after each operation:\nz0(`)\n(p;t)=WO2\n664s(`;1)\n(p;t)\n...\ns(`;A)\n(p;t)3\n775+z(`\u00001)\n(p;t)(8)\nz(`)\n(p;t)= MLP\u0010\nLN\u0010\nz0(`)\n(p;t)\u0011\u0011\n+z0(`)\n(p;t): (9)\nClassi\ufb01cation embedding. The \ufb01nal clip embedding is\nobtained from the \ufb01nal block for the classi\ufb01cation token:\ny= LN\u0010\nz(L)\n(0;0)\u0011\n2RD: (10)\nOn top of this representation we append a 1-hidden-layer\nMLP, which is used to predict the \ufb01nal video classes.\nSpace-Time Self-Attention Models. We can reduce the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee61-6db4-11ee-9131-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-248", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "MLP, which is used to predict the \ufb01nal video classes.\nSpace-Time Self-Attention Models. We can reduce the\ncomputational cost by replacing the spatiotemporal attention of Eq. 5 with spatial attention within each frame only\n(Eq. 6). However, such a model neglects to capture temporaldependencies across frames. As shown in our experiments,\nthis approach leads to degraded classi\ufb01cation accuracy compared to full spatiotemporal attention, especially on benchmarks where strong temporal modeling is necessary.\nWe propose a more ef\ufb01cient architecture for spatiotemporal\nattention, named \u201cDivided Space-Time Attention\u201d (denoted\nwith T+S), where temporal attention and spatial attention\nare separately applied one after the other. This architecture\nis compared to that of Space and Joint Space-Time attention\nin Fig. 1. A visualization of the different attention models\non a video example is given in Fig. 2. For Divided Attention,\nwithin each block `, we \ufb01rst compute temporal attention by\ncomparing each patch (p;t)with all the patches at the same\nspatial location in the other frames:\n\u000b\u000b\u000b(`;a)time\n(p;t)= SM0\n@q(`;a)\n(p;t)pDh>\n\u0001\u0014"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee62-6db4-11ee-847c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-249", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "(`;a)time\n(p;t)= SM0\n@q(`;a)\n(p;t)pDh>\n\u0001\u0014\nk(`;a)\n(0;0)n\nk(`;a)\n(p;t0)o\nt0=1;:::;F\u00151\nA:\n(11)\nThe encoding z0(`)time\n(p;t)resulting from the application of\nEq. 8 using temporal attention is then fed back for spatial\nattention computation instead of being passed to the MLP. In\nother words, new key/query/value vectors are obtained from\nz0(`)time\n(p;t)and spatial attention is then computed using Eq. 6.\nFinally, the resulting vector z0(`)space\n(p;t)is passed to the MLP\nof Eq. 9 to compute the \ufb01nal encoding z(`)\n(p;t)of the patch at\nblock`. For the model of divided attention, we learn distinct query/key/value matrices fW(`;a)\nQtime;W(`;a)\nKtime;W(`;a)\nVtimeg\nIs Space-Time Attention All You Need for Video Understanding?"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee63-6db4-11ee-8b86-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-250", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Qtime;W(`;a)\nKtime;W(`;a)\nVtimeg\nIs Space-Time Attention All You Need for Video Understanding?\nAttention Params K400 SSv2\nSpace 85.9M 76.9 36.6\nJoint Space-Time 85.9M 77.4 58.5\nDivided Space-Time 121.4M 78.0 59.5\nSparse Local Global 121.4M 75.9 56.3\nAxial 156.8M 73.5 56.2\nTable 1. Video-level accuracy for different space-time attention\nschemes in TimeSformer. We evaluate the models on the validation sets of Kinetics-400 (K400), and Something-Something-V2\n(SSv2). We observe that divided space-time attention achieves the\nbest results on both datasets.\nandfW(`;a)\nQspace;W(`;a)\nKspace;W(`;a)\nVspacegover the time and space\ndimensions. Note that compared to the (NF+ 1) comparisons per patch needed by the joint spatiotemporal attention\nmodel of Eq. 5, Divided Attention performs only (N+F+2)\ncomparisons per patch. Our experiments demonstrate that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee64-6db4-11ee-8b04-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-251", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "model of Eq. 5, Divided Attention performs only (N+F+2)\ncomparisons per patch. Our experiments demonstrate that\nthis space-time factorization is not only more ef\ufb01cient but it\nalso leads to improved classi\ufb01cation accuracy.\nWe have also experimented with a \u201cSparse Local Global\u201d\n(L+G) and an \u201cAxial\u201d (T+W+H) attention models. Their\narchitectures are illustrated in Fig. 1, while Fig. 2 shows\nthe patches considered for attention by these models. For\neach patch (p;t), (L+G) \ufb01rst computes a local attention by\nconsidering the neighboring F\u0002H=2\u0002W=2patches and\nthen calculates a sparse global attention over the entire clip\nusing a stride of 2 patches along the temporal dimension and\nalso the two spatial dimensions. Thus, it can be viewed as a\nfaster approximation of full spatiotemporal attention using a\nlocal-global decomposition and a sparsity pattern, similar to\nthat used in (Child et al., 2019). Finally, \u201cAxial\u201d attention\ndecomposes the attention computation in three distinct steps:\nover time, width and height. A decomposed attention over\nthe two spatial axes of the image was proposed in (Ho"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee65-6db4-11ee-bbee-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-252", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "decomposes the attention computation in three distinct steps:\nover time, width and height. A decomposed attention over\nthe two spatial axes of the image was proposed in (Ho\net al., 2019; Huang et al., 2019; Wang et al., 2020b) and\nour (T+W+H) adds a third dimension (time) for the case\nof video. All these models are implemented by learning\ndistinct query/key/value matrices for each attention step.\n4. Experiments\nWe evaluate TimeSformer on four popular action recognition datasets: Kinetics-400 (Carreira & Zisserman, 2017),\nKinetics-600 (Carreira et al., 2018), Something-SomethingV2 (Goyal et al., 2017b), and Diving-48 (Li et al., 2018). We\nadopt the \u201cBase\u201d ViT architecture (Dosovitskiy et al., 2020)\npretrained on either ImageNet-1K or ImageNet-21K (Deng\net al., 2009), as speci\ufb01ed for each experiment. Unless differently indicated, we use clips of size 8\u0002224\u0002224, with\nframes sampled at a rate of 1=32. The patch size is 16\u000216\npixels. During inference, unless otherwise noted, we sample a single temporal clip in the middle of the video. We"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee66-6db4-11ee-b312-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-253", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "pixels. During inference, unless otherwise noted, we sample a single temporal clip in the middle of the video. We\nuse3spatial crops (top-left, center, bottom-right) from the\ntemporal clip and obtain the \ufb01nal prediction by averaging\nthe scores for these 3crops.\n8 32 64 96# of Input frames0510TFLOPsJoint Space-TimeDivided Space-Time224 336 448 560Spatial Crop (Px)0123TFLOPsJoint Space-TimeDivided Space-Time\nOut of memory\nOut of memoryFigure 3. We compare the video classi\ufb01cation cost (in TFLOPs) of\nJoint Space-Time versus Divided Space-Time attention. We plot\nthe number of TFLOPs as a function of spatial crop size in pixels\n(left), and the number of input frames (right). As we increase the\nspatial resolution (left), or the video length (right), our proposed divided space-time attention leads to dramatic computational savings\ncompared to the scheme of joint space-time attention.\n4.1. Analysis of Self-Attention Schemes\nFor this \ufb01rst set of experiments we start from a ViT pretrained on ImageNet-21K. In Table 1, we present the results"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee67-6db4-11ee-b38d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-254", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "For this \ufb01rst set of experiments we start from a ViT pretrained on ImageNet-21K. In Table 1, we present the results\nobtained with TimeSformer for the \ufb01ve proposed space-time\nattention schemes on Kinetics-400 (K400) and SomethingSomething-V2 (SSv2). First, we note that TimeSformer\nwith space-only attention (S) performs well on K400. This\nis an interesting \ufb01nding. Indeed, prior work (Sevilla-Lara\net al., 2021) has shown that on K400, spatial cues are more\nimportant than temporal information in order to achieve\nstrong accuracy. Here, we show that it is possible to obtain\nsolid accuracy on K400 without any temporal modeling.\nNote, however, that space-only attention performs poorly on\nSSv2. This stresses the importance of temporal modeling\non this latter dataset.\nFurthermore, we observe that divided space-time attention\nachieves the best accuracy on both K400 and SSv2. This\nmakes sense because compared to joint space-time attention,\ndivided space-time attention has a larger learning capacity\n(see Table 1) as it contains distinct learning parameters for\ntemporal attention and spatial attention.\nIn Figure 3, we also compare the computational cost of joint\nspace-time versus divided space-time attention when using"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee68-6db4-11ee-bbc9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-255", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "temporal attention and spatial attention.\nIn Figure 3, we also compare the computational cost of joint\nspace-time versus divided space-time attention when using\nhigher spatial resolution (left) and longer (right) videos. We\nnote that the scheme of divided space-time scales gracefully\nunder both of these settings. In contrast, the scheme of\njoint space-time attention leads to a dramatically higher cost\nwhen resolution or video length is increased. In practice,\njoint space-time attention causes a GPU memory over\ufb02ow\nonce the spatial frame resolution reaches 448pixels, or once\nthe number of frames is increased to 32and thus it is effectively not applicable to large frames or long videos. Thus,\ndespite a larger number of parameters, divided space-time\nattention is more ef\ufb01cient than joint space-time attention\nwhen operating on higher spatial resolution, or longer videos.\nThus, for all subsequent experiments we use a TimeSformer\nconstructed with divided space-time self-attention blocks.\nIs Space-Time Attention All You Need for Video Understanding?\nModel Pretrain K400 Training\nTime (hours)K400\nAcc.Inference\nTFLOPsParams\nI3D 8x8 R50 ImageNet-1K 444 71.0 1.11 28.0M"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee69-6db4-11ee-b9f2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-256", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Acc.Inference\nTFLOPsParams\nI3D 8x8 R50 ImageNet-1K 444 71.0 1.11 28.0M\nI3D 8x8 R50 ImageNet-1K 1440 73.4 1.11 28.0M\nSlowFast R50 ImageNet-1K 448 70.0 1.97 34.6M\nSlowFast R50 ImageNet-1K 3840 75.6 1.97 34.6M\nSlowFast R50 N/A 6336 76.4 1.97 34.6M\nTimeSformer ImageNet-1K 416 75.8 0.59 121.4M\nTimeSformer ImageNet-21K 416 78.0 0.59 121.4M\nTable 2. Comparing TimeSformer to SlowFast and I3D. We observe that TimeSformer has lower inference cost despite having\na larger number of parameters. Furthermore, the cost of training\nTimeSformer on video data is much lower compared to SlowFast\nand I3D, even when all models are pretrained on ImageNet-1K.\n4.2. Comparison to 3D CNNs\nIn this subsection we perform an empirical study aimed\nat understanding the distinguishing properties of TimeSformer compared to 3D convolutional architectures, which\nhave been the prominent approach to video understanding"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee6a-6db4-11ee-8aba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-257", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "In this subsection we perform an empirical study aimed\nat understanding the distinguishing properties of TimeSformer compared to 3D convolutional architectures, which\nhave been the prominent approach to video understanding\nin recent years. We focus our comparison on two 3D CNN\nmodels: 1) SlowFast (Feichtenhofer et al., 2019b), which is\nthe state-of-the-art in video classi\ufb01cation, and 2) I3D (Carreira & Zisserman, 2017), which has been shown to bene\ufb01t\nfrom image-based pretraining, similarly to our own model.\nWe present quantitative comparisons to these two networks\nin Table 2 and highlight key observations below.\nModel Capacity. From Table 2, we \ufb01rst observe that although TimeSformer has a large learning capacity (the number of parameters is 121:4M), it has low inference cost\n(0:59in TFLOPs). In contrast, SlowFast 8x8 R50 has a\nlarger inference cost ( 1:97TFLOPs) despite containing only\n34:6Mparameters. Similarly, I3D 8x8 R50 also has a larger\ninference cost ( 1:11TFLOPs) despite containing fewer parameters ( 28:0M). This suggests that TimeSformer is better"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee6b-6db4-11ee-8194-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-258", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "inference cost ( 1:11TFLOPs) despite containing fewer parameters ( 28:0M). This suggests that TimeSformer is better\nsuited for settings that involve large-scale learning. In contrast, the large computational cost of modern 3D CNNs\nmakes it dif\ufb01cult to further increase their model capacity\nwhile also maintaining ef\ufb01ciency.\nVideo Training Time. One signi\ufb01cant advantage of ImageNet pretraining is that it enables very ef\ufb01cient training of\nTimeSformer on video data. Conversely, state-of-the-art 3D\nCNNs are much more expensive to train even if pretrained\non image datasets. In Table 2, we compare the video training time on Kinetics-400 (in Tesla V100 GPU hours) of\nTimeSformer to that of SlowFast and I3D. Starting from a\nResNet50 pretrained on ImageNet-1K, SlowFast 8\u00028R50\nrequires 3 840 Tesla V100 GPU hours in order to reach an\naccuracy of 75:6%on Kinetics-400. Training I3D, under\nsimilar settings, requires 1 440 Tesla V100 GPU hours for a\n73:4%accuracy. In contrast, TimeSformer, also pretrained\non ImageNet-1K, only requires 416Tesla V100 GPU hours"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee6c-6db4-11ee-b5e7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-259", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "73:4%accuracy. In contrast, TimeSformer, also pretrained\non ImageNet-1K, only requires 416Tesla V100 GPU hours\nto achieve a higher 75:8%accuracy (see Table 2). Furthermore, if we constrain SlowFast to be trained under a\nsomewhat similar computational budget as TimeSformerMethod Pretraining K400 SSv2\nTimeSformer ImageNet-1K 75.8 59.5\nTimeSformer ImageNet-21K 78.0 59.5\nTimeSformer-HR ImageNet-1K 77.8 62.2\nTimeSformer-HR ImageNet-21K 79.7 62.5\nTimeSformer-L ImageNet-1K 78.1 62.4\nTimeSformer-L ImageNet-21K 80.7 62.3\nTable 3. Comparing the effectiveness of ImageNet-1K and\nImageNet-21K pretraining on Kinetics-400 (K400) and SomethingSomething-V2 (SSv2). On K400, ImageNet-21K pretraining leads\nconsistently to a better performance compared to ImageNet-1K pretraining. On SSv2, ImageNet-1K and ImageNet-21K pretrainings\nlead to similar accuracy."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee6d-6db4-11ee-93f8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-260", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "lead to similar accuracy.\n(i.e.,448GPU hours), its accuracy drops to 70:0%. Similarly, training I3D using a similar computational budget (i.e.,\n444GPU hours) leads to a lower accuracy of 71:0%. This\nhighlights the fact that some of the latest 3D CNNs (Feichtenhofer et al., 2019b; Feichtenhofer, 2020) require a very\nlong optimization schedule to achieve good performance\n(even when using ImageNet pretraining). In contrast, TimeSformer provides a more ef\ufb01cient alternative to labs that do\nnot have access to hundreds of GPUs.\nThe Importance of Pretraining. Due to a large number\nof parameters, training our model from scratch is dif\ufb01cult.\nThus, before training TimeSformer on video data, we initialize it with weights learned from ImageNet. In contrast,\nSlowFast can be learned on video data from scratch although\nat the expense of a very high training cost (see Table 2). We\nalso attempted to train TimeSformer on Kinetics-400 directly, without any ImageNet pretraining. By using a longer\ntraining schedule and more data augmentations, we found\nit possible to train the model from scratch, albeit to a much\nlower video level accuracy of 64:8%. Thus, based on these"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1cee6e-6db4-11ee-a927-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-261", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "training schedule and more data augmentations, we found\nit possible to train the model from scratch, albeit to a much\nlower video level accuracy of 64:8%. Thus, based on these\nresults, for all subsequent studies we continued to use ImageNet for pretraining (Deng et al., 2009)\nIn Table 3 we study the bene\ufb01ts of ImageNet-1K vs\nImageNet-21K pretraining on K400 and SSv2. For these\nexperiments, we use three variants of our model: (1) TimeSformer , which is the default version of our model operating\non8\u0002224\u0002224video clips, (2) TimeSformer-HR , a high\nspatial resolution variant that operates on 16\u0002448\u0002448\nvideo clips, and lastly (3) TimeSformer-L , a long-range\ncon\ufb01guration of our model that operates on 96\u0002224\u0002224\nvideo clips with frames sampled at a rate of 1=4.\nBased on the results in Table 3, we observe that ImageNet21K pretraining is bene\ufb01cial for K400, where it leads to\na consistently higher accuracy compared to ImageNet-1K\npretraining. On the other hand, on SSv2, we observe that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d148c-6db4-11ee-b5d2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-262", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "a consistently higher accuracy compared to ImageNet-1K\npretraining. On the other hand, on SSv2, we observe that\nImageNet-1K and ImageNet-21K pretrainings lead to similar accuracy. This makes sense as SSv2 requires complex\nspatiotemporal reasoning, whereas K400 is biased more towards spatial scene information, and thus, it bene\ufb01ts more\nfrom the features learned on the larger pretraining dataset.\nIs Space-Time Attention All You Need for Video Understanding?\n60K 120K 180K 240K# of Training Videos60657075AccuracyKinetics\nTimeSformerSlowFastI3D42K 85K 127K 170K# of Training Videos354045505560AccuracySomething-Something-V2\nTimeSformerSlowFastI3D\nFigure 4. Accuracy on Kinetics-400 (K400), and SomethingSomething-V2 (SSv2) as a function of the number of training\nvideos. On K400, TimeSformer performs best in all cases. On\nSSv2, which requires more complex temporal reasoning, TimeSformer outperforms the other models only when using enough\ntraining videos. All models are pretrained on ImageNet-1K.\nThe Impact of Video-Data Scale. To understand the\neffects of video-data scale on performance, we trained"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d148d-6db4-11ee-ac59-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-263", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "training videos. All models are pretrained on ImageNet-1K.\nThe Impact of Video-Data Scale. To understand the\neffects of video-data scale on performance, we trained\nTimeSformer on different subsets of K400 and SSv2:\nf25%;50%;75%;100%gof the full datasets. We show\nthese results in Figure 4, where we also compare our method\nwith SlowFast R50 (Feichtenhofer et al., 2019b), and I3D\nR50 (Carreira & Zisserman, 2017) trained on the same subsets and using the same pretraining. Since we do not have\naccess to a ResNet pretrained on ImageNet-21K, we use\nImageNet-1K pretraining for all 3architectures.\nThe results of Figure 4 show that, on K400, TimeSformer\noutperforms the other models for all training subsets. However, we observe a different trend on SSv2, where TimeSformer is the strongest model only when trained on 75% or\n100% of the full data. This may be explained by the fact that\ncompared to K400, SSv2 requires learning more complex\ntemporal patterns, and thus more examples are needed by\nTimeSformer to learn effectively those patterns.\n4.3. Varying the Number of Tokens"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d148e-6db4-11ee-ab8d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-264", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "temporal patterns, and thus more examples are needed by\nTimeSformer to learn effectively those patterns.\n4.3. Varying the Number of Tokens\nThe scalability of our model allows it to operate at higher\nspatial resolution and on longer videos compared to most 3D\nCNNs. We note that both of these aspects affect the length of\nthe sequence of tokens fed to the Transformer. Speci\ufb01cally,\nincreasing the spatial resolution results in a higher number\nof patches (N) per frame. The number of input tokens is\nalso increased when using more frames. To investigate the\nbene\ufb01ts, we conduct an empirical study where we separately\nincrease the number of tokens along each of these two axes.\nWe report the \ufb01ndings in Figure 5. We see that increasing\nthe spatial resolution (up to a certain point) leads to a boost\nin performance. Similarly, we observe that increasing the\nlength of the input clip leads to consistent accuracy gains.\nDue to GPU memory constraints, we are not able to test\nour model on clips longer than 96frames. Still, we would\nlike to point out that using clips of 96frames is a signi\ufb01cant\ndeparture from current convolutional models, which are\ntypically limited to processing inputs of 8\u000032frames."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d148f-6db4-11ee-a5ec-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-265", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "departure from current convolutional models, which are\ntypically limited to processing inputs of 8\u000032frames.\n8 32 64 96Number of Input Frames65707580Clip Accuracy224 336 448 560Spatial Crop Side (Px)65707580Clip AccuracyFigure 5. Clip-level accuracy on Kinetics-400 as a function of spatial crop size in pixels (left), and the number of input frames (right).\nPositional Embedding K400 SSv2\nNone 75.4 45.8\nSpace-only 77.8 52.5\nSpace-Time 78.0 59.5\nTable 4. Ablation on positional embeddings. The version of TimeSformer using space-time positional embeddings yields the highest\naccuracy on both Kinetics-400 and SSv2.\n4.4. The Importance of Positional Embeddings\nTo investigate the importance of our learned spatiotemporal positional embeddings, we also conduct experiments\nwith a few variants of TimeSformer that use: (1) no positional embedding, (2) space-only positional embedding,\nand (3) space-time positional embedding. We report these\nresults in Table 4. Based on these results, we observe\nthat the variant of our model that uses space-time positional embeddings produces the best accuracy on both"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1490-6db4-11ee-a19e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-266", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "results in Table 4. Based on these results, we observe\nthat the variant of our model that uses space-time positional embeddings produces the best accuracy on both\nKinetics-400, and Something-Something-V2. Interestingly,\nwe also observe that using space-only positional embeddings leads to solid results on Kinetics-400, but much worse\nresults on Something-Something-V2. This makes sense as\nKinetics-400 is more spatially biased, whereas SomethingSomething-V2 requires complex temporal reasoning.\n4.5. Comparison to the State-of-the-Art\nKinetics-400 & Kinetics-600. In Table 5 we present our\nresults on the validation set of K400. For these experiments,\nwe use TimeSformer pretrained on ImageNet-21K. In addition to the accuracy metrics, we also include inference\ncost, given in TFLOPs. We note that whereas most previous\nMethod Top-1 Top-5 TFLOPs\nR(2+1)D (Tran et al., 2018) 72.0 90.0 17.5\nbLVNet (Fan et al., 2019) 73.5 91.2 0.84\nTSM (Lin et al., 2019) 74.7 N/A N/A\nS3D-G (Xie et al., 2018) 74.7 93.4 N/A"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1491-6db4-11ee-a652-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-267", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "S3D-G (Xie et al., 2018) 74.7 93.4 N/A\nOct-I3D+NL (Chen et al., 2019) 75.7 N/A 0.84\nD3D (Stroud et al., 2020) 75.9 N/A N/A\nI3D+NL (Wang et al., 2018b) 77.7 93.3 10.8\nip-CSN-152 (Tran et al., 2019) 77.8 92.8 3.2\nCorrNet (Wang et al., 2020a) 79.2 N/A 6.7\nLGD-3D-101 (Qiu et al., 2019) 79.4 94.4 N/A\nSlowFast (Feichtenhofer et al., 2019b) 79.8 93.9 7.0\nX3D-XXL (Feichtenhofer, 2020) 80.4 94.6 5.8\nTimeSformer 78.0 93.7 0.59\nTimeSformer-HR 79.7 94.4 5.11\nTimeSformer-L 80.7 94.7 7.14\nTable 5. Video-level accuracy on Kinetics-400.\nIs Space-Time Attention All You Need for Video Understanding?\nMethod Top-1 Top-5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1492-6db4-11ee-b555-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-268", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Table 5. Video-level accuracy on Kinetics-400.\nIs Space-Time Attention All You Need for Video Understanding?\nMethod Top-1 Top-5\nI3D-R50+Cell (Wang et al., 2020c) 79.8 94.4\nLGD-3D-101 (Qiu et al., 2019) 81.5 95.6\nSlowFast (Feichtenhofer et al., 2019b) 81.8 95.1\nX3D-XL (Feichtenhofer, 2020) 81.9 95.5\nTimeSformer 79.1 94.4\nTimeSformer-HR 81.8 95.8\nTimeSformer-L 82.2 95.6\nTable 6. Video-level accuracy on Kinetics-600.\n1 3 5 10\n# of Testing Clips65707580Accuracy\nTimeSformer-L\nTimeSformer\nSlowFast-R101+NL\nX3D-XL\nFigure 6. Video-level accuracy on Kinetics-400 vs the number of\ntemporal clips used during inference. TimeSformer-L achieves\nexcellent accuracy using a small number of clips, which leads to\nstrong performance at low inference cost.\nmethods use 10temporal clips with 3spatial crops (for a total of 30space-time views) during inference, TimeSformer"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1493-6db4-11ee-ad22-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-269", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "strong performance at low inference cost.\nmethods use 10temporal clips with 3spatial crops (for a total of 30space-time views) during inference, TimeSformer\nachieves solid accuracy with only 3views (3 spatial crops),\nwhich reduces the inference cost. Our long-range variant,\nTimeSformer-L achieves a top-1 accuracy of 80:7%. Furthermore, our default TimeSformer has the lowest inference\ncost among recent state-of-the-art models. Yet, it still provides a solid accuracy of 78:0%, outperforming many more\ncostly models.\nWe also measured the actual inference runtime on 20Kvalidation videos of Kinetics-400 (using 8Tesla V100 GPUs).\nWhereas SlowFast takes 14:88hours to complete the inference, TimeSformer, TimeSformer-HR, and TimeSformer-L\ntake36minutes, 1:06hours and 2:6hours, respectively.\nThus, even though SlowFast and TimeSformer-L have comparable cost in terms of TFLOPs, in practice the runtimes\nof all our versions of TimeSformer are much lower.\nIn Table 6, we also present our results on Kinetics-600.\nJust like on Kinetics-400, we observe that TimeSformer\nperforms well on this benchmark, outperforming all prior\nmethods."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1494-6db4-11ee-ab25-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-270", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Just like on Kinetics-400, we observe that TimeSformer\nperforms well on this benchmark, outperforming all prior\nmethods.\nFinally, in Figure 6, we study the effect of using multiple temporal clips during inference (each with a single\nspatial crop). We plot accuracy using K2f1;3;5;10g\ntemporal clips for testing. We compare our model against\nX3D (Feichtenhofer, 2020), and SlowFast (Feichtenhofer\net al., 2019b). X3D and SlowFast require multiple ( \u00155)\nclips to approach their top accuracy. Conversely, our longrange variant, TimeSformer-L, does not require multiple\nclips to achieve its best performance, since it is able to spanMethod SSv2 Diving-48\u0003\u0003\nSlowFast (Feichtenhofer et al., 2019b) 61.7 77.6\nTSM (Lin et al., 2019) 63.4 N/A\nSTM (Jiang et al., 2019) 64.2 N/A\nMSNet (Kwon et al., 2020) 64.7 N/A\nTEA (Li et al., 2020b) 65.1 N/A\nbLVNet (Fan et al., 2019) 65.2 N/A\nTimeSformer 59.5 74.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1495-6db4-11ee-8e5d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-271", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "bLVNet (Fan et al., 2019) 65.2 N/A\nTimeSformer 59.5 74.9\nTimeSformer-HR 62.2 78.0\nTimeSformer-L 62.4 81.0\nTable 7. Video-level accuracy on Something-Something-V2 and\nDiving-48.\u0003\u0003Due to an issue with Diving-48 labels used in previously published results, we only compare our method with a\nreproduced SlowFast 16\u00028R101 model. All models are pretained on ImageNet-1K.\nabout 12seconds of a Kinetics video with a single clip.\nSomething-Something-V2 & Diving-48. In Table 7, we\nalso validate our model on SSv2 and Diving-48. Since\nImageNet-21K pretraining does not improve accuracy on\nSSv2 (see Table 3), in this case, we use TimeSformer pretrained on ImageNet-1K. This also allows us to apply the\nsame pretraining to all other models in this comparison,\nusing a ResNet pretrained on ImageNet-1K. Our results suggest that TimeSformer achieves lower accuracy than the best\nmodels on this dataset. However, considering that our model\nuses a completely different design, we take these results as\nsuggesting that TimesSformer is a promising approach even"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1496-6db4-11ee-8d62-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-272", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "models on this dataset. However, considering that our model\nuses a completely different design, we take these results as\nsuggesting that TimesSformer is a promising approach even\nfor challenging temporally-heavy datasets, such as SSv2.\nIn Table 7, we also present our method on another\n\u201ctemporally-heavy\u201d dataset, Diving-48. Due to a recently\ndiscovered issue with a previous version of Diving-48 labels,\nhere, we only compare our method with a reproduced SlowFast16\u00028R101 model. Our results show that TimeSformer\noutperforms SlowFast by a substantial margin.\n4.6. Long-Term Video Modeling\nLastly, we evaluate TimeSformer on the task of long-term\nvideo modeling using HowTo100M (Miech et al., 2019).\nHowTo100M is an instructional video dataset that contains\naround 1M instructional Web videos showing humans performing over 23K different tasks, such as cooking, repairing,\nmaking arts, etc. The average duration of these videos is\naround 7minutes, which is orders of magnitude longer than\nthe duration of videos in standard action recognition benchmarks. Each HowTo100M video has a label indicating the\ntask demonstrated in the video (one out of the 23K classes),\nwhich can be used for supervised training. Thus, it is a good"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1497-6db4-11ee-9a87-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-273", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "task demonstrated in the video (one out of the 23K classes),\nwhich can be used for supervised training. Thus, it is a good\nbenchmark to assess the ability of a model to recognize\nactivities exhibited over very long temporal extents.\nFor this evaluation, we consider only categories that have\nat least 100 video examples. This gives a subset of\nHowTo100M corresponding to 120Kvideos spanning 1059\ntask categories. We randomly partition this collection into\n85Ktraining videos and 35Ktesting videos.\nIs Space-Time Attention All You Need for Video Understanding?\nMethod # Input\nFramesSingle Clip\nCoverage# Test\nClipsTop-1\nAcc\nSlowFast 8 8.5s 48 48.2\nSlowFast 32 34.1s 12 50.8\nSlowFast 64 68.3s 6 51.5\nSlowFast 96 102.4s 4 51.2\nTimeSformer 8 8.5s 48 56.8\nTimeSformer 32 34.1s 12 61.2\nTimeSformer 64 68.3s 6 62.2\nTimeSformer 96 102.4s 4 62.6\nTable 8. Long-term task classi\ufb01cation on HowTo100M. Given a\nvideo spanning several minutes, the goal is to predict the long-term\ntask demonstrated in the video (e.g., cooking breakfast, cleaning"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1498-6db4-11ee-a1b8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-274", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "video spanning several minutes, the goal is to predict the long-term\ntask demonstrated in the video (e.g., cooking breakfast, cleaning\nhouse, etc). We evaluate a few variants of SlowFast and TimeSformer on this task. \u201cSingle Clip Coverage\u201d denotes the number\nof seconds spanned by a single clip. \u201c# Test Clips\u201d is the average\nnumber of clips needed to cover the entire video during inference.\nAll models in this comparison are pretrained on Kinetics-400.\nWe present our results in Table 8. As our baselines, we use\nfour variants of SlowFast R101, all operating on video clips\nsampled at a frame rate of 1=32but having varying number\nof frames: 8;32;64and96. We use the same four con\ufb01gurations for TimeSformer, starting from a ViT pretrained\non ImageNet-21K. All models in this comparison are pretrained on Kinetics-400 before \ufb01netuning on HowTo100M.\nDuring inference, for each method, we sample as many\nnon-overlapping temporal clips as needed to cover the full\ntemporal extent of a video, e.g., if a single clip spans 8:5\nseconds, we would sample 48test clips to cover a video of\n410seconds. Video-level classi\ufb01cation is done by averaging"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1499-6db4-11ee-b34e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-275", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "seconds, we would sample 48test clips to cover a video of\n410seconds. Video-level classi\ufb01cation is done by averaging\nthe clip predictions.\nFrom the results in Table 8 we \ufb01rst note that, for the same\nsingle clip coverage, TimeSformer outperforms the corresponding SlowFast by a large margin of 8\u000011%. We also\nobserve that longer-range TimeSformers do better, i.e., our\nlongest-range variant achieves the best video-level classi\ufb01cation accuracy. These results suggest that our model is highly\nsuitable for tasks that require long-term video modeling.\nWe also experimented with \ufb01netuning TimeSformer directly\nfrom a ViT pretrained on ImageNet-1K and ImageNet21K (skipping the Kinetics-400 training). We report that\nwhen pretrained only on ImageNet-1K, our model achieves\ntop-1 accuracies of 52:8;58:4;59:2;59:4for8;32;64;96\nframe inputs, respectively. When considering ImagNet21K pretraining, TimeSformer produces top-1 accuracies of\n56:0;59:2;60:2;62:1for8;32;64;96frame inputs, respectively. These results demonstrate that our model can effectively exploit long-range temporal dependencies regardless"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d149a-6db4-11ee-af1f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-276", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "of the pretraining dataset that we use.\n4.7. Additional Ablations\nSmaller & Larger Transformers. In addition to the \u201cBase\u201d\nViT model (Dosovitskiy et al., 2020), we also experimented\nwith the \u201cLarge\u201d ViT. We report that this yielded results 1%\nFigure 7. Visualization of space-time attention from the output\ntoken to the input space on Something-Something-V2. Our model\nlearns to focus on the relevant parts in the video in order to perform\nspatiotemporal reasoning.\nViT-50 -40 -30 -20 -10 0 10 20 30 40 50-50-40-30-20-1001020304050\n-60 -40 -20 0 20 40-40-30-20-1001020304050       TimeSformer w/ Divided            Space-Time Attention-60 -40 -20 0 20 40 60-60-40-200204060    TimeSformer  w/ Space Attention\nFigure 8. Feature visualization with t-SNE (van der Maaten & Hinton, 2008) on Something-Something-V2. Each video is visualized\nas a point. Videos belonging to the same action category have the\nsame color. The TimeSformer with divided space-time attention"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d149b-6db4-11ee-957b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-277", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "as a point. Videos belonging to the same action category have the\nsame color. The TimeSformer with divided space-time attention\nlearns semantically more separable features than the TimeSformer\nwith space-only attention or ViT (Dosovitskiy et al., 2020).\nworse on both Kinetics-400, and Something-Something-V2.\nGiven that our \u201cBase\u201d model already has 121Mparameters,\nwe suspect that the current datasets are not big enough to\njustify a further increase in model capacity. We also tried\nthe \u201cSmall\u201d ViT variant, which produced accuracies about\n5%worse than our default \u201cBase\u201d ViT model.\nLarger Patch Size. We also experimented with a different\npatch size, i.e., P= 32 . We report that this variant of our\nmodel produced results about 3%worse than our default\nvariant using P= 16 . We conjecture that the performance\ndecrease with P= 32 is due to the reduced spatial granularity. We did not train any models with Pvalues lower than\n16 as those models have a much higher computational cost.\nThe Order of Space and Time Self-Attention. Our proposed \u201cDivided Space-Time Attention\u201d scheme applies temporal attention and spatial attention one after the other. Here,\nwe investigate whether reversing the order of time-space"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d149c-6db4-11ee-a9df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-278", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "we investigate whether reversing the order of time-space\nattention (i.e., applying spatial attention \ufb01rst, then temporal) has an impact on our results. We report that applying spatial attention \ufb01rst, followed by temporal attention\nleads to a 0:5%drop in accuracy on both Kinetics-400, and\nSomething-Something-V2. We also tried a parallel spacetime self-attention. We report that it produces 0:4%lower\naccuracy compared to our adopted \u201cDivided Space-Time\nAttention\u201d scheme.\nIs Space-Time Attention All You Need for Video Understanding?\n4.8. Qualitative Results\nVisualizing Learned Space-Time Attention. In Figure 7,\nwe present space-time attention visualizations obtained by\napplying TimeSformer on Something-Something-V2 videos.\nTo visualize the learned attention, we use the Attention\nRollout scheme presented in (Abnar & Zuidema, 2020).\nOur results suggest that TimeSformer learns to attend to the\nrelevant regions in the video in order to perform complex\nspatiotemporal reasoning. For example, we can observe that\nthe model focuses on the con\ufb01guration of the hand when\nvisible and the object-only when not visible.\nVisualizing Learned Feature Embeddings. In Figure 8,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d149d-6db4-11ee-9495-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-279", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "the model focuses on the con\ufb01guration of the hand when\nvisible and the object-only when not visible.\nVisualizing Learned Feature Embeddings. In Figure 8,\nwe also visualize the features learned by TimeSformer on\nSomething-Something-V2. The visualization is done using\nt-SNE (van der Maaten & Hinton, 2008) where each point\nrepresents a single video, and different colors depict different action categories. Based on this illustration, we observe\nthat TimeSformer with divided space-time attention learns\nsemantically more separable features than the TimeSformer\nwith space-only attention or ViT (Dosovitskiy et al., 2020).\n5. Conclusion\nIn this work, we introduced TimeSformer, a fundamentally\ndifferent approach to video modeling compared to the established paradigm of convolution-based video networks.\nWe showed that it is possible to design an effective, and\nscalable video architecture built exclusively on space-time\nself-attention. Our method (1) is conceptually simple, (2)\nachieves state-of-the-art results on major action recognition\nbenchmarks, (3) has low training and inference cost, and (4)\ncan be applied to clips of over one minute, thus enabling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d149e-6db4-11ee-854d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-280", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "benchmarks, (3) has low training and inference cost, and (4)\ncan be applied to clips of over one minute, thus enabling\nlong-term video modeling. In the future, we plan to extend our method to other video analysis tasks such as action\nlocalization, video captioning and question-answering.\nAppendix\nA. Implementation Details\nOur TimeSformer implementation is built using PySlowFast (Fan et al., 2020) and pytorch-image-models (Wightman, 2019) packages. Below, we describe speci\ufb01c implementation details regarding the training and inference procedures of our model.\nTraining. We train our model for 15epochs with an initial\nlearning rate of 0:005, which is divided by 10at epochs\n11;and14. During training, we \ufb01rst resize the shorter\nside of the video to a random value in [256;320]. We then\nrandomly sample a 224\u0002224crop from the resized video.\nFor our high-resolution model, TimeSformer-HR, we resize\nthe shorter side of the video to a random value in [448;512],\nand then randomly sample a 448\u0002448crop. We randomlysample clips from the full-length videos with a frame rate\nof1=32. The batch size is set to 16. We train all our models"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d149f-6db4-11ee-9111-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-281", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "of1=32. The batch size is set to 16. We train all our models\nusing synchronized SGD across 32GPUs. The momentum\nis set to 0:9, while the weight decay is set to 0:0001 .\nUnless otherwise noted, in our experiments we use the\n\u201cBase\u201d ViT model (Dosovitskiy et al., 2020). Temporal and\nspatial attention layers in each block are initialized with the\nsame weights, which are obtained from the corresponding\nattention layer in ViT.\nInference. As discussed in the main draft, during inference\nwe sample a single temporal clip in the middle of the video.\nWe scale the shorter spatial side of a video to 224pixels (or\n448for TimeSformer-HR) and take 3crops of size 224\u0002224\n(448\u0002448for TimeSformer-HR) to cover a larger spatial\nextent within the clip. The \ufb01nal prediction is obtained by\naveraging the softmax scores of these 3predictions.\nOther models in our comparison. To train I3D (Carreira\n& Zisserman, 2017), and SlowFast (Feichtenhofer et al.,\n2019b), we use the training protocols that were used in the\noriginal papers. For I3D, we initialize it with a 2D ImageNet"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a0-6db4-11ee-8fde-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-282", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "2019b), we use the training protocols that were used in the\noriginal papers. For I3D, we initialize it with a 2D ImageNet\nCNN, and then train it for 118epochs with a base learning\nrate of 0:01, which is divided by 10at epochs 44and88.\nWe use synchronized SGD across 32GPUs following the\nlinear scaling recipe of Goyal et al. (2017a). We set the\nmomentum to 0:9, and weight decay to 0:0001 . The batch\nsize is set to 64. For the SlowFast model, when initialized\nfrom ImageNet weights, we use this same exact training\nprotocol. When training SlowFast from scratch, we use the\ntraining protocol described by the authors (Feichtenhofer\net al., 2019b). More speci\ufb01cally, in that case, the training\nis done for 196epochs with a cosine learning rate schedule,\nand the initial learning rate is set to 0:1. We use a linear\nwarm-up for the \ufb01rst 34epochs starting with a learning rate\nof0:01. A dropout of 0:5is used before the \ufb01nal classi\ufb01cation layer. The momentum is set to 0:9, the weight decay is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a1-6db4-11ee-8edb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-283", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "0:0001 , and the batch size is set to 64. Just as before, we\nadopt the linear scaling recipe (Goyal et al., 2017a).\nDatasets. Kinetics-400 (Carreira & Zisserman, 2017) consists of 240Ktraining videos and 20Kvalidation videos\nthat span 400human action categories. Kinetics-600 (Carreira et al., 2018) has 392Ktraining videos and 30Kvalidation videos spanning 600action categories. SomethingSomething-V2 (Goyal et al., 2017b) contains 170Ktraining\nvideos and 25Kvalidation videos that span 174action categories. Lastly, Diving-48 (Li et al., 2018) has 16Ktraining\nvideos and 3Ktesting videos spanning 48\ufb01ne-grained diving categories. For all of these datasets, we use standard\nclassi\ufb01cation accuracy as our main performance metric.\nIs Space-Time Attention All You Need for Video Understanding?\nReferences\nAbnar, S. and Zuidema, W. Quantifying attention \ufb02ow in\ntransformers, 2020.\nBa, L. J., Kiros, J. R., and Hinton, G. E. Layer normalization.\nCoRR , 2016.\nBello, I., Zoph, B., Le, Q., Vaswani, A., and Shlens, J."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a2-6db4-11ee-9bb3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-284", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "CoRR , 2016.\nBello, I., Zoph, B., Le, Q., Vaswani, A., and Shlens, J.\nAttention augmented convolutional networks. In 2019\nIEEE/CVF International Conference on Computer Vision,\nICCV , 2019.\nBertasius, G. and Torresani, L. Classifying, segmenting, and\ntracking object instances in video with mask propagation.\nInThe IEEE Conference on Computer Vision and Pattern\nRecognition (CVPR) , June 2020.\nBrown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,\nJ., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,\nAskell, A., Agarwal, S., Herbert-V oss, A., Krueger, G.,\nHenighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,\nJ., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.,\nGray, S., Chess, B., Clark, J., Berner, C., McCandlish,\nS., Radford, A., Sutskever, I., and Amodei, D. Language\nmodels are few-shot learners. 2020."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a3-6db4-11ee-b5a1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-285", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "S., Radford, A., Sutskever, I., and Amodei, D. Language\nmodels are few-shot learners. 2020.\nCarion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov,\nA., and Zagoruyko, S. End-to-end object detection with\ntransformers. In European Conference Computer Vision\n(ECCV) , 2020.\nCarreira, J. and Zisserman, A. Quo vadis, action recognition? A new model and the kinetics dataset. In 2017 IEEE\nConference on Computer Vision and Pattern Recognition,\nCVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , 2017.\nCarreira, J., Noland, E., Banki-Horvath, A., Hillier, C., and\nZisserman, A. A short note about kinetics-600. CoRR ,\n2018.\nChen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey,\nW., Foster, G., Jones, L., Schuster, M., Shazeer, N., Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Chen,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a4-6db4-11ee-8eab-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-286", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Z., Wu, Y ., and Hughes, M. The best of both worlds:\nCombining recent advances in neural machine translation.\nInProceedings of the 56th Annual Meeting of the Association for Computational Linguistics . Association for\nComputational Linguistics, 2018a.\nChen, Y ., Kalantidis, Y ., Li, J., Yan, S., and Feng, J. A\u02c62nets: Double attention networks. In Advances in Neural\nInformation Processing Systems 31 , 2018b.\nChen, Y ., Fan, H., Xu, B., Yan, Z., Kalantidis, Y ., Rohrbach,\nM., Yan, S., and Feng, J. Drop an octave: Reducing\nspatial redundancy in convolutional neural networks with\noctave convolution. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision (ICCV) ,\nOctober 2019.\nChild, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. CoRR ,\n2019.\nCordonnier, J., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In\n8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a5-6db4-11ee-864e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-287", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,\n2020 , 2020.\nDai, Z., Yang, Z., Yang, Y ., Carbonell, J., Le, Q., and\nSalakhutdinov, R. Transformer-XL: Attentive language\nmodels beyond a \ufb01xed-length context. In Proceedings of\nthe 57th Annual Meeting of the Association for Computational Linguistics , 2019.\nDeng, J., Dong, W., Socher, R., Li, L., Kai Li, and Li Fei-Fei.\nImagenet: A large-scale hierarchical image database. In\n2009 IEEE Conference on Computer Vision and Pattern\nRecognition , pp. 248\u2013255, 2009. doi: 10.1109/CVPR.\n2009.5206848.\nDevlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert:\nPre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 ,\n2018.\nDevlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a6-6db4-11ee-b9e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-288", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "2018.\nDevlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT:\nPre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association\nfor Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.\nDosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,\nD., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,\nM., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N.\nAn image is worth 16x16 words: Transformers for image\nrecognition at scale. CoRR , 2020.\nFan, H., Li, Y ., Xiong, B., Lo, W.-Y ., and Feichtenhofer, C. Pyslowfast. https://github.com/\nfacebookresearch/slowfast , 2020.\nFan, Q., Chen, C.-F. R., Kuehne, H., Pistoia, M., and Cox,\nD. More is less: Learning ef\ufb01cient video representations\nby big-little network and depthwise temporal aggregation."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a7-6db4-11ee-a1d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-289", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "D. More is less: Learning ef\ufb01cient video representations\nby big-little network and depthwise temporal aggregation.\nInAdvances in Neural Information Processing Systems ,\nvolume 32, 2019.\nFeichtenhofer, C. X3d: Expanding architectures for ef\ufb01cient\nvideo recognition. CVPR , pp. 200\u2013210, 2020.\nFeichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast\nnetworks for video recognition. In Proceedings of the\nIs Space-Time Attention All You Need for Video Understanding?\nIEEE/CVF International Conference on Computer Vision\n(ICCV) , 2019a.\nFeichtenhofer, C., Fan, H., Malik, J., and He, K. Slowfast\nnetworks for video recognition. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV , 2019b.\nGavrilyuk, K., Sanford, R., Javan, M., and Snoek, C. G. M.\nActor-transformers for group activity recognition. In 2020\nIEEE/CVF Conference on Computer Vision and Pattern\nRecognition, CVPR , 2020.\nGirdhar, R., Carreira, J., Doersch, C., and Zisserman, A.\nVideo action transformer network. In IEEE Conference on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a8-6db4-11ee-82a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-290", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Girdhar, R., Carreira, J., Doersch, C., and Zisserman, A.\nVideo action transformer network. In IEEE Conference on\nComputer Vision and Pattern Recognition, CVPR , 2019.\nGoyal, P., Doll\u00e1r, P., Girshick, R., Noordhuis, P.,\nWesolowski, L., Kyrola, A., Tulloch, A., Jia, Y ., and\nHe, K. Accurate, large minibatch sgd: Training imagenet\nin 1 hour. arXiv preprint arXiv:1706.02677 , 2017a.\nGoyal, R., Kahou, S. E., Michalski, V ., Materzynska, J.,\nWestphal, S., Kim, H., Haenel, V ., Fr\u00fcnd, I., Yianilos,\nP., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I.,\nand Memisevic, R. The \"something something\" video\ndatabase for learning and evaluating visual common sense.\nCoRR , 2017b.\nHo, J., Kalchbrenner, N., Weissenborn, D., and Salimans, T.\nAxial attention in multidimensional transformers. CoRR ,\n2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14a9-6db4-11ee-8177-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-291", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Axial attention in multidimensional transformers. CoRR ,\n2019.\nHu, H., Gu, J., Zhang, Z., Dai, J., and Wei, Y . Relation networks for object detection. In 2018 IEEE Conference on\nComputer Vision and Pattern Recognition, CVPR , 2018.\nHuang, Z., Wang, X., Huang, L., Huang, C., Wei, Y ., and\nLiu, W. Ccnet: Criss-cross attention for semantic segmentation. 2019.\nJiang, B., Wang, M., Gan, W., Wu, W., and Yan, J. Stm:\nSpatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International\nConference on Computer Vision (ICCV) , October 2019.\nKwon, H., Kim, M., Kwak, S., and Cho, M. Motionsqueeze:\nNeural motion feature learning for video understanding.\nInECCV , 2020.\nLe, H., Sahoo, D., Chen, N., and Hoi, S. Multimodal transformer networks for end-to-end video-grounded dialogue\nsystems. In Proceedings of the 57th Annual Meeting of\nthe Association for Computational Linguistics , 2019."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14aa-6db4-11ee-bb98-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-292", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "systems. In Proceedings of the 57th Annual Meeting of\nthe Association for Computational Linguistics , 2019.\nLi, L., Chen, Y .-C., Cheng, Y ., Gan, Z., Yu, L., and\nLiu, J. Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint\narXiv:2005.00200 , 2020a.Li, Y ., Li, Y ., and Vasconcelos, N. Resound: Towards action\nrecognition without representation bias. In The European Conference on Computer Vision (ECCV) , September\n2018.\nLi, Y ., Ji, B., Shi, X., Zhang, J., Kang, B., and Wang, L. Tea:\nTemporal excitation and aggregation for action recognition. In Proceedings of the IEEE/CVF Conference on\nComputer Vision and Pattern Recognition (CVPR) , June\n2020b.\nLin, J., Gan, C., and Han, S. Tsm: Temporal shift module for\nef\ufb01cient video understanding. In Proceedings of the IEEE\nInternational Conference on Computer Vision , 2019.\nMiech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ab-6db4-11ee-af73-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-293", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "International Conference on Computer Vision , 2019.\nMiech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev,\nI., and Sivic, J. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video\nClips. In ICCV , 2019.\nOtt, M., Edunov, S., Grangier, D., and Auli, M. Scaling\nneural machine translation. In Proceedings of the Third\nConference on Machine Translation: Research Papers ,\n2018.\nParmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer,\nN., Ku, A., and Tran, D. Image transformer. In Dy, J. G.\nand Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML , 2018.\nQiu, Z., Yao, T., Ngo, C.-W., Tian, X., and Mei, T. Learning spatio-temporal representation with local and global\ndiffusion. In CVPR , 2019.\nRadford, A., Narasimhan, K., Salimans, T., and Sutskever,\nI. Improving language understanding by generative pretraining. 2018."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ac-6db4-11ee-af9e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-294", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Radford, A., Narasimhan, K., Salimans, T., and Sutskever,\nI. Improving language understanding by generative pretraining. 2018.\nRadford, A., Wu, J., Child, R., Luan, D., Amodei, D., and\nSutskever, I. Language models are unsupervised multitask\nlearners. 2019.\nRamachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., and Shlens, J. Stand-alone self-attention in\nvision models. In Advances in Neural Information Processing Systems , pp. 68\u201380, 2019.\nSevilla-Lara, L., Zha, S., Yan, Z., Goswami, V ., Feiszli,\nM., and Torresani, L. Only time can tell: Discovering\ntemporal data for temporal modeling. In Proceedings\nof the IEEE/CVF Winter Conference on Applications of\nComputer Vision (WACV) , pp. 535\u2013544, January 2021.\nSimonyan, K. and Zisserman, A. Very deep convolutional\nnetworks for large-scale image recognition. In ICLR ,\n2015.\nIs Space-Time Attention All You Need for Video Understanding?"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ad-6db4-11ee-bb5e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-295", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "networks for large-scale image recognition. In ICLR ,\n2015.\nIs Space-Time Attention All You Need for Video Understanding?\nStroud, J., Ross, D., Sun, C., Deng, J., and Sukthankar, R.\nD3d: Distilled 3d networks for video action recognition.\nInProceedings of the IEEE/CVF Winter Conference on\nApplications of Computer Vision (WACV) , March 2020.\nSun, C., Myers, A., V ondrick, C., Murphy, K., and Schmid,\nC. Videobert: A joint model for video and language\nrepresentation learning, 2019.\nSzegedy, C., Liu, W., Jia, Y ., Sermanet, P., Reed, S.,\nAnguelov, D., Erhan, D., Vanhoucke, V ., and Rabinovich,\nA. Going deeper with convolutions. In Computer Vision\nand Pattern Recognition (CVPR) , 2015.\nTeed, Z. and Deng, J. RAFT: recurrent all-pairs \ufb01eld transforms for optical \ufb02ow. In Computer Vision - ECCV 2020\n- 16th European Conference, Glasgow, UK, August 23-28,\n2020, Proceedings, Part II , 2020."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ae-6db4-11ee-ae6b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-296", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "- 16th European Conference, Glasgow, UK, August 23-28,\n2020, Proceedings, Part II , 2020.\nTran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y ., and\nPaluri, M. A closer look at spatiotemporal convolutions\nfor action recognition. In 2018 IEEE Conference on\nComputer Vision and Pattern Recognition, Salt Lake City,\nUSA, 2018 , 2018.\nTran, D., Wang, H., Feiszli, M., and Torresani, L. Video\nclassi\ufb01cation with channel-separated convolutional networks. ICCV , pp. 5551\u20135560, 2019.\nvan der Maaten, L. and Hinton, G. Visualizing data using t-SNE. Journal of Machine Learning Research , 9:\n2579\u20132605, 2008. URL http://www.jmlr.org/\npapers/v9/vandermaaten08a.html .\nVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,\nL., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information\nProcessing Systems , 2017a."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14af-6db4-11ee-a083-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-297", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Processing Systems , 2017a.\nVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones,\nL., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information\nProcessing Systems 30 . 2017b.\nWang, H., Tran, D., Torresani, L., and Feiszli, M. Video\nmodeling with correlation networks. In Proceedings of\nthe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020a.\nWang, H., Zhu, Y ., Green, B., Adam, H., Yuille, A. L., and\nChen, L. Axial-deeplab: Stand-alone axial-attention for\npanoptic segmentation. In Computer Vision - ECCV 2020\n- 16th European Conference , 2020b.\nWang, X., Girshick, R., Gupta, A., and He, K. Non-local\nneural networks. In Proceedings of the IEEE Conference\non Computer Vision and Pattern Recognition (CVPR) ,\nJune 2018a.Wang, X., Girshick, R. B., Gupta, A., and He, K. Non-local"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b0-6db4-11ee-bca3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-298", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "June 2018a.Wang, X., Girshick, R. B., Gupta, A., and He, K. Non-local\nneural networks. In 2018 IEEE Conference on Computer\nVision and Pattern Recognition, CVPR 2018, Salt Lake\nCity, UT, USA, June 18-22, 2018 , 2018b.\nWang, X., Xiong, X., Neumann, M., Piergiovanni, A. J.,\nRyoo, M. S., Angelova, A., Kitani, K. M., and Hua, W.\nAttentionnas: Spatiotemporal attention cell search for\nvideo classi\ufb01cation. In Computer Vision - ECCV 2020 16th European Conference, Glasgow, UK, August 23-28,\n2020, Proceedings, Part VIII , 2020c.\nWeissenborn, D., T\u00e4ckstr\u00f6m, O., and Uszkoreit, J. Scaling autoregressive video models. In 8th International\nConference on Learning Representations, ICLR , 2020.\nWightman, R. Pytorch image models. https://github.\ncom/rwightman/pytorch-image-models ,\n2019.\nXie, S., Sun, C., Huang, J., Tu, Z., and Murphy, K."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b1-6db4-11ee-accb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-299", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "com/rwightman/pytorch-image-models ,\n2019.\nXie, S., Sun, C., Huang, J., Tu, Z., and Murphy, K.\nRethinking spatiotemporal feature learning: Speedaccuracy trade-offs in video classi\ufb01cation. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XV , pp. 318\u2013335, 2018. doi: 10.1007/\n978-3-030-01267-0\\_19. URL https://doi.org/\n10.1007/978-3-030-01267-0_19 .\nYang, Z., Garcia, N., Chu, C., Otani, M., Nakashima, Y ., and\nTakemura, H. Bert representations for video question answering. In The IEEE Winter Conference on Applications\nof Computer Vision , 2020.\nZhao, H., Jia, J., and Koltun, V . Exploring self-attention\nfor image recognition. In 2020 IEEE/CVF Conference on\nComputer Vision and Pattern Recognition, CVPR , 2020.\nZhou, L., Zhou, Y ., Corso, J. J., Socher, R., and Xiong, C."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b2-6db4-11ee-8296-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2102.05095-300", "title": "Is Space-Time Attention All You Need for Video Understanding?", "source": "http://arxiv.org/pdf/2102.05095", "start_index": 0, "chroma:document": "Zhou, L., Zhou, Y ., Corso, J. J., Socher, R., and Xiong, C.\nEnd-to-end dense video captioning with masked transformer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b3-6db4-11ee-846c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-0", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Uni\ufb01ed Graph Structured Models for Video Understanding\nAnurag Arnab Chen Sun Cordelia Schmid\nGoogle Research\nfaarnab, chensun, cordelias g@google.com\nAbstract\nAccurate video understanding involves reasoning about\nthe relationships between actors, objects and their environment, often over long temporal intervals. In this paper,\nwe propose a message passing graph neural network that\nexplicitly models these spatio-temporal relations and can\nuse explicit representations of objects, when supervision is\navailable, and implicit representations otherwise. Our formulation generalises previous structured models for video\nunderstanding, and allows us to study how different design choices in graph structure and representation affect\nthe model\u2019s performance. We demonstrate our method on\ntwo different tasks requiring relational reasoning in videos\n\u2013 spatio-temporal action detection on AVA and UCF10124, and video scene graph classi\ufb01cation on the recent Action Genome dataset \u2013 and achieve state-of-the-art results\non all three datasets. Furthermore, we show quantitatively\nand qualitatively how our method is able to more effectively\nmodel relationships between relevant entities in the scene.\n1. Introduction\nDeep learning has enabled rapid advances in many image understanding tasks such as image classi\ufb01cation [20],\nobject detection [46] and semantic segmentation [7]. However, progress on recent video understanding datasets such"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b4-6db4-11ee-a578-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-1", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "object detection [46] and semantic segmentation [7]. However, progress on recent video understanding datasets such\nas A V A [17] and Charades [49] has lagged behind in comparison. Further progress in the video understanding tasks\nposed by these datasets would facilitate applications in autonomous vehicles, health monitoring and automated media\nanalysis and production among others.\nA reason why video understanding is so challenging is\nbecause, as shown in Fig. 1, it requires understanding the\ninteractions between actors, objects and other context in the\nscene. Furthermore, these interactions are not always observable from a single frame, and thus require reasoning\nover long temporal intervals. This is illustrated in Fig. 1,\nwhere understanding the actions of the person in the centreframe is not possible from the target keyframe alone. In order to know that the woman is \u201clistening\u201d, we need to con-sider the man who is speaking but no longer in the scene.\nAnd to correctly infer that the woman is \u201cdriving\u201d the car,\nrather than \u201criding\u201d like the man, we must take into account\nthat she is later holding the steering wheel.\nVideo is a signi\ufb01cantly higher-dimensional signal than\nsingle images, due to its additional temporal axis, and so we\nbelieve learning these unlabelled interactions directly from\ncurrent datasets with large convolutional networks is not"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b5-6db4-11ee-b1b5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-2", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "single images, due to its additional temporal axis, and so we\nbelieve learning these unlabelled interactions directly from\ncurrent datasets with large convolutional networks is not\nfeasible. In this paper, we propose a structured graph neural\nnetwork to explicitly model these spatio-temporal interactions. We model actors and objects (explicitly with bounding boxes when we have supervision, and implicitly otherwise) as nodes in our spatio-temporal graph and perform\nmessage passing inference to directly model their relations.\nAlthough a wide range of graph-structured models have\nbeen proposed for action recognition, we note that there has\nbeen no unifying formulation for these models. As such,\nsome works only model spatial relations between actors\nand objects [14, 54], but not how these interactions evolve\nover time. Other approaches model long-range temporal\ninteractions [61], but do not capture spatial relations and\nare not trained end-to-end. And whilst some methods do\nmodel spatio-temporal interactions of objects [3, 60], their\nexplicit representations of objects need additional supervision, and are not evaluated on spatio-temporal localisation\ntasks which requires detailed understanding and is necessary for analysing untrimmed videos.\nOur graph network formulation based on the messagepassing neural network [13] abstraction, allows us to explicitly model interactions between actors, objects and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b6-6db4-11ee-9ba8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-3", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Our graph network formulation based on the messagepassing neural network [13] abstraction, allows us to explicitly model interactions between actors, objects and\nthe scene, and how these interactions evolve over time.\nOur \ufb02exible model allows us to use explicit object representations from a pretrained Region Proposal Network\n(RPN) [46], and/or implicitly from convolutional feature\nmaps without additional supervision. Moreover, our general\nformulation allows us to interpret previous work [14, 54, 60,\n61, 71] as special cases, and thus understand how different\ndesign choices in object representation, graph connectivity\nand message passing functions affect the model\u2019s performance. We demonstrate our versatile model on two different tasks: spatio-temporal action detection on A V A [17]arXiv:2103.15662v1  [cs.CV]  29 Mar 2021\nTimeTarget keyframeLabels: listen, drive, sit\u2026\u2026\u2026\u2026Figure 1: Understanding videos requires reasoning over long-term spatio-temporal interactions between actors, objects and the environment. The actions of the woman in the centre frame are ambiguous given the nearby frames which typical 3D CNN architectures consider.\nHowever, by considering her interactions with the man from nearby frames, we know she is \u201clistening\u201d to him. And the fact that she is later"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b7-6db4-11ee-9eda-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-4", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "However, by considering her interactions with the man from nearby frames, we know she is \u201clistening\u201d to him. And the fact that she is later\nholding the steering wheel indicates she is \u201cdriving\u201d in contrast to the man who is \u201criding\u201d. In this paper, we propose a spatio-temporal\ngraph on which we perform message passing to explicitly model these spatio-temporal interactions. Example from the A V A dataset [17].\nand UCF101-24 [53], and video scene graph prediction on\nthe recent Action Genome [23] dataset. Both of these tasks\nrequire modelling the spatio-temporal interactions between\nthe actors and/or objects in the scene, and we show consistent improvements from using each component of our\nmodel and achieve state-of-the-art results on each dataset.\nFurthermore, we observe that the largest improvements\nin A V A are indeed achieved on action classes involving\nhuman-to-human and human-to-object interactions, and visualisations of our network show that it is focusing on scene\ncontext that is intuitively relevant to its action classi\ufb01cation.\n2. Related Work\nModelling contextual relationships has a long history in\nscene understanding. Relevant examples of early works in\nthis area included modelling interactions between humans\nand objects [18, 68], different objects [45] and relationships"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b8-6db4-11ee-b37f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-5", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "scene understanding. Relevant examples of early works in\nthis area included modelling interactions between humans\nand objects [18, 68], different objects [45] and relationships\nbetween human actions and scene context [41, 16]. Furthermore, it has also been shown that human vision is reliant on\ncontext too [43]. In this paper, we consider video understanding tasks, speci\ufb01cally spatio-temporal action recognition and video scene graph parsing, which involve reasoning\nabout interactions between actors, objects and their environment in both space and time.\nEarly work in action recognition used hand-crafted features to encode motion information [32, 58]. Advances\nin deep learning \ufb01rst saw repurposing of 2D image convolutional neural networks (CNNs) for video as \u201ctwo\nstream\u201d networks [27, 50] followed by spatio-temporal 3D\nCNNs [6, 12, 55, 72]. However, these architectures focus on extracting coarse, video-level features and are not\nsuitable for learning the \ufb01ne-grained relations depicted in\nFig. 1. Consequently, whilst initial approaches to spatiotemporal action detection involved extending 2D object detectors [38, 46] temporally [26, 51, 1, 65], current leading\nmethods [14, 61, 71] on the A V A dataset [17] all explicitly"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14b9-6db4-11ee-b024-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-6", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "methods [14, 61, 71] on the A V A dataset [17] all explicitly\nmodel relationships, with approaches which we show can\nbe interpreted as variants of graph neural networks.\nGraph neural networks (GNNs) explicitly model relations between entities by modelling them as nodes in adirected or undirected graph [5, 48, 29], which interact\nvia a neighbourhood de\ufb01ned for each node. The selfattention [56] and Non-local [59] operators can also be\nthought of as GNNs [5, 13] where each element in a feature map is a node, and all nodes are fully-connected to\neach other. Such attention-based models have excelled at\na number of natural language processing and computer vision tasks and have inspired many follow-up methods [21,\n8, 9, 62, 70].\nMany structured models have also recently been employed in video understanding. However, there has previously been no coherent framework to unify these approaches. Consequently, some works only model spatial relations between actors and objects, but not how these evolve\nover time [54, 14]. And whilst LFB [61] models long range\ntemporal interactions, it does not capture spatial relationships within a keyframe. Moreover, in order to model longrange interactions, [61] uses precomputed features and is\nthus not trained end-to-end. Our proposed method, which is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ba-6db4-11ee-b702-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-7", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "thus not trained end-to-end. Our proposed method, which is\nbased on the message passing neural network (MPNN) [13]\nframework, coherently models both spatial and temporal\ninteractions. After describing our model in Sec. 3, we\nshow how previous structured models for action detection [54, 14, 60, 61, 71] can be thought of as special cases\nof our model. Furthermore, our \ufb02exible model can reason\nabout objects both when we have explicit supervision for\nthem, and when we do not. Prior work in video understanding, which have proposed less generic models, have either\nassumed the case of explicit object supervision [3, 60] or\nnot [54, 14, 71], but have not considered the scenario when\nboth options are available. Furthermore, our general formulation allows us to ablate graph modelling design choices,\nsuch as the object representation, message passing functions and temporal context, in a manner not possible with\nthese previous, more speci\ufb01c approaches. Moreover, we\nnote that some approaches which have employed spatiotemporal graphs in video have only considered frame-toframe interactions [22, 60, 3, 40, 42, 44], rather than longrange relations, and have not evaluated on spatio-temporal\nlocalisation which requires more detailed understanding\n3D CNN\nsit, answer phone, carry/hold \nobject"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14bb-6db4-11ee-a647-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-8", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "localisation which requires more detailed understanding\n3D CNN\nsit, answer phone, carry/hold \nobject\nsit, touch (an object), watch (a \nperson), talk (a person)\nsit, touch (an object), watch (a \nperson), listen (a person)\nReadout Function, \n\u0000\nSpatio\ntemporal graph \nconstruction and message passing\nForeground node\nImplicit context\nExplicit contextFigure 2: Overview of our method: We construct a spatio-temporal graph, and perform message-passing inference on it, to model interactions between actors, objects and their environment. Foreground nodes (circles) have readout functions associated with them for the task of\ninterest ( i.e. for action recognition, the nodes represent person bounding boxes which are classi\ufb01ed into actions). Context nodes (squares)\nmodel additional information, and can either be implicit, as cells of the original feature map, or explicit by ROI-pooling external region\nproposals (RPN [46] not shown for clarity). The initial state of each node is a spatio-temporal feature vector extracted from a 3D CNN.\nand is essential for analysing untrimmed videos.\nWe also note that scene graph parsing [25, 30] is another task that evaluates a model\u2019s ability to reason about"}


