# embeddings_queue

created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14bc-6db4-11ee-ae2b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-9", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "and is essential for analysing untrimmed videos.\nWe also note that scene graph parsing [25, 30] is another task that evaluates a model\u2019s ability to reason about\nthe interactions between different objects by representing\nobjects as nodes and relationships as edges in a graph. Although the task was originally posed for single images [30],\nthe recent Action Genome [23] dataset extends this task\nto video by adding annotations to Charades [49]. While\nGNN-based approaches have also been used in scene graph\nparsing [10, 33, 64, 66] for single images, to our knowledge, they have not been applied to model temporal relations in video. Moreover, the same model has not also been\ndemonstrated on spatio-temporal action recognition like our\nmethod.\n3. Proposed Approach\nOur model aims to build a structured representation of\na video by representing it as a graph of actors, objects\nand contextual elements in the scene, as shown in Fig. 2.\nThis structured representation is then used to perform tasks\nwhich require understanding the interactions between the\nelements in the graph, such as action recognition and scene\ngraph prediction. Note that we do not assume we have annotations for relevant scene context.\nOur approach is based on Message Passing Neural Networks (MPNN) [13], as it is a \ufb02exible framework that generalises many previous graph neural network algorithms [4,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14bd-6db4-11ee-a23f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-10", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Our approach is based on Message Passing Neural Networks (MPNN) [13], as it is a \ufb02exible framework that generalises many previous graph neural network algorithms [4,\n5, 29, 57] We review this approach, and describe how we\nadapt it for video understanding in Sec. 3.1, before detailing our model in Sec. 3.2 through 3.5. Finally, we dis-cuss how previous structured models for video understanding [14, 54, 60, 61, 71] can be regarded as speci\ufb01c instantiations of our model in Sec. 3.6.\n3.1. Message Passing Neural Networks (MPNNs)\nMPNNs operate on a directed or undirected graph, G\nconsisting of nodes, v2V, and a neighbourhood for each\nnode,Nv, that de\ufb01nes the graph\u2019s connectivity. For video\nmodels we distinguish between the spatial, Sv, and temporal,Tv, neighbourhoods for node v(Nv=Sv[Tv). Each\nnode,v, is associated with a latent state, hv. Inference in\nthis model consists of a message passing phase, and a \ufb01nal readout phase. In the message passing phase, messages\nfor each node, mv, are \ufb01rst computed by applying spatial"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14be-6db4-11ee-80ac-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-11", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "for each node, mv, are \ufb01rst computed by applying spatial\nand temporal message passing functions, MsandMtrespectively, to all nodes in its neighbourhood as described in\nEq. (1). An update function, U, then aggregates the received\nmessages to update the latent state, hv,\nmi+1\nv=X\nw2SvMs(hi\nv;hi\nw;\u0012i\ns) +X\nu2TvMt(hi\nv;hi\nu;\u0012i\nt)(1)\nhi+1\nv=U(mi+1\nv;hi\nv); (2)\nwhere\u0012denotes learnable function parameters. Intuitively,\nthe state of a node, hv, is updated by aggregating the messages passed from its neighbours. Finally, after I\u00151iterations of message passing, a readout function, R, uses the\nupdated node features for the classi\ufb01cation task of interest\ny=R(fhi\nvgjv2G): (3)\nAs illustrated in Fig. 2, our graph consists of a set of\n\u201cForeground\u201d nodes, F=ff1;f2;:::;fngand \u201cContext\u201d"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14bf-6db4-11ee-8114-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-12", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "\u201cForeground\u201d nodes, F=ff1;f2;:::;fngand \u201cContext\u201d\nnodes,C=fc1;c2;:::;cmgwherenandmvary for each\nvideo. The \u201cForeground\u201d nodes, F, have readout and loss\nfunctions associated with them, and correspond to the objects that will subsequently be classi\ufb01ed. For the task of action recognition,Fcorresponds to bounding boxes of each\nactor in the keyframe. Whilst for scene graph prediction, F\nrefers to bounding boxes for all potential objects of interest. The \u201cContext\u201d nodes, C, capture additional information\nextracted from the scene for relational reasoning. The representations of these nodes, and their spatial connectivity,\nare detailed next.\n3.2. Spatial Model\nThe spatial connections in our graph model relationships between actors, objects and scene context in the same\nframe. For example, recognising an action such as \u201cgive\nobject to a person\u201d in the A V A dataset [17] requires understanding both people involved in the action and also the object being transferred.\nTo model such interactions, we \ufb01rst represent the foreground nodes of our model, F, by extracting convolutional features from the last layer of the network, X2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c0-6db4-11ee-96ef-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-13", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "To model such interactions, we \ufb01rst represent the foreground nodes of our model, F, by extracting convolutional features from the last layer of the network, X2\nRt\u0002h\u0002w\u0002c, and using ROI-Align [19] followed by spatiotemporal pooling and a linear projection to obtain fi2Rd.\nWe model scene context by considering the features from\neach spatial position in the feature map, X, as a contextual\nnode,ci, in our graph. Note that these features are projected\ntoRd. A similar representation was used by [54], which we\nrefer to as an implicit object model as it enables the network\nto encode information about the scene and relevant objects\nwithout any extra supervision. This approach is also known\nas \u201cgrid features\u201d in visual question answering [4, 24].\nIt is also possible to augment our set of contextual nodes\nwith an explicit object representation by computing classagnostic object proposals with a Region Proposal Network\n(RPN) [46]. We use an RPN pretrained on the OpenImages\ndataset [31] and obtain a d-dimensional feature from each\nproposal by using ROI-Align and a linear projection as for\nthe foreground nodes. A similar idea of using explicit object\nrepresentations in video has also been employed by [3] and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c1-6db4-11ee-8865-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-14", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "the foreground nodes. A similar idea of using explicit object\nrepresentations in video has also been employed by [3] and\n[60], though not for spatio-temporal action recognition.\nMessages are then computed and passed to foreground\nnodes from both foreground and contextual nodes in the\ngraph. We only update the foreground nodes during message passing, as these are the nodes which are subsequently\nclassi\ufb01ed in the \ufb01nal Readout phase. Concretely, the spatial\nneighbourhood for each foreground node in our graph is\nSv=F[C8v2F: (4)\nContextForegroundFrames\n\u26ab\n\u26ab\n\u26ab\ud835\udf0f\"\ud835\udf0f#Figure 3: Illustration of temporal connectivity. Foreground (blue)\nand context (green) nodes have initial states computed from spatiotemporal features over \u001ctframes (shaded in yellow), centred on\na keyframe (denoted by a black circle). \u001cscontrols the distance\nbetween selected keyframes; \u001ccis the total number of keyframes\nconsidered. Here, \u001cc= 3,\u001cs= 7frames and\u001ct= 5 frames.\n3.3. Temporal Model\nWe also include temporal connections in our graph to\nmodel long-range interactions between actors and objects."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c2-6db4-11ee-8c7f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-15", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "3.3. Temporal Model\nWe also include temporal connections in our graph to\nmodel long-range interactions between actors and objects.\nAs shown in Fig. 1, understanding actions often requires\nreasoning about actors who are no longer visible in the current frame, thus requiring large temporal contexts.\nWe model these temporal interactions by connecting\nforeground nodes in keyframe twith all other foreground\nnodes in neighbouring keyframes t02T . Concretely, we\nde\ufb01ne the temporal context, \u001cc, and temporal stride, \u001cs, hyperparameters. As shown in Fig. 3, \u001ccis the total number of\nnumber of keyframes in the video which we consider in our\ntemporal graph. \u001csis the sampling rate at which we select\nkeyframes, as \u001cs\u00151allows us to consider a wider temporal\ninterval in a more computationally ef\ufb01cient manner. This is\nnecessary to train the entire model end-to-end. Moreover,\nas each foreground feature node in the graph, fi, is itself\na spatio-temporal feature computed over a period of \u001ctby\na 3D CNN, selecting adjacent keyframes (and effectively\nsetting\u001cs= 1 keyframe) could result in redundant information being captured by the temporal connections in the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c3-6db4-11ee-8fc4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-16", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "a 3D CNN, selecting adjacent keyframes (and effectively\nsetting\u001cs= 1 keyframe) could result in redundant information being captured by the temporal connections in the\ngraph. Note that our de\ufb01nition of keyframe follows common datasets [17, 23] as a frame we aim to classify given\nsurrounding temporal context.\nMore formally, we can describe the neighbourhood of\neach foreground node, v, for temporal message passing as\nTv=b\u001cc=2c[\ndt=\u0000\u001cc=2eFt\u0001\u001cs: (5)\nHere, we use the superscript to denote the temporal index,\nand without loss of generality consider t= 0 to be the\ncentre keyframe, meaning that negative time indices correspond to frames in the past. We set \u001ccto be an odd, positive\ninteger to employ an equal-sized temporal window on either\nside of the centre keyframe.\nWe \ufb01rst perform spatial message passing, before passing messages temporally. This allows information from the\ncontext nodes in frame t0,ct0\nj, to ef\ufb01ciently propagate to a\nforeground node ft\ni(wheret6=t0) viaft0\nj, as the foreground\nnodes are fully-connected temporally.\nThe following section now describes the messages that\nare passed along the graph described above."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c4-6db4-11ee-a797-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-17", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "j, as the foreground\nnodes are fully-connected temporally.\nThe following section now describes the messages that\nare passed along the graph described above.\n3.4. Message Passing Functions\nWe \ufb01rst observe that the Non-local operator or selfattention [56, 59] can be considered as a message passing\nfunction in a fully-connected graph, where each node, hv,\nis an element in the input feature map, H, and the neighbourhood comprises of all other feature elements. And\nsince self-attention is employed with a residual connection [59, 56] and layer normalisation [2, 56], the update\nfunction of Non-local when viewed as an MPNN is\nU=LN(hv+Self-Attention (H)): (6)\nSimilar analysis [5, 13] has shown that Graph Attention\nNetworks (GAT) [57], Relational Networks [47] and many\nother graph neural networks [48, 29, 35, 4] can also be interpreted as MPNNs. In this paper, we consider Non-local [59]\nand Graph Attention (GAT) [57] as message passing functions within the update function, Eq. (6).\nNon-local We modify Non-local [59] to pass messages\nfrom all Foreground and Context nodes to only Foreground"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c5-6db4-11ee-adc6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-18", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Non-local We modify Non-local [59] to pass messages\nfrom all Foreground and Context nodes to only Foreground\nnodes (Eq. 4), as these are the nodes which are used for the\n\ufb01nal classi\ufb01cation in the Readout phase,\nM=Softmax\u0012QK>\np\nd\u0013\nV (7)\nQ=AWqK= [AjjC]WkV= [AjjC]Wv:\nHere,A2Rn\u0002dandC2Rm\u0002dare matrices where each\nrow is a foreground and context feature node respectively,\nMis theRn\u0002dmatrix of messages received by each node in\nA,[AjjC]denotes the concatenation of these matrices and\nWq,WkandWvare learnable d\u0002dprojection matrices.\nGraph attention (GAT) The graph attention [57] message for a node vis computed as\nmv=\u001b0\n@X\nj2Nv\u000bijWahj1\nA (8)\n\u000bij=Softmax\u0000\n\u001b\u0000\nw>\nb[hijjhj]\u0001\u0001\n(9)\nwhere\u001bis a ReLU non-linearity and Waandwbare a\nlearnable matrix and vector respectively."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c6-6db4-11ee-b9c8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-19", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "b[hijjhj]\u0001\u0001\n(9)\nwhere\u001bis a ReLU non-linearity and Waandwbare a\nlearnable matrix and vector respectively.\nParallel messages It is also possible to compute multiple\nincoming messages in parallel for a node, hv. When usingmultiple Non-local or Graph Attention functions, this corresponds to multi-headed attention [56, 57]. By viewing Nonlocal and GAT as message passing functions, we can also\naggregate messages from a combination of these two. In\nthese cases, we aggregate the messages using an attentionweighted convex combination, as performed in Eq. (9).\n3.5. Readout Function\nAfterI\u00151iterations of message passing, a readout\nfunction is applied on the Foreground nodes to obtain the\n\ufb01nal predictions. For action detection, the readout function is a linear classi\ufb01er operating on each element of F,\nwhere each foreground node, fi, corresponds to the features\nof an actor in the keyframe. For scene graph prediction, the\nreadout function consists of two classi\ufb01ers: The \ufb01rst linear\nclassi\ufb01er predicts the object-class label of each foreground\nnode. The second is a function of each pair of foreground\nnodes, and predicts the relationship label between them.\n3.6. Discussion"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c7-6db4-11ee-9b9c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-20", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "node. The second is a function of each pair of foreground\nnodes, and predicts the relationship label between them.\n3.6. Discussion\nWe note that many previous structured models for video\nunderstanding can be considered as special cases of our proposed MPNN framework:\nGirdhar et al . [14] only consider a spatial model, i.e.\nN=S, using implicit objects and Non-local [60] for\nmessage passing. ACRN [54] has the same graph structure, but uses Relational Networks [47] for message passing instead. LFB [61], in contrast, considers only a temporal model, i.e.N=T, using Non-local as the message passing function in a graph that is fully-connected in\ntime. However, [61] do not consider a spatial model to capture interactions between actors in the keyframe. Zhang et\nal. [71], on the other hand, model both temporal and spatial connections. However, they effectively model three\nseparate graphs: the \ufb01rst models actors in short, 3-second\ntubelets using GCN [29] for message passing. The other\ntwo graphs model actor-actor and actor-object relations using a message passing method similar to GAT [57], but using a hand-de\ufb01ned weighting function rather than a learned"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c8-6db4-11ee-a69a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-21", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "two graphs model actor-actor and actor-object relations using a message passing method similar to GAT [57], but using a hand-de\ufb01ned weighting function rather than a learned\none as in GAT (Eq. 9). Wang et al. [60] also model a spatiotemporal graph, using GCN for message passing, and an explicit object representation in the spatial model. However,\ntheir temporal connections are only among adjacent frames,\nwhich does not allow information propagation between all\nframes in long sequences, in contrast to our model which\nis fully-connected temporally. With our uni\ufb01ed framework,\nwe study the effect of graph modelling design choices and\nshow how we outperform previous work next in Sec. 4.\n4. Experiments\nWe evaluate our method\u2019s ability to model spatiotemporal interactions with experiments on spatio-temporal\naction detection and video scene graph classi\ufb01cation.\nTable 1: Ablation study on Action Genome using a 3D ResNet 50 backbone. We report the effect of (a) different message passing functions,\n(b) temporal connections in the graph ( \u001cc= 1;\u001cs= 1corresponds to only spatial message passing) and (c) iterations of message passing.\n(a) Spatial message passing functions\nSGCls PredCls"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14c9-6db4-11ee-92e5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-22", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "(a) Spatial message passing functions\nSGCls PredCls\nR@20 R@50 R@10 R@20\nBaseline 48.9 51.3 78.7 93.8\nNon-local in backbone 49.1 51.4 78.8 93.9\nNon-local 50.4 52.6 79.3 94.2\nGAT 51.1 53.2 79.7 94.4\nGAT + Non-local 51.3 53.4 79.4 94.3(b) Temporal message passing\nTemporal parameters SGCls PredCls\n\u001cc\u001cs R@20 R@50 R@10 R@20\n1 1 51.1 53.2 79.7 94.4\n3 2 52.9 55.0 79.4 94.2\n3 5 53.3 55.5 79.4 94.2\n3 7 53.5 55.7 79.3 94.2\n5 2 53.4 55.5 79.4 94.2\n5 5 53.8 56.0 79.3 94.2\n5 7 53.6 55.8 79.3 94.2(c) Message passing iterations\nIterations R@20\n1 51.1\n2 51.6\n3 51.6\n5 51.8\nSpatio-temporal action recognition We evaluate on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ca-6db4-11ee-bf4a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-23", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Iterations R@20\n1 51.1\n2 51.6\n3 51.6\n5 51.8\nSpatio-temporal action recognition We evaluate on\nA V A [17], the largest dataset for this task consisting of\n15-minute video clips obtained from movies, and UCF10124 [53], the previous standard benchmark for this task. A V A\nis labelled with atomic actions, where one person is typically performing multiple actions simultaneously, whilst actors perform only a single high-level action in UCF101-24.\nWe follow standard protocol and evaluate using the Frame\nAP at an IoU threshold of 0.5 on both datasets. For A V A, we\nuse v2.2 annotations for ablations, and either v2.1 or v2.2\nannotations for fair comparisons with prior work.\nVideo scene graph prediction We evaluate on the recent Action Genome dataset [23], which adds scene graph\nannotations to Charades [49], on two scene graph tasks:\nscene graph classi\ufb01cation (SGCls) and predicate classi\ufb01cation (PredCls). Both tasks are evaluated using the standard\nRecall@K metric [39, 23] (R@K) which measures the fraction of ground truth relationship triplets (subject-predicateobject) that appear in the top K scoring predicted triplets. In"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14cb-6db4-11ee-be2b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-24", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "SGCls, ground truth bounding box co-ordinates are given,\nand the aim is to predict their object classes, as well as relationship labels between pairs of objects. PredCls is simpler,\nas both bounding box co-ordinates and object classes are\ngiven, and only the relationship label must be predicted.\nImplementation details We use the public implementation of SlowFast [11] as our baseline with the 3D ResNet\n50 or ResNet 101 backbones [20], as it is the current stateof-the-art. The network is similar to Fast-RCNN [15] as it\nuses external region proposals to extract features from the\nlast feature map of res 5using ROI-Align [19]. These features are then spatio-temporally pooled and classi\ufb01ed.\nIn our graph model, we initialise the internal states, hi,\nof our Foreground nodes using these ROI-Aligned res 5features. For our action detection experiments, the Foreground\nnodes in our graph correspond to bounding boxes of actors,\nand we use the same person detector as [11, 61] for our actor region proposals. For scene graph experiments, we use\nground truth boxes of people and objects as our Foreground\nnodes as we evaluate scene graph classi\ufb01cation (SGCls) and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14cc-6db4-11ee-b539-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-25", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "ground truth boxes of people and objects as our Foreground\nnodes as we evaluate scene graph classi\ufb01cation (SGCls) and\npredicate classi\ufb01cation (PredCls).Unless otherwise speci\ufb01ed, we use SlowFast 8\u00028which\ncorresponds to 32 input frames where the video is subsampled by a factor of 2. This means that each feature node,\nhi, in the graph aggregates \u001ct= 2:1seconds of temporal\ninformation on A V A as the videos are sampled at 30 frames\nper second (fps). As keyframes in A V A are de\ufb01ned at 1 fps,\nit means that we need to set \u001cs\u00152keyframes for the temporal information captured by temporally-adjacent nodes in\nthe graph to not overlap. For Action Genome, the Charades\nvideos are sampled at 24 fps and the keyframes are on average 0.85 seconds, or 20.5 frames apart.\nWe train our network for 20 epochs using synchronous\nSGD on 8 GPUs and a total batch size of 64, initialising from a Kinetics-400 [28] pretrained model. The baseline model in all our experiments is SlowFast without any\ngraph module trained in an identical manner. When training\nspatio-temporal graph models, we \ufb01netune for 10 epochs"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14cd-6db4-11ee-a47b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-26", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "graph module trained in an identical manner. When training\nspatio-temporal graph models, we \ufb01netune for 10 epochs\nfrom a model trained with only a spatial graph and reduce\nthe batch size by a factor of \u001cc. We also \u201cfreeze\u201d batch\nnormalisation statistics as they have high variance for small\nbatches [19]. Full training details are in the appendix.\n4.1. Experiments on video scene graph prediction\nSpatial message passing We \ufb01rst ablate only the spatial component (Sec. 3.2) of our model in Tab. 1a. Graph\nAttention (GAT) [57] performs slightly better than Nonlocal [56, 59] as a message passing function, and we obtain\na further small improvement by combining the two methods\nin parallel (Sec. 3.4). Overall, we improve upon our baseline model, which is SlowFast based on ResNet 50 without\nany graph modelling, by 2.4 points for R@20 on SGCls.\nAnother baseline is to insert a Non-Local layer [59] after the \ufb01nal res 5layer and before the ROI-Align layer, since\nNon-local can also be viewed as a graph network (Sec. 3.4)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ce-6db4-11ee-a575-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-27", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Non-local can also be viewed as a graph network (Sec. 3.4).\nIn this case, the Foreground and Context nodes are not explicitly modelled as in our method and the overall performance is 2.2 points less than our method for the R@20 of\nSGCls. This suggests that explicit modelling of the Foreground nodes which are subsequently classi\ufb01ed is important to performance. We note, however, that Non-Local [59]\nis typically employed earlier in a network ( i.e. in res 3or\nres4[59, 61, 11]) and can thus be seen as a complementary\nTable 2: Comparison to existing methods on the Action Genome\ndataset [23]. Previous methods reported by [23].\nSGCls PredCls\nR@20 R@50 R@10 R@20\nMSDN [34] 44.0 47.2 \u2013 \u2013\nIMP [64] 44.1 47.4 \u2013 \u2013\nRelDN [69] 46.7 49.4 \u2013 \u2013\nSlowFast (ResNet 50) 48.9 51.3 78.7 93.8\nOurs (ResNet 50) 53.8 56.0 79.3 94.2\npersonsofacouch\ntable\nlaptopnot looking atnot contacting\nnot contactingnot looking at\nnot contacting\nin front of\nlooking aton the side of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14cf-6db4-11ee-9f40-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-28", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "personsofacouch\ntable\nlaptopnot looking atnot contacting\nnot contactingnot looking at\nnot contacting\nin front of\nlooking aton the side of\nin front of\nperson\nsofacouchtelevision\nlaptop\ntable\nnot looking atnot contactingin front of\nnot contacting\nbehindlooking at\nnot looking atbeneathlooking at\nin front of\nsitting on\non the side ofabove\ntouching\nFigure 4: Example scene graph predictions for consecutive\nkeyframes of Action Genome [23]. Classi\ufb01ed bounding boxes are\non top, and the corresponding scene graph (objects are shown as\ncoloured rectangles, and relations with light-blue ovals) below.\nmethod to improve features learned by the network.\nTemporal message passing Table 1b adds temporal connections (Sec. 3.3) to our model, using GAT for message\npassing, as it outperformed Non-local in Tab. 1a. We observe consistent improvements for a wide range of temporal\ncontexts,\u001cc, and strides, \u001cs, showing the utility of modelling\ntemporal dynamics. We improve over a graph with only\nspatial connections by as much as 2.7 points for R@20.\nFor a temporal model, another baseline to consider is\nto simply increase the temporal information available to\nthe spatial-only graph model by increasing the number of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d0-6db4-11ee-ae21-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-29", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "For a temporal model, another baseline to consider is\nto simply increase the temporal information available to\nthe spatial-only graph model by increasing the number of\nframes at the input. When tripling the number of input\nframes, the R@20 is 52.4, less than all variants of our temporal model with \u001cc= 3. Similarly, increasing input frames\nby a factor of 5, the R@20 is 52.8, less than all our temporal models with \u001cc= 5. This improvement demonstrates the\nbene\ufb01ts of message passing on an explicit temporal graph.\nIterations of message passing Table 1c shows that the\nmodel\u2019s performance plateaus after 2 iterations of message\npassing, where parameters are not tied across iterations.\nComparison to state-of-the-art Finally, we compare to\nprior works in Tab. 2 which are single-image models evalu-Table 3: Analysis of message passing neighbourhood on A V A using SlowFast with a ResNet-50 backbone as the baseline. By varying the neighbourhood, we study the effect of object representations (implicit, explicit or none) and temporal connections. We\nreport the Frame mAP on the three types of action classes in A V A.\nMessage passing neighbourhood PoseHumanHumanHumanObjectAll\nSlowFast baseline (none) 43.1 25.2 17.4 24.8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d1-6db4-11ee-9969-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-30", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Message passing neighbourhood PoseHumanHumanHumanObjectAll\nSlowFast baseline (none) 43.1 25.2 17.4 24.8\nActors only 43.2 27.0 17.8 25.6\nImplicit objects only 43.4 26.7 18.0 25.7\nExplicit objects only 43.0 26.7 17.8 25.5\nActor + Implicit 43.4 26.8 18.3 25.9\nActor + Implicit + Explicit 43.7 27.0 18.4 26.1\nSpatio-temporal 43.8 27.5 19.9 27.0\nated by [23]. Our SlowFast, ResNet 50 3D baseline outperforms these, showing the importance of using spatiotemporal features for this task. As Action Genome contains\nvideos of humans acting to scripts, there is temporal structure in the interactions between actors and objects in the\nscene (Fig. 4). Our \ufb01nal spatio-temporal graph structured\nmodel improves substantially upon this baseline by 4.9 and\n4.7 points for the R@20 and R@50 for SGCls respectively.\nOur improvements over the baseline for PredCls are less, as\nthis task is easier and the performance is saturated.\n4.2. Experiments on action recognition\nGraph structure and object representation Table 3\ncompares the effect of changing the neighbourhood that"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d2-6db4-11ee-916d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-31", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "this task is easier and the performance is saturated.\n4.2. Experiments on action recognition\nGraph structure and object representation Table 3\ncompares the effect of changing the neighbourhood that\npasses messages to each foreground node that is subsequently classi\ufb01ed. We report the performance across the\nthree types of action categories in A V A [17] \u2013 Pose, HumanHuman and Human-Object \u2013 to show the effect that different object representations (implicit, explicit or neither as\ndescribed in Sec. 3.2) and also temporal connections have.\nOur SlowFast baseline does not model any explicit graph\nand hence effectively performs no message passing. For\nour A V A experiments, Foreground nodes in the graph correspond to bounding boxes of the actors in the scene, using\nthe same person detections as [61, 11]. Thus, in this section,\n\u201cForeground\u201d and \u201cactor\u201d nodes are used interchangeably.\nWe observe from the second row of Tab. 3 that passing messages only between actor nodes (and thus not modelling object interactions) provides an overall improvement\nof 0.8 points. The largest gain, as expected, are for HumanHuman action classes, as these are the interactions modelled by passing messages between actor nodes in the graph.\nWhen passing messages from implicit context nodes to\nthe actors (third row), we observe an improvement in all"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d3-6db4-11ee-b438-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-32", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "When passing messages from implicit context nodes to\nthe actors (third row), we observe an improvement in all\ntypes of action classes, but primarily in Human-Human and\nHuman-Object classes. This is because implicit context\nnodes encompass the entire feature map, and thus capture\ninformation about the whole scene (as also shown in Fig. 5).\nModelling objects explicitly with external regional proTable 4: Comparison on A V A. We report the Mean AP using v2.1\nand v2.2 annotations. All methods pretrained on Kinetics 400.\n\u201cMultiscale\u201d refers to averaging results over three scales [61].\nMethod v2.1 v2.2\nACRN (S3D) [54] 17.4 \u2013\nZhang et al. (R50) [71] 22.2 \u2013\nSlowFast baseline (R50) 24.5 24.8\nGirdhar et al. (I3D) [14] 25.0 \u2013\nLFB (R50) [61] 25.8 \u2013\nOurs (R50) 26.5 27.0\nOurs Multiscale (R50) 27.3 27.7\nSlowFast baseline (R101) 26.3 26.7\nLFB (R101) [61] 26.8 \u2013\nOurs (R101) 28.3 28.8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d4-6db4-11ee-b76b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-33", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "SlowFast baseline (R101) 26.3 26.7\nLFB (R101) [61] 26.8 \u2013\nOurs (R101) 28.3 28.8\nLFB Multiscale (R101) [61] 27.7 \u2013\nOurs Multiscale (R101) 29.5 30.0\nposals (fourth row) improves on the same action types as the\nimplicit object model, but performs marginally worse. This\nsuggests that our RPN trained on OpenImages [31] is unable\nto detect the objects that are most discriminative of A V A actions. As object annotations are not provided in A V A, it is\nnot possible to evaluate the recall of our external region proposals on relevant objects. It is not clear if we should expect\nmore improvement from explicit objects, because although\nprevious works have considered implicit [14, 54, 71] and\nexplicit representations [3, 60], we are not aware of any that\nhave compared the two. Note that these explicit context\nnodes also model Human-Human interactions as our RPN\nis trained on people in OpenImages [31]. We obtain further\nimprovements by combining messages from actors, implicit\nand explicit context nodes, as shown by the next two rows.\nFinally, we evaluate our spatio-temporal model, which\npasses messages from actors, implicit- and explicit-context"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d5-6db4-11ee-9665-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-34", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "and explicit context nodes, as shown by the next two rows.\nFinally, we evaluate our spatio-temporal model, which\npasses messages from actors, implicit- and explicit-context\nnodes with\u001cc= 3and\u001cs= 3, corresponding to an overall\ntemporal window of 8.5 seconds. This model performs the\nbest, improving upon the baseline by 2.2 points, or a relative\nimprovement of 8.9%. In particular, the temporal connections help to improve on Human-Object action classes.\nA V A is a long-tailed dataset, and the most common failure modes of all variants of our model are the classes with\nfew training examples. However, our model improves on\nboth head and tail classes with respect to the baseline. This,\nand detailed per-class results, are in the appendix.\nState-of-the-art-comparison on A V A Table 4 compares\nto recent, published work on A V A. Our method builds upon\nSlowFast [11] as the base architecture, and our graph model\nshows substantial improvements with either a 3D ResNet 50\nor ResNet 101 backbone. As discussed in Sec. 3.6, Girdharet al. [14] and LFB [61] can be considered as special"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d6-6db4-11ee-b2f1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-35", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "or ResNet 101 backbone. As discussed in Sec. 3.6, Girdharet al. [14] and LFB [61] can be considered as special\ncases of our graph model as they model only spatial andTable 5: Comparison to state-of-the-art on UCF101-24. We report\nthe Frame AP at 0.5 using the corrected annotations of [51].\nMethod Modality Mean AP\nACT [26] RGB + Flow 69.5\nSong et al. [52] RGB + Flow 72.1\nSTEP [67] RGB + Flow 75.0\nGuet al. [17] RGB + Flow 76.3\nMOC [36] RGB + Flow 78.0\nSlowFast (R50) RGB 76.6\nSlowFast (R101) RGB 77.4\nOurs (R50) RGB 78.6\nOurs (R101) RGB 79.3\ntemporal edges respectively. Our method, which constructs\na spatio-temporal graph, outperforms both. Note that we\noutperform LFB [61] across both ResNet 50 and ResNet\n101 backbones. Zhang et al . [71] also model a spatiotemporal interactions, but employ three separate graphs\nwith a hand-crafted aggregation function which we also outperform with our single, coherent spatio-temporal graph."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d7-6db4-11ee-9cb1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-36", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "with a hand-crafted aggregation function which we also outperform with our single, coherent spatio-temporal graph.\nState-of-the-art comparison on UCF101-24 Table 5\nshows that we outperform recent, published work on\nUCF101-24 [53], using either a 3D ResNet 50 or ResNet\n101 backbone. We do so without using optical \ufb02ow as an\nadditional input modality, showing that our network can\ncapture temporal information without it. Moreover, the improvements obtained from our graph model are consistent\nwith our results on the Action Genome and A V A datasets.\nQualitative results We visualise the spatial and temporal messages received by an actor node in the graph, on the\nA V A dataset, when using GAT as the message passing function. In particular, we visualise the attention weights, \u000bijin\n(9), on each implicit object node (Fig. 5) and actor in neighbouring keyframes (Fig. 6), for a given actor. We observe\nthat the network places more weight on graph nodes that are\nintuitively consistent with the \ufb01nal action prediction. For\nexample, in Fig. 5, the model focuses on the actor\u2019s face\nwhen his action is \u201ctalk to\u201d and the other person\u2019s face when\nhis action is \u201clisten to\u201d."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d8-6db4-11ee-aade-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-37", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "when his action is \u201ctalk to\u201d and the other person\u2019s face when\nhis action is \u201clisten to\u201d.\n5. Conclusion and Future Work\nWe have presented a novel spatio-temporal graph neural network framework to explicitly model interactions between actors, objects and their environment. Our formulation can model objects either implicitly or explicitly, and\ngeneralises previous structured models for video understanding [14, 54, 61, 71]. Using our versatile approach, we\nhave achieved state-of-the-art results on two diverse tasks\nacross three datasets. Future work remains to harness explicit object representations more effectively on A V A.\n(a)talk to (0.6), sit (0.92), watch person (0.81) (b)listen to person (0.9), watch person (0.89), sit (0.97) (c)carry/hold (1.0), sit(0.88)\nFigure 5: Visualisation of spatial message passing when using implicit objects as detailed in Sec. 4.2. Note in (a) and (b) that the network\nfocuses on the selected actor (denoted by the green box on the left) when the action is \u201ctalk to\u201d, and on the person speaking to him when"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14d9-6db4-11ee-b781-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-38", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "focuses on the selected actor (denoted by the green box on the left) when the action is \u201ctalk to\u201d, and on the person speaking to him when\nthe action is \u201clisten to person\u201d. Attention weights, \u000bijare colour-coded according to the bar on the far right. Best viewed in colour.\ntalk to (0.85), watch a person (0.82), sit(0.51)\nFigure 6: Visualisation of temporal graph as detailed in Sec. 4.2. We show the neighbours for the actor depicted by the red box in the\ncentre keyframe, colour-coding the respective boxes with their attention weights ( \u000bij). Predicted action scores of the red actor are below,\nand selected keyframes are \u001cs= 2:1seconds apart. Notice that the model focuses on the people 2 keyframes (4.2 seconds) on either side\nof the actor to recognise that she is \u201ctalking to\u201d and \u201cwatching\u201d them.\nReferences\n[1] Anurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia\nSchmid. Uncertainty-aware weakly supervised action detection from untrimmed videos. In ECCV , 2020. 2\n[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14da-6db4-11ee-bf42-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-39", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.\nLayer normalization. In arXiv preprint arXiv:1607.06450 ,\n2016. 5\n[3] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien\nMille, and Greg Mori. Object level visual reasoning in\nvideos. In ECCV , 2018. 1, 2, 4, 8\n[4] Peter Battaglia, Razvan Pascanu, Matthew Lai, and\nDanilo Jimenez Rezende. Interaction networks for learning\nabout objects, relations and physics. In NeurIPS , 2016. 3, 4,\n5\n[5] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases,\ndeep learning, and graph networks. In arXiv preprint\narXiv:1806.01261 , 2018. 2, 3, 5\n[6] Joao Carreira and Andrew Zisserman. Quo vadis, action\nrecognition? a new model and the kinetics dataset. In CVPR ,\n2017. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14db-6db4-11ee-9e18-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-40", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "recognition? a new model and the kinetics dataset. In CVPR ,\n2017. 2\n[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian\nSchroff, and Hartwig Adam. Encoder-decoder with atrous\nseparable convolution for semantic image segmentation. In\nECCV , 2018. 1\n[8] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng\nYan, and Jiashi Feng. A\u02c6 2-nets: Double attention networks.\nInNeurIPS , 2018. 2\n[9] Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan\nShuicheng, Jiashi Feng, and Yannis Kalantidis. Graph-based\nglobal reasoning networks. In CVPR , 2019. 2\n[10] Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In CVPR , 2017.\n3[11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and\nKaiming He. Slowfast networks for video recognition. In\nICCV , 2019. 6, 7, 8, 12\n[12] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14dc-6db4-11ee-8ae5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-41", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "ICCV , 2019. 6, 7, 8, 12\n[12] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes.\nSpatiotemporal residual networks for video action recognition. In NeurIPS , 2016. 2\n[13] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol\nVinyals, and George E Dahl. Neural message passing for\nquantum chemistry. In ICML , 2017. 1, 2, 3, 5\n[14] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In CVPR , 2019.\n1, 2, 3, 5, 8\n[15] Ross Girshick. Fast r-cnn. In ICCV , 2015. 6\n[16] Georgia Gkioxari, Ross Girshick, and Jitendra Malik. Contextual action recognition with r* cnn. In ICCV , 2015. 2\n[17] Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan,\nGeorge Toderici, Susanna Ricco, Rahul Sukthankar, et al.\nAva: A video dataset of spatio-temporally localized atomic"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14dd-6db4-11ee-8762-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-42", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "George Toderici, Susanna Ricco, Rahul Sukthankar, et al.\nAva: A video dataset of spatio-temporally localized atomic\nvisual actions. In CVPR , 2018. 1, 2, 4, 6, 7, 8, 12\n[18] Abhinav Gupta, Aniruddha Kembhavi, and Larry S Davis.\nObserving human-object interactions: Using spatial and\nfunctional compatibility for recognition. PAMI , 31(10),\n2009. 2\n[19] Kaiming He, Georgia Gkioxari, Piotr Doll \u00b4ar, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 4, 6\n[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\nDeep residual learning for image recognition. In CVPR ,\n2016. 1, 6\n[21] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR , 2018. 2\n[22] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh\nSaxena. Structural-rnn: Deep learning on spatio-temporal\ngraphs. In CVPR , 2016. 2\n[23] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14de-6db4-11ee-be29-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-43", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "graphs. In CVPR , 2016. 2\n[23] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos\nNiebles. Action genome: Actions as compositions of spatiotemporal scene graphs. In CVPR , 2020. 2, 3, 4, 6, 7, 12\n[24] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik LearnedMiller, and Xinlei Chen. In defense of grid features for visual\nquestion answering. In CVPR , 2020. 4\n[25] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li,\nDavid Shamma, Michael Bernstein, and Li Fei-Fei. Image\nretrieval using scene graphs. In CVPR , 2015. 3\n[26] Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari,\nand Cordelia Schmid. Action tubelet detector for spatiotemporal action localization. In ICCV , 2017. 2, 8\n[27] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas\nLeung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video\nclassi\ufb01cation with convolutional neural networks. In CVPR ,\n2014. 2\n[28] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14df-6db4-11ee-a3c3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-44", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "2014. 2\n[28] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,\nChloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola,\nTim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. In arXiv preprint\narXiv:1705.06950 , 2017. 6\n[29] Thomas N Kipf and Max Welling. Semi-supervised classi\ufb01cation with graph convolutional networks. In ICLR , 2017. 2,\n3, 5\n[30] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson,\nKenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome:\nConnecting language and vision using crowdsourced dense\nimage annotations. IJCV , 123(1), 2017. 3\n[31] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e0-6db4-11ee-a316-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-45", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Uni\ufb01ed image classi\ufb01cation, object detection, and visual relationship detection at scale. IJCV , 2020.\n4, 8\n[32] Ivan Laptev. On space-time interest points. IJCV , 64(2-3),\n2005. 2\n[33] Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao\u2019ou\nTang. Vip-cnn: Visual phrase guided convolutional neural\nnetwork. In CVPR , 2017. 3\n[34] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases\nand region captions. In ICCV , 2017. 7\n[35] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard\nZemel. Gated graph sequence neural networks. In ICLR ,\n2016. 5\n[36] Yixuan Li, Zixu Wang, Limin Wang, and Gangshan Wu. Actions as moving points. In ECCV , 2020. 8\n[37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e1-6db4-11ee-9962-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-46", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "[37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,\nPietro Perona, Deva Ramanan, Piotr Doll \u00b4ar, and C Lawrence\nZitnick. Microsoft coco: Common objects in context. In\nECCV , 2014. 12\n[38] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian\nSzegedy, Scott Reed, Cheng-Yang Fu, and Alexander C\nBerg. Ssd: Single shot multibox detector. In ECCV , 2016. 2\n[39] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li FeiFei. Visual relationship detection with language priors. In\nECCV , 2016. 6\n[40] Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan\nAlRegib, and Hans Peter Graf. Attend and interact: Higher-order object interactions for video understanding. In CVPR ,\n2018. 2\n[41] Marcin Marszalek, Ivan Laptev, and Cordelia Schmid. Actions in Context. In CVPR , 2009. 2\n[42] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e2-6db4-11ee-97f2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-47", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "[42] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu,\nXiaolong Wang, and Trevor Darrell. Something-else: Compositional action recognition with spatial-temporal interaction networks. In CVPR , 2020. 2\n[43] Aude Oliva and Antonio Torralba. The role of context in object recognition. Trends in cognitive sciences , 11(12), 2007.\n2\n[44] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen,\nand Song-Chun Zhu. Learning human-object interactions by\ngraph parsing neural networks. In ECCV , 2018. 2\n[45] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos,\nEric Wiewiora, and Serge Belongie. Objects in context. In\nICCV , 2007. 2\n[46] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.\nFaster r-cnn: Towards real-time object detection with region\nproposal networks. In NeurIPS , 2015. 1, 2, 3, 4, 12\n[47] Adam Santoro, David Raposo, David G Barrett, Mateusz\nMalinowski, Razvan Pascanu, Peter Battaglia, and Timothy"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e3-6db4-11ee-a91b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-48", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "[47] Adam Santoro, David Raposo, David G Barrett, Mateusz\nMalinowski, Razvan Pascanu, Peter Battaglia, and Timothy\nLillicrap. A simple neural network module for relational reasoning. In NeurIPS , 2017. 5\n[48] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 20(1),\n2008. 2, 5\n[49] Gunnar A. Sigurdsson, G. Varol, X. Wang, Ali Farhadi, I.\nLaptev, and A. Gupta. Hollywood in homes: Crowdsourcing\ndata collection for activity understanding. In ECCV , 2016.\n1, 3, 6\n[50] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In\nNeurIPS , 2014. 2\n[51] Gurkirt Singh, Suman Saha, Michael Sapienza, Philip HS\nTorr, and Fabio Cuzzolin. Online real-time multiple spatiotemporal action localisation and prediction. In ICCV ,\n2017. 2, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e4-6db4-11ee-8345-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-49", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "Torr, and Fabio Cuzzolin. Online real-time multiple spatiotemporal action localisation and prediction. In ICCV ,\n2017. 2, 8\n[52] Lin Song, Shiwei Zhang, Gang Yu, and Hongbin Sun. Tacnet: Transition-aware context network for spatio-temporal\naction detection. In CVPR , 2019. 8\n[53] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.\nUcf101: A dataset of 101 human actions classes from videos\nin the wild. In arXiv preprint arXiv:1212.0402 , 2012. 2, 6,\n8, 12\n[54] Chen Sun, Abhinav Shrivastava, Carl V ondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric\nrelation network. In ECCV , 2018. 1, 2, 3, 4, 5, 8\n[55] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann\nLeCun, and Manohar Paluri. A closer look at spatiotemporal\nconvolutions for action recognition. In CVPR , 2018. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e5-6db4-11ee-a233-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-50", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "LeCun, and Manohar Paluri. A closer look at spatiotemporal\nconvolutions for action recognition. In CVPR , 2018. 2\n[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia\nPolosukhin. Attention is all you need. In NeurIPS , 2017. 2,\n5, 6, 13\n[57] Petar Veli \u02c7ckovi \u00b4c, Guillem Cucurull, Arantxa Casanova,\nAdriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR , 2018. 3, 5, 6, 12\n[58] Heng Wang, Alexander Kl \u00a8aser, Cordelia Schmid, and\nCheng-Lin Liu. Dense trajectories and motion boundary descriptors for action recognition. IJCV , 103(1), 2013. 2\n[59] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR , 2018. 2, 5, 6,\n12\n[60] Xiaolong Wang and Abhinav Gupta. Videos as space-time"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e6-6db4-11ee-b57c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-51", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "12\n[60] Xiaolong Wang and Abhinav Gupta. Videos as space-time\nregion graphs. In ECCV , 2018. 1, 2, 3, 4, 5, 8\n[61] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term\nfeature banks for detailed video understanding. In CVPR ,\n2019. 1, 2, 3, 5, 6, 7, 8, 12\n[62] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin,\nand Michael Auli. Pay less attention with lightweight and\ndynamic convolutions. In ICLR , 2019. 2\n[63] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen\nLo, and Ross Girshick. Detectron2. https://github.\ncom/facebookresearch/detectron2 , 2019. 12\n[64] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei.\nScene graph generation by iterative message passing. In\nCVPR , 2017. 3, 7\n[65] Huijuan Xu, Lizhi Yang, Stan Sclaroff, Kate Saenko, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e7-6db4-11ee-8f1e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-52", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "CVPR , 2017. 3, 7\n[65] Huijuan Xu, Lizhi Yang, Stan Sclaroff, Kate Saenko, and\nTrevor Darrell. Spatio-temporal action detection with multiobject interaction. In arXiv preprint arXiv:2004.00180 ,\n2020. 2\n[66] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi\nParikh. Graph r-cnn for scene graph generation. In ECCV ,\n2018. 3\n[67] Xitong Yang, Xiaodong Yang, Ming-Yu Liu, Fanyi Xiao,\nLarry S Davis, and Jan Kautz. Step: Spatio-temporal progressive learning for video action detection. In CVPR , 2019.\n8\n[68] Bangpeng Yao and Li Fei-Fei. Modeling mutual context of\nobject and human pose in human-object interaction activities. In CVPR , 2010. 2\n[69] Ji Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao,\nand Bryan Catanzaro. Graphical contrastive losses for scene\ngraph parsing. In CVPR , 2019. 7\n[70] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In CVPR , 2020. 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e8-6db4-11ee-b0cb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-53", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "[70] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In CVPR , 2020. 2\n[71] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia\nSchmid. A structured model for action detection. In CVPR ,\n2019. 1, 2, 3, 5, 8\n[72] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV ,\n2018. 2\nAppendix\nIn this appendix, we additional analysis on A V A (Sec. A)\nand implementation details (Sec. B).\nA. A V A per-class analysis\nFigure 7 presents per-class results on A V A for our SlowFast baseline, spatial and spatio-temporal graph models\nwhen using a ResNet-50 backbone. We additionally show\nthe number of examples in the training set. Our proposed\ngraph models improve on both head and tail classes in A V A.\nIn fact, the largest absolute improvements are observed on\ntail classes such as \u201ccut\u201d, \u201cswim\u201d and \u201csing to\u201d. However,\nnote that the absolute accuracy of each class is still correlated with the number of training examples. The classes\nwhich all variants of our model perform the worst on, are"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14e9-6db4-11ee-974c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-54", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "note that the absolute accuracy of each class is still correlated with the number of training examples. The classes\nwhich all variants of our model perform the worst on, are\nthe classes with the least training examples such as \u201cpoint\nto (an object)\u201d, \u201ctake a photo\u201d and \u201cturn (eg a screwdriver)\u201d.\nTo emphasise the long-tailed distribution of A V A, note that\n\u201cpoint to (an object)\u201d has the fewest training examples (just\n97), whilst \u201cwatch (a person)\u201d has the most with 168 148.\nB. Additional Implementation details\nPerson detections For our spatio-temporal action detection experiments, we use external person detections as our\nactor region proposals like [11, 61]. For our experiments\non A V A, we use the person detections publicly released\nby [11, 61] during both training and testing. More speci\ufb01cally, this is a Faster-RCNN [46] model, pretrained on Microsoft COCO [37] and then \ufb01netuned on the training set of\nA V A [17], using Detectron [63].\nFor our experiments on UCF101-24 [53], we \ufb01netune a Faster-RCNN detector pretrained on COCO on the\nUCF101-24 training set using Detectron."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ea-6db4-11ee-9468-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-55", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "UCF101-24 training set using Detectron.\nWhen training on both A V A and UCF101-24 datasets, we\nuse both ground-truth and predicted person bounding boxes.\nPredicted bounding boxes are assigned labels by matching\nthem to ground-truth boxes using an IoU threshold of 0.75.\nPredicted bounding boxes which do not match any ground\ntruth boxes act as negative examples for all action classes.\nOpenImages object detector For our \u201cexplicit object\nrepresentation\u201d experiments on A V A, we use the publicly\navailable Faster-RCNN detector trained on OpenImages\nv41.\nOptimiser hyperparameters We train our models following the settings of the publicly released SlowFast\ncode [11]. We train for 20 epochs using synchronous\nStochastic Gradient Descent (SGD) and a momentum of 0.9\n1https://tfhub.dev/google/faster_rcnn/\nopenimages_v4/inception_resnet_v2/1on 8 Nvidia V100 (16 GB) GPUs. The learning rate was set\nto 0.1, and reduced by a factor of 10 after 10 and 15 epochs\nrespectively. We employed a linear-warmup schedule for\nthe learning rate, increasing linearly from 1:25\u000210\u00004to0:1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14eb-6db4-11ee-ac3d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-56", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "respectively. We employed a linear-warmup schedule for\nthe learning rate, increasing linearly from 1:25\u000210\u00004to0:1\nin the \ufb01rst 5 epochs. We also used a weight decay of 10\u00007.\nAn epoch is de\ufb01ned as all the keyframes in the dataset.\nTraining loss functions For spatio-temporal action\nrecognition on A V A and UCF101-24, we use the binary\ncross-entropy loss function. The loss function for a single\nexample is:\nL(x;y) =\u00001\nCCX\niyilog (\u001b(xi))+(1\u0000yi) log (1\u0000\u001b(xi));\n(10)\nwhereCis the number of classes, \u001bis the sigmoid activation function, x2RCdenotes the logits predicted by the\nnetwork and yi2f0;1gdenotes the binary ground truth for\ntheithclass.\nFor scene graph classi\ufb01cation, we use two loss functions:\nOne for predicting the object class of each bounding box\nproposal, and another for predicting the relationship label\nbetween each of theN(N\u00001)\n2pairs of object proposals. For\nthe former, we use the softmax cross-entropy, as each proposal can only be assigned a single object class. And for"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ec-6db4-11ee-a927-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-57", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "2pairs of object proposals. For\nthe former, we use the softmax cross-entropy, as each proposal can only be assigned a single object class. And for\nthe latter, we use the sigmoid cross-entropy, as there can\nbe multiple relationship labels between any pair of objects.\nConcretely, the loss is\nL(x;w;y;z ) =\u0015Lobject(x;y) +Lrel(w;z) (11)\nLobject(x;y) =\u00001\nNNX\niCX\njyijlog (Softmax (x)ij)(12)\nLrel(w;z) =\u00002\nN(N\u00001)RNX\nii\u00001X\njRX\nrzijrlog (\u001b(wijr))\n+ (1\u0000zijr) log (1\u0000\u001b(wijr)): (13)\nHere,Nis the number of object proposals in the video clip,\nRthe number of relationship classes and Cthe number of\nobject classes. yis the one-hot, ground-truth object class label,zthe binary ground-truth relationship label, and xand\nwdenote the object- and relationship-logits respectively.\nOn Action Genome [23], there are R= 25 relationship"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ed-6db4-11ee-947a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-58", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "wdenote the object- and relationship-logits respectively.\nOn Action Genome [23], there are R= 25 relationship\nclasses, and C= 35 object classes. We set \u0015= 0:5during\ntraining on Action Genome to prevent the object class loss\nfrom dominating the overall loss (the softmax cross entropy\nhas a higher loss than the binary sigmoid cross entropy at\nthe start of training).\nMulti-headed attention For the GAT [57] and NonLocal [59] message-passing functions used in the paper\nMean AP\nNumber of training examples\n0.000.250.500.751.00\n100100010000100000watch (a person)\nstand\ntalk to (e.g. self a person a \nlisten to (a person)\nsit\ncarry/hold (an object)\nwalk\ntouch (an object)\nbend/bow (at the waist)\nlie/sleep\nride (e.g. a bike a car a horse)\nrun/jog\nanswer phone\ndance\neat\nsmoke\nfight/hit (a person)\ndrink\ncrouch/kneel\nread\nmartial art\ngrab (a person)\nwatch (e.g. TV)\nsing to (e.g. self a person a \nplay musical instrument"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ee-6db4-11ee-a3e6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-59", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "read\nmartial art\ngrab (a person)\nwatch (e.g. TV)\nsing to (e.g. self a person a \nplay musical instrument\nopen (e.g. a window a car \ndrive (e.g. a car a truck)\nhand clap\nget up\nhug (a person)\ngive/serve (an object) to (a \nclose (e.g. a door a box)\nwrite\nsail boat\nkiss (a person)\nlisten (e.g. to music)\nhand shake\ntake (an object) from (a \nput down\nlift/pick up\ntext on/look at a cellphone\nlift (a person)\npush (an object)\npush (another person)\nhand wave\npull (an object)\ndress/put on clothing\nfall down\nclimb (e.g. a mountain)\nthrow\njump/leap\nenter\nshoot\ncut\ntake a photo\nhit (an object)\nwork on a computer\nturn (e.g. a screwdriver)\nswim\npoint to (an object)\nBaseline Spatial Spatio-temporal Training examplesFigure 7: Mean Average Precision for each action class on A V A, for the SlowFast baseline, spatial- and spatio-temporal graph models with"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ef-6db4-11ee-8af2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15662-60", "title": "Unified Graph Structured Models for Video Understanding", "source": "http://arxiv.org/pdf/2103.15662", "start_index": 0, "chroma:document": "a ResNet-50 backbone. The number of examples in the training set are shown with the blue line on the right, vertical axis. Note that a\nlogarithmic scale is used, as A V A has a long-tailed distribution of labels.\nTable 6: Ablation study of the number of heads when using GAT\nas the message passing function. Mean AP reported on A V A using\na ResNet-50 backbone.\nNumber of heads 1 2 3 4 5\nMean AP 25.4 25.8 25.9 25.9 25.9\n(Sec 3.4), it is common to compute \u201cmulti-headed\u201d attention [56]. In all experiments reported in the main paper,\nwe used 4 heads, as motivated by the ablation study using\nGAT on the A V A dataset in Tab. 6. To combine the messages from each head, we performed an attention-weighted\nconvex combination as done in Eq. 9.\nMultiscale testing We use three scales when performing\nmultiscale testing (as done in Table 3 of the main paper).\nSpeci\ufb01cally, we resize input video-clips so that the shortest\nside is 224, 256 and 320 respectively."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f0-6db4-11ee-b7d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-0", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "ViViT: A Video Vision Transformer\nAnurag Arnab*Mostafa Dehghani*Georg Heigold Chen Sun Mario Lu \u02c7ci\u00b4c\u2020Cordelia Schmid\u2020\nGoogle Research\nfaarnab, dehghani, heigold, chensun, lucic, cordelias g@google.com\nAbstract\nWe present pure-transformer based models for video\nclassi\ufb01cation, drawing upon the recent success of such models in image classi\ufb01cation. Our model extracts spatiotemporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we\npropose several, ef\ufb01cient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show\nhow we can effectively regularise the model during training\nand leverage pretrained image models to be able to train on\ncomparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple\nvideo classi\ufb01cation benchmarks including Kinetics 400 and\n600, Epic Kitchens, Something-Something v2 and Moments\nin Time, outperforming prior methods based on deep 3D"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f1-6db4-11ee-b549-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-1", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "video classi\ufb01cation benchmarks including Kinetics 400 and\n600, Epic Kitchens, Something-Something v2 and Moments\nin Time, outperforming prior methods based on deep 3D\nconvolutional networks. To facilitate further research, we\nrelease code at https://github.com/google-research/scenic.\n1. Introduction\nApproaches based on deep convolutional neural networks have advanced the state-of-the-art across many standard datasets for vision problems since AlexNet [38]. At\nthe same time, the most prominent architecture of choice in\nsequence-to-sequence modelling (e.g. in natural language\nprocessing) is the transformer [68], which does not use convolutions, but is based on multi-headed self-attention. This\noperation is particularly effective at modelling long-range\ndependencies and allows the model to attend over all elements in the input sequence. This is in stark contrast to\nconvolutions where the corresponding \u201creceptive \ufb01eld\u201d is\nlimited, and grows linearly with the depth of the network.\nThe success of attention-based models in NLP has recently inspired approaches in computer vision to integrate\ntransformers into CNNs [75, 7], as well as some attempts to\nreplace convolutions completely [49, 3, 53]. However, it is\n*Equal contribution"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f2-6db4-11ee-83fc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-2", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "transformers into CNNs [75, 7], as well as some attempts to\nreplace convolutions completely [49, 3, 53]. However, it is\n*Equal contribution\n\u2020Equal advisingonly very recently with the Vision Transformer (ViT) [18],\nthat a pure-transformer based architecture has outperformed\nits convolutional counterparts in image classi\ufb01cation. Dosovitskiy et al. [18] closely followed the original transformer\narchitecture of [68], and noticed that its main bene\ufb01ts\nwere observed at large scale \u2013 as transformers lack some\nof the inductive biases of convolutions (such as translational equivariance), they seem to require more data [18]\nor stronger regularisation [64].\nInspired by ViT, and the fact that attention-based architectures are an intuitive choice for modelling longrange contextual relationships in video, we develop several transformer-based models for video classi\ufb01cation. Currently, the most performant models are based on deep 3D\nconvolutional architectures [8, 20, 21] which were a natural extension of image classi\ufb01cation CNNs [27, 60]. Recently, these models were augmented by incorporating selfattention into their later layers to better capture long-range\ndependencies [75, 23, 79, 1]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f3-6db4-11ee-bd36-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-3", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "dependencies [75, 23, 79, 1].\nAs shown in Fig. 1, we propose pure-transformer models for video classi\ufb01cation. The main operation performed\nin this architecture is self-attention, and it is computed on\na sequence of spatio-temporal tokens that we extract from\nthe input video. To effectively process the large number of\nspatio-temporal tokens that may be encountered in video,\nwe present several methods of factorising our model along\nspatial and temporal dimensions to increase ef\ufb01ciency and\nscalability. Furthermore, to train our model effectively on\nsmaller datasets, we show how to reguliarise our model during training and leverage pretrained image models.\nWe also note that convolutional models have been developed by the community for several years, and there are\nthus many \u201cbest practices\u201d associated with such models.\nAs pure-transformer models present different characteristics, we need to determine the best design choices for such\narchitectures. We conduct a thorough ablation analysis of\ntokenisation strategies, model architecture and regularisation methods. Informed by this analysis, we achieve stateof-the-art results on multiple standard video classi\ufb01cation\nbenchmarks, including Kinetics 400 and 600 [35], Epic"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f4-6db4-11ee-863c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-4", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "benchmarks, including Kinetics 400 and 600 [35], Epic\nKitchens 100 [13], Something-Something v2 [26] and Moments in Time [45].arXiv:2103.15691v2  [cs.CV]  1 Nov 2021\n\u20261CLS032NPosition + Token EmbeddingMLP HeadClassFactorisedEncoderL\u00d7\nK         V         Q Self-AttentionTransformer  Encoder MLPLayer Norm\nLayer NormMulti-HeadDot-Product Attention\nEmbed to tokensFactorisedSelf-Attention\n21NFactorisedDot-Product\n\u25cf\u25cf\u25cfSpatialSpatial\u25cf\u25cf\u25cfTemporalTemporal\nSpatialTemporalSpatialTemporal\u25cf\u25cf\u25cfSpatialTemporal\u25cf\u25cf\u25cfFuseSpatialTemporalFuse\n\u25cf\u25cf\u25cf21N\u25cf\u25cf\u25cf21N\u25cf\u25cf\u25cfFigure 1: We propose a pure-transformer architecture for video classi\ufb01cation, inspired by the recent success of such models for images [18].\nTo effectively process a large number of spatio-temporal tokens, we develop several model variants which factorise different components\nof the transformer encoder over the spatial- and temporal-dimensions. As shown on the right, these factorisations correspond to different\nattention patterns over space and time.\n2. Related Work"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f5-6db4-11ee-bff4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-5", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "attention patterns over space and time.\n2. Related Work\nArchitectures for video understanding have mirrored advances in image recognition. Early video research used\nhand-crafted features to encode appearance and motion\ninformation [41, 69]. The success of AlexNet on ImageNet [38, 16] initially led to the repurposing of 2D image convolutional networks (CNNs) for video as \u201ctwostream\u201d networks [34, 56, 47]. These models processed\nRGB frames and optical \ufb02ow images independently before\nfusing them at the end. Availability of larger video classi\ufb01cation datasets such as Kinetics [35] subsequently facilitated the training of spatio-temporal 3D CNNs [8, 22, 65]\nwhich have signi\ufb01cantly more parameters and thus require\nlarger training datasets. As 3D convolutional networks require signi\ufb01cantly more computation than their image counterparts, many architectures factorise convolutions across\nspatial and temporal dimensions and/or use grouped convolutions [59, 66, 67, 81, 20]. We also leverage factorisation\nof the spatial and temporal dimensions of videos to increase\nef\ufb01ciency, but in the context of transformer-based models.\nConcurrently, in natural language processing (NLP),"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f6-6db4-11ee-82d1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-6", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "of the spatial and temporal dimensions of videos to increase\nef\ufb01ciency, but in the context of transformer-based models.\nConcurrently, in natural language processing (NLP),\nVaswani et al. [68] achieved state-of-the-art results by replacing convolutions and recurrent networks with the transformer network that consisted only of self-attention, layer\nnormalisation and multilayer perceptron (MLP) operations.\nCurrent state-of-the-art architectures in NLP [17, 52] remain transformer-based, and have been scaled to web-scale\ndatasets [5]. Many variants of the transformer have also\nbeen proposed to reduce the computational cost of selfattention when processing longer sequences [10, 11, 37,\n62, 63, 73] and to improve parameter ef\ufb01ciency [40, 14].\nAlthough self-attention has been employed extensively in\ncomputer vision, it has, in contrast, been typically incorporated as a layer at the end or in the later stages of\nthe network [75, 7, 32, 77, 83] or to augment residualblocks [30, 6, 9, 57] within a ResNet architecture [27]."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f7-6db4-11ee-8697-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-7", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "the network [75, 7, 32, 77, 83] or to augment residualblocks [30, 6, 9, 57] within a ResNet architecture [27].\nAlthough previous works attempted to replace convolutions in vision architectures [49, 53, 55], it is only very recently that Dosovitisky et al. [18] showed with their ViT architecture that pure-transformer networks, similar to those\nemployed in NLP, can achieve state-of-the-art results for\nimage classi\ufb01cation too. The authors showed that such\nmodels are only effective at large scale, as transformers lack\nsome of inductive biases of convolutional networks (such\nas translational equivariance), and thus require datasets\nlarger than the common ImageNet ILSRVC dataset [16] to\ntrain. ViT has inspired a large amount of follow-up work\nin the community, and we note that there are a number\nof concurrent approaches on extending it to other tasks in\ncomputer vision [71, 74, 84, 85] and improving its dataef\ufb01ciency [64, 48]. In particular, [4, 46] have also proposed\ntransformer-based models for video.\nIn this paper, we develop pure-transformer architectures\nfor video classi\ufb01cation. We propose several variants of our"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f8-6db4-11ee-a81b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-8", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "transformer-based models for video.\nIn this paper, we develop pure-transformer architectures\nfor video classi\ufb01cation. We propose several variants of our\nmodel, including those that are more ef\ufb01cient by factorising the spatial and temporal dimensions of the input video.\nWe also show how additional regularisation and pretrained\nmodels can be used to combat the fact that video datasets\nare not as large as their image counterparts that ViT was\noriginally trained on. Furthermore, we outperform the stateof-the-art across \ufb01ve popular datasets.\n3. Video Vision Transformers\nWe start by summarising the recently proposed Vision\nTransformer [18] in Sec. 3.1, and then discuss two approaches for extracting tokens from video in Sec. 3.2. Finally, we develop several transformer-based architectures\nfor video classi\ufb01cation in Sec. 3.3 and 3.4.\n3.1. Overview of Vision Transformers (ViT)\nVision Transformer (ViT) [18] adapts the transformer\narchitecture of [68] to process 2D images with minimal\nchanges. In particular, ViT extracts Nnon-overlapping image patches, xi2Rh\u0002w, performs a linear projection and\nthen rasterises them into 1D tokens zi2Rd. The sequence"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14f9-6db4-11ee-9f0a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-9", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "then rasterises them into 1D tokens zi2Rd. The sequence\nof tokens input to the following transformer encoder is\nz= [zcls;Ex1;Ex2;:::;ExN] +p; (1)\nwhere the projection by Eis equivalent to a 2D convolution.\nAs shown in Fig. 1, an optional learned classi\ufb01cation token\nzclsis prepended to this sequence, and its representation at\nthe \ufb01nal layer of the encoder serves as the \ufb01nal representation used by the classi\ufb01cation layer [17]. In addition, a\nlearned positional embedding, p2RN\u0002d, is added to the\ntokens to retain positional information, as the subsequent\nself-attention operations in the transformer are permutation\ninvariant. The tokens are then passed through an encoder\nconsisting of a sequence of Ltransformer layers. Each layer\n`comprises of Multi-Headed Self-Attention [68], layer normalisation (LN) [2], and MLP blocks as follows:\ny`=MSA (LN(z`)) +z`(2)\nz`+1=MLP(LN(y`)) +y`: (3)\nThe MLP consists of two linear projections separated by a"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14fa-6db4-11ee-92c0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-10", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "z`+1=MLP(LN(y`)) +y`: (3)\nThe MLP consists of two linear projections separated by a\nGELU non-linearity [28] and the token-dimensionality, d,\nremains \ufb01xed throughout all layers. Finally, a linear classi\ufb01er is used to classify the encoded input based on zL\ncls2Rd,\nif it was prepended to the input, or a global average pooling\nof all the tokens, zL, otherwise.\nAs the transformer [68], which forms the basis of\nViT [18], is a \ufb02exible architecture that can operate on any\nsequence of input tokens z2RN\u0002d, we describe strategies\nfor tokenising videos next.\n3.2. Embedding video clips\nWe consider two simple methods for mapping a video\nV2RT\u0002H\u0002W\u0002Cto a sequence of tokens ~ z2\nRnt\u0002nh\u0002nw\u0002d. We then add the positional embedding and\nreshape into RN\u0002dto obtain z, the input to the transformer.\nUniform frame sampling As illustrated in Fig. 2, a\nstraightforward method of tokenising the input video is to\nuniformly sample ntframes from the input video clip, embed each 2D frame independently using the same method"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14fb-6db4-11ee-b781-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-11", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "straightforward method of tokenising the input video is to\nuniformly sample ntframes from the input video clip, embed each 2D frame independently using the same method\nas ViT [18], and concatenate all these tokens together. Concretely, ifnh\u0001nwnon-overlapping image patches are extracted from each frame, as in [18], then a total of nt\u0001nh\u0001nw\ntokens will be forwarded through the transformer encoder.\nIntuitively, this process may be seen as simply constructing\na large 2D image to be tokenised following ViT. We note\nthat this is the input embedding method employed by the\nconcurrent work of [4].\n#!\"Figure 2: Uniform frame sampling: We simply sample ntframes,\nand embed each 2D frame independently following ViT [18].\n!\"\n#\nFigure 3: Tubelet embedding. We extract and linearly embed nonoverlapping tubelets that span the spatio-temporal input volume.\nTubelet embedding An alternate method, as shown in\nFig. 3, is to extract non-overlapping, spatio-temporal\n\u201ctubes\u201d from the input volume, and to linearly project this to\nRd. This method is an extension of ViT\u2019s embedding to 3D,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14fc-6db4-11ee-9e36-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-12", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "\u201ctubes\u201d from the input volume, and to linearly project this to\nRd. This method is an extension of ViT\u2019s embedding to 3D,\nand corresponds to a 3D convolution. For a tubelet of dimensiont\u0002h\u0002w,nt=bT\ntc,nh=bH\nhcandnw=bW\nwc,\ntokens are extracted from the temporal, height, and width\ndimensions respectively. Smaller tubelet dimensions thus\nresult in more tokens which increases the computation.\nIntuitively, this method fuses spatio-temporal information\nduring tokenisation, in contrast to \u201cUniform frame sampling\u201d where temporal information from different frames is\nfused by the transformer.\n3.3. Transformer Models for Video\nAs illustrated in Fig. 1, we propose multiple transformerbased architectures. We begin with a straightforward extension of ViT [18] that models pairwise interactions between all spatio-temporal tokens, and then develop more\nef\ufb01cient variants which factorise the spatial and temporal\ndimensions of the input video at various levels of the transformer architecture.\nModel 1: Spatio-temporal attention This model simply forwards all spatio-temporal tokens extracted from the\nvideo, z0, through the transformer encoder. We note that\nthis has also been explored concurrently by [4] in their"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14fd-6db4-11ee-81cd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-13", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "video, z0, through the transformer encoder. We note that\nthis has also been explored concurrently by [4] in their\n\u201cJoint Space-Time\u201d model. In contrast to CNN architectures, where the receptive \ufb01eld grows linearly with the\nnumber of layers, each transformer layer models all pair\u20261CLSNPositional + Token EmbeddingTemporal + Token Embedding\nEmbed to tokens\u20261N2\u20261N\u2026TTemporal Transformer EncoderMLP Head\nClass\u2026CLS100CLS0CLS0Spatial Transformer EncoderSpatial Transformer EncoderSpatial Transformer EncoderFigure 4: Factorised encoder (Model 2). This model consists of\ntwo transformer encoders in series: the \ufb01rst models interactions\nbetween tokens extracted from the same temporal index to produce\na latent representation per time-index. The second transformer\nmodels interactions between time steps. It thus corresponds to a\n\u201clate fusion\u201d of spatial- and temporal information.\nwise interactions between all spatio-temporal tokens, and it\nthus models long-range interactions across the video from\nthe \ufb01rst layer. However, as it models all pairwise interactions, Multi-Headed Self Attention (MSA) [68] has\nquadratic complexity with respect to the number of tokens."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14fe-6db4-11ee-84b7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-14", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "quadratic complexity with respect to the number of tokens.\nThis complexity is pertinent for video, as the number of tokens increases linearly with the number of input frames, and\nmotivates the development of more ef\ufb01cient architectures\nnext.\nModel 2: Factorised encoder As shown in Fig. 4, this\nmodel consists of two separate transformer encoders. The\n\ufb01rst, spatial encoder, only models interactions between tokens extracted from the same temporal index. A representation for each temporal index, hi2Rd, is obtained after Ls\nlayers: This is the encoded classi\ufb01cation token, zLs\nclsif it was\nprepended to the input (Eq. 1), or a global average pooling\nfrom the tokens output by the spatial encoder, zLs, otherwise. The frame-level representations, hi, are concatenated\nintoH2Rnt\u0002d, and then forwarded through a temporal\nencoder consisting of Lttransformer layers to model interactions between tokens from different temporal indices.\nThe output token of this encoder is then \ufb01nally classi\ufb01ed.\nThis architecture corresponds to a \u201clate fusion\u201d [34,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d14ff-6db4-11ee-9c6f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-15", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "The output token of this encoder is then \ufb01nally classi\ufb01ed.\nThis architecture corresponds to a \u201clate fusion\u201d [34,\n56, 72, 46] of temporal information, and the initial spatial encoder is identical to the one used for image classi\ufb01cation. It is thus analogous to CNN architectures such\nas [24, 34, 72, 86] which \ufb01rst extract per-frame features, and then aggregate them into a \ufb01nal representation\nbefore classifying them. Although this model has more\ntransformer layers than Model 1 (and thus more parameters), it requires fewer \ufb02oating point operations (FLOPs),\nas the two separate transformer blocks have a complexity\nofO((nh\u0001nw)2+n2\nt)compared toO((nt\u0001nh\u0001nw)2)of\nModel 1.\nTransformer Block xLK         V       Q MLPLayer Norm\nLayer NormMulti-HeadAttention\nK         V       Q Layer NormMulti-HeadAttention\nTemporal Self-Attention BlockSpatial Self-Attention BlockToken embeddingPositional embeddingFigure 5: Factorised self-attention (Model 3). Within each transformer block, the multi-headed self-attention operation is factorised into two operations (indicated by striped boxes) that \ufb01rst"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1500-6db4-11ee-b8db-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-16", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "only compute self-attention spatially, and then temporally.\nModel 3: Factorised self-attention This model, in contrast, contains the same number of transformer layers as\nModel 1. However, instead of computing multi-headed\nself-attention across all pairs of tokens, z`, at layerl, we\nfactorise the operation to \ufb01rst only compute self-attention\nspatially (among all tokens extracted from the same temporal index), and then temporally (among all tokens extracted from the same spatial index) as shown in Fig. 5.\nEach self-attention block in the transformer thus models\nspatio-temporal interactions, but does so more ef\ufb01ciently\nthan Model 1 by factorising the operation over two smaller\nsets of elements, thus achieving the same computational\ncomplexity as Model 2. We note that factorising attention\nover input dimensions has also been explored in [29, 78],\nand concurrently in the context of video by [4] in their \u201cDivided Space-Time\u201d model.\nThis operation can be performed ef\ufb01ciently by reshaping\nthe tokens zfromR1\u0002nt\u0001nh\u0001nw\u0001dtoRnt\u0002nh\u0001nw\u0001d(denoted\nbyzs) to compute spatial self-attention. Similarly, the input"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1501-6db4-11ee-8636-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-17", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "byzs) to compute spatial self-attention. Similarly, the input\nto temporal self-attention, ztis reshaped to Rnh\u0001nw\u0002nt\u0001d.\nHere we assume the leading dimension is the \u201cbatch dimension\u201d. Our factorised self-attention is de\ufb01ned as\ny`\ns= MSA(LN( z`\ns)) +z`\ns (4)\ny`\nt= MSA(LN( y`\ns)) +y`\ns (5)\nz`+1= MLP(LN( y`\nt)) +y`\nt: (6)\nWe observed that the order of spatial-then-temporal selfattention or temporal-then-spatial self-attention does not\nmake a difference, provided that the model parameters are\ninitialised as described in Sec. 3.4. Note that the number\nof parameters, however, increases compared to Model 1, as\nthere is an additional self-attention layer (cf. Eq. 7). We do\nnot use a classi\ufb01cation token in this model, to avoid ambiguities when reshaping the input tokens between spatial and\ntemporal dimensions.\nModel 4: Factorised dot-product attention Finally, we"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1502-6db4-11ee-a06a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-18", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "temporal dimensions.\nModel 4: Factorised dot-product attention Finally, we\ndevelop a model which has the same computational complexity as Models 2 and 3, while retaining the same number\nof parameters as the unfactorised Model 1. The factorisation of spatial- and temporal dimensions is similar in spirit\nK         V       Q Self-Attention BlockLayer NormMulti-HeadDot-product AttentionConcatenateLinear\nK             V                Q Scaled Dot-Product AttentionLinearLinearLinearSpatial HeadsK             V                Q Scaled Dot-Product AttentionLinearLinearLinearTemporal HeadsFigure 6: Factorised dot-product attention (Model 4). For half of\nthe heads, we compute dot-product attention over only the spatial\naxes, and for the other half, over only the temporal axis.\nto Model 3, but we factorise the multi-head dot-product attention operation instead (Fig. 6). Concretely, we compute\nattention weights for each token separately over the spatialand temporal-dimensions using different heads. First, we\nnote that the attention operation for each head is de\ufb01ned as\nAttention( Q;K;V) = Softmax\u0012QK>\npdk\u0013\nV: (7)\nIn self-attention, the queries Q=XW q, keys K=XW k,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1503-6db4-11ee-b366-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-19", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "pdk\u0013\nV: (7)\nIn self-attention, the queries Q=XW q, keys K=XW k,\nand values V=XW vare linear projections of the input X\nwithX;Q;K;V2RN\u0002d. Note that in the unfactorised\ncase (Model 1), the spatial and temporal dimensions are\nmerged asN=nt\u0001nh\u0001nw.\nThe main idea here is to modify the keys and values for\neach query to only attend over tokens from the same spatialand temporal index by constructing Ks;Vs2Rnh\u0001nw\u0002d\nandKt;Vt2Rnt\u0002d, namely the keys and values corresponding to these dimensions. Then, for half of the attention heads, we attend over tokens from the spatial dimension by computing Ys= Attention( Q;Ks;Vs), and for\nthe rest we attend over the temporal dimension by computingYt= Attention( Q;Kt;Vt). Given that we are only\nchanging the attention neighbourhood for each query, the\nattention operation has the same dimension as in the unfactorised case, namely Ys;Yt2RN\u0002d. We then combine\nthe outputs of multiple heads by concatenating them and\nusing a linear projection [68], Y= Concat( Ys;Yt)WO."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1504-6db4-11ee-a736-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-20", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "the outputs of multiple heads by concatenating them and\nusing a linear projection [68], Y= Concat( Ys;Yt)WO.\n3.4. Initialisation by leveraging pretrained models\nViT [18] has been shown to only be effective when\ntrained on large-scale datasets, as transformers lack some of\nthe inductive biases of convolutional networks [18]. However, even the largest video datasets such as Kinetics [35],\nhave several orders of magnitude less labelled examples\nwhen compared to their image counterparts [16, 39, 58]. As\na result, training large models from scratch to high accuracy\nis extremely challenging. To sidestep this issue, and enable\nmore ef\ufb01cient training we initialise our video models from\npretrained image models. However, this raises several practical questions, speci\ufb01cally on how to initialise parameters\nnot present or incompatible with image models. We nowdiscuss several effective strategies to initialise these largescale video classi\ufb01cation models.\nPositional embeddings A positional embedding pis\nadded to each input token (Eq. 1). However, our video\nmodels have nttimes more tokens than the pretrained image model. As a result, we initialise the positional embeddings by \u201crepeating\u201d them temporally from Rnw\u0001nh\u0002dto"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1505-6db4-11ee-ab19-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-21", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Rnt\u0001nh\u0001nw\u0002d. Therefore, at initialisation, all tokens with\nthe same spatial index have the same embedding which is\nthen \ufb01ne-tuned.\nEmbedding weights, EWhen using the \u201ctubelet embedding\u201d tokenisation method (Sec. 3.2), the embedding \ufb01lter\nEis a 3D tensor, compared to the 2D tensor in the pretrained model, Eimage. A common approach for initialising\n3D convolutional \ufb01lters from 2D \ufb01lters for video classi\ufb01cation is to \u201cin\ufb02ate\u201d them by replicating the \ufb01lters along the\ntemporal dimension and averaging them [8, 22] as\nE=1\nt[Eimage;:::;Eimage;:::;Eimage]: (8)\nWe consider an additional strategy, which we denote as\n\u201ccentral frame initialisation\u201d, where Eis initialised with zeroes along all temporal positions, except at the centre bt\n2c,\nE= [0;:::;Eimage;:::;0]: (9)\nTherefore, the 3D convolutional \ufb01lter effectively behaves\nlike \u201cUniform frame sampling\u201d (Sec. 3.2) at initialisation,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1506-6db4-11ee-a9fb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-22", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Therefore, the 3D convolutional \ufb01lter effectively behaves\nlike \u201cUniform frame sampling\u201d (Sec. 3.2) at initialisation,\nwhile also enabling the model to learn to aggregate temporal\ninformation from multiple frames as training progresses.\nTransformer weights for Model 3 The transformer\nblock in Model 3 (Fig. 5) differs from the pretrained ViT\nmodel [18], in that it contains two multi-headed self attention (MSA) modules. In this case, we initialise the spatial\nMSA module from the pretrained module, and initialise all\nweights of the temporal MSA with zeroes, such that Eq. 5\nbehaves as a residual connection [27] at initialisation.\n4. Empirical evaluation\nWe \ufb01rst present our experimental setup and implementation details in Sec. 4.1, before ablating various components\nof our model in Sec. 4.2. We then present state-of-the-art\nresults on \ufb01ve datasets in Sec. 4.3.\n4.1. Experimental Setup\nNetwork architecture and training Our backbone architecture follows that of ViT [18] and BERT [17]. We consider ViT-Base (ViT-B, L=12,NH=12,d=768), ViT-Large"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1507-6db4-11ee-83d2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-23", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "(ViT-L,L=24,NH=16,d=1024 ), and ViT-Huge (ViT-H,\nL=32,NH=16,d=1280 ), whereLis the number of transformer layers, each with a self-attention block of NHheads\nTable 1: Comparison of input encoding methods using ViViT-B\nand spatio-temporal attention on Kinetics. Further details in text.\nTop-1 accuracy\nUniform frame sampling 78.5\nTubelet embedding\nRandom initialisation [25] 73.2\nFilter in\ufb02ation [8] 77.6\nCentral frame 79.2\nand hidden dimension d. We also apply the same naming\nscheme to our models (e.g., ViViT-B/16x2 denotes a ViTBase backbone with a tubelet size of h\u0002w\u0002t= 16\u000216\u00022).\nIn all experiments, the tubelet height and width are equal.\nNote that smaller tubelet sizes correspond to more tokens at\nthe input, and thus more computation.\nWe train our models using synchronous SGD and momentum, a cosine learning rate schedule and TPU-v3 accelerators. We initialise our models from a ViT image"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1508-6db4-11ee-a2f2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-24", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "We train our models using synchronous SGD and momentum, a cosine learning rate schedule and TPU-v3 accelerators. We initialise our models from a ViT image\nmodel trained either on ImageNet-21K [16] (unless otherwise speci\ufb01ed) or the larger JFT [58] dataset. We implement our method using the Scenic library [15] and have released our code and models.\nDatasets We evaluate the performance of our proposed\nmodels on a diverse set of video classi\ufb01cation datasets:\nKinetics [35] consists of 10-second videos sampled at\n25fps from YouTube. We evaluate on both Kinetics 400\nand 600, containing 400 and 600 classes respectively. As\nthese are dynamic datasets (videos may be removed from\nYouTube), we note our dataset sizes are approximately 267\n000 and 446 000 respectively.\nEpic Kitchens-100 consists of egocentric videos capturing daily kitchen activities spanning 100 hours and 90 000\nclips [13]. We report results following the standard \u201caction\nrecognition\u201d protocol. Here, each video is labelled with a\n\u201cverb\u201d and a \u201cnoun\u201d and we therefore predict both categories using a single network with two \u201cheads\u201d. The topscoring verb and action pair predicted by the network form\nan \u201caction\u201d, and action accuracy is the primary metric."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1509-6db4-11ee-8819-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-25", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "an \u201caction\u201d, and action accuracy is the primary metric.\nMoments in Time [45] consists of 800 000, 3-second\nYouTube clips that capture the gist of a dynamic scene involving animals, objects, people, or natural phenomena.\nSomething-Something v2 (SSv2) [26] contains 220 000\nvideos, with durations ranging from 2 to 6 seconds. In contrast to the other datasets, the objects and backgrounds in\nthe videos are consistent across different action classes, and\nthis dataset thus places more emphasis on a model\u2019s ability\nto recognise \ufb01ne-grained motion cues.\nInference The input to our network is a video clip of 32\nframes using a stride of 2, unless otherwise mentioned, similar to [21, 20]. Following common practice, at inference\ntime, we process multiple views of a longer video and aver-Table 2: Comparison of model architectures using ViViT-B as the\nbackbone, and tubelet size of 16\u00022. We report Top-1 accuracy on\nKinetics 400 (K400) and action accuracy on Epic Kitchens (EK).\nRuntime is during inference on a TPU-v3.\nK400 EKFLOPs\n(\u0002109)Params\n(\u0002106)Runtime\n(ms)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d150a-6db4-11ee-9920-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-26", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Runtime is during inference on a TPU-v3.\nK400 EKFLOPs\n(\u0002109)Params\n(\u0002106)Runtime\n(ms)\nModel 1: Spatio-temporal 80.0 43.1 455.2 88.9 58.9\nModel 2: Fact. encoder 78.8 43.7 284.4 115.1 17.4\nModel 3: Fact. self-attention 77.4 39.1 372.3 117.3 31.7\nModel 4: Fact. dot product 76.3 39.5 277.1 88.9 22.9\nModel 2: Ave. pool baseline 75.8 38.8 283.9 86.7 17.3\nTable 3: The effect of varying the number of temporal transformers,Lt, in the Factorised encoder model (Model 2). We report the\nTop-1 accuracy on Kinetics 400. Note that Lt= 0corresponds to\nthe \u201caverage pooling baseline\u201d.\nLt 0 1 4 8 12\nTop-1 75.8 78.6 78.8 78.8 78.9\nage per-view logits to obtain the \ufb01nal result. Unless otherwise speci\ufb01ed, we use a total of 4 views per video (as this"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d150b-6db4-11ee-aae8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-27", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "age per-view logits to obtain the \ufb01nal result. Unless otherwise speci\ufb01ed, we use a total of 4 views per video (as this\nis suf\ufb01cient to \u201csee\u201d the entire video clip across the various\ndatasets), and ablate these and other design choices next.\n4.2. Ablation study\nInput encoding We \ufb01rst consider the effect of different\ninput encoding methods (Sec. 3.2) using our unfactorised\nmodel (Model 1) and ViViT-B on Kinetics 400. As we pass\n32-frame inputs to the network, sampling 8 frames and extracting tubelets of length t= 4 correspond to the same\nnumber of tokens in both cases. Table 1 shows that tubelet\nembedding initialised using the \u201ccentral frame\u201d method\n(Eq. 9) performs well, outperforming the commonly-used\n\u201c\ufb01lter in\ufb02ation\u201d initialisation method [8, 22] by 1.6%, and\n\u201cuniform frame sampling\u201d by 0.7%. We therefore use this\nencoding method for all subsequent experiments.\nModel variants We compare our proposed model variants (Sec. 3.3) across the Kinetics 400 and Epic Kitchens"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d150c-6db4-11ee-ba8f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-28", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "encoding method for all subsequent experiments.\nModel variants We compare our proposed model variants (Sec. 3.3) across the Kinetics 400 and Epic Kitchens\ndatasets, both in terms of accuracy and ef\ufb01ciency, in Tab. 2.\nIn all cases, we use the \u201cBase\u201d backbone and tubelet size of\n16\u00022. Model 2 (\u201cFactorised Encoder\u201d) has an additional\nhyperparameter, the number of temporal transformers, Lt.\nWe setLt= 4 for all experiments and show in Tab. 3 that\nthe model is not sensitive to this choice.\nThe unfactorised model (Model 1) performs the best\non Kinetics 400. However, it can also over\ufb01t on smaller\ndatasets such as Epic Kitchens, where we \ufb01nd our \u201cFactorised Encoder\u201d (Model 2) to perform the best. We also\nconsider an additional baseline (last row), based on Model\n2, where we do not use any temporal transformer, and simply average pool the frame-level representations from the\nspatial encoder before classifying. This average pooling\nbaseline performs the worst, and has a larger accuracy drop\nTable 4: The effect of progressively adding regularisation (each\nrow includes all methods above it) on Top-1 action accuracy on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d150d-6db4-11ee-88fc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-29", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "baseline performs the worst, and has a larger accuracy drop\nTable 4: The effect of progressively adding regularisation (each\nrow includes all methods above it) on Top-1 action accuracy on\nEpic Kitchens. We use a Factorised encoder model with tubelet\nsize16\u00022.\nTop-1 accuracy\nRandom crop, \ufb02ip, colour jitter 38.4\n+ Kinetics 400 initialisation 39.6\n+ Stochastic depth [31] 40.2\n+ Random augment [12] 41.1\n+ Label smoothing [61] 43.1\n+ Mixup [82] 43.7\n16x8 16x4 16x2\nInput tubelet size787980Top-1 Accuracy\n16x8 16x4 16x2\nInput tubelet size0.51.01.5TFLOPs\nViViT-B ViViT-L\n(a) Accuracy (b) Compute\nFigure 7: The effect of the backbone architecture on (a) accuracy\nand (b) computation on Kinetics 400, for the spatio-temporal attention model (Model 1).\n16x8 16x4 16x2\nInput tubelet size72.575.077.580.0Top-1 Accuracy\n16x8 16x4 16x2\nInput tubelet size0.20.4TFLOPs"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d150e-6db4-11ee-912a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-30", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "16x8 16x4 16x2\nInput tubelet size0.20.4TFLOPs\nSpatio-temporal Factorised encoder Factorised self-attention Factorised dot-product\n(a) Accuracy (b) Compute\nFigure 8: The effect of varying the number of temporal tokens on\n(a) accuracy and (b) computation on Kinetics 400, for different\nvariants of our model with a ViViT-B backbone.\non Epic Kitchens, suggesting that this dataset requires more\ndetailed modelling of temporal relations.\nAs described in Sec. 3.3, all factorised variants of our\nmodel use signi\ufb01cantly fewer FLOPs than the unfactorised\nModel 1, as the attention is computed separately over\nspatial- and temporal-dimensions. Model 4 adds no additional parameters to the unfactorised Model 1, and uses the\nleast compute. The temporal transformer encoder in Model\n2 operates on only nttokens, which is why there is a barely\na change in compute and runtime over the average pooling baseline, even though it improves the accuracy substantially (3% on Kinetics and 4.9% on Epic Kitchens). Finally, Model 3 requires more compute and parameters than\nthe other factorised models, as its additional self-attention"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d150f-6db4-11ee-9030-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-31", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "the other factorised models, as its additional self-attention\nblock means that it performs another query-, key-, valueand output-projection in each transformer layer [68].\nModel regularisation Pure-transformer architectures\nsuch as ViT [18] are known to require large training\ndatasets, and we observed over\ufb01tting on smaller datasets\nlike Epic Kitchens and SSv2, even when using an ImageNet\npretrained model. In order to effectively train our modelsTable 5: The effect of spatial resolution on the performance of\nViViT-L/16x2 and spatio-temporal attention on Kinetics 400.\nCrop size 224 288 320\nAccuracy 80.3 80.7 81.0\nGFLOPs 1446 2919 3992\nRuntime 58.9 147.6 238.8\non such datasets, we employed several regularisation strategies that we ablate using our \u201cFactorised encoder\u201d model\nin Tab. 4. We note that these regularisers were originally\nproposed for training CNNs, and that [64] have recently\nexplored them for training ViT for image classi\ufb01cation.\nEach row of Tab. 4 includes all the methods from the\nrows above it, and we observe progressive improvements"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1510-6db4-11ee-a7bc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-32", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "explored them for training ViT for image classi\ufb01cation.\nEach row of Tab. 4 includes all the methods from the\nrows above it, and we observe progressive improvements\nfrom adding each regulariser. Overall, we obtain a substantial overall improvement of 5.3% on Epic Kitchens. We\nalso achieve a similar improvement of 5% on SSv2 by using all the regularisation in Tab. 4. Note that the Kineticspretrained models that we initialise from are from Tab. 2,\nand that all Epic Kitchens models in Tab. 2 were trained\nwith all the regularisers in Tab. 4. For larger datasets like\nKinetics and Moments in Time, we do not use these additional regularisers (we use only the \ufb01rst row of Tab. 4),\nas we obtain state-of-the-art results without them. The appendix contains hyperparameter values and additional details for all regularisers.\nVarying the backbone Figure 7 compares the ViViTB and ViViT-L backbones for the unfactorised spatiotemporal model. We observe consistent improvements in\naccuracy as the backbone capacity increases. As expected,\nthe compute also grows as a function of the backbone size.\nVarying the number of tokens We \ufb01rst analyse the performance as a function of the number of tokens along the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1511-6db4-11ee-bad8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-33", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "the compute also grows as a function of the backbone size.\nVarying the number of tokens We \ufb01rst analyse the performance as a function of the number of tokens along the\ntemporal dimension in Fig. 8. We observe that using smaller\ninput tubelet sizes (and therefore more tokens) leads to consistent accuracy improvements across all of our model architectures. At the same time, computation in terms of\nFLOPs increases accordingly, and the unfactorised model\n(Model 1) is impacted the most.\nWe then vary the number of tokens fed into the model\nby increasing the spatial crop-size from the default of 224\nto 320 in Tab. 5. As expected, there is a consistent increase\nin both accuracy and computation. We note that when comparing to prior work we consistently obtain state-of-the-art\nresults (Sec. 4.3) using a spatial resolution of 224, but we\nalso highlight that further improvements can be obtained at\nhigher spatial resolutions.\nVarying the number of input frames In our experiments\nso far, we have kept the number of input frames \ufb01xed at 32.\nWe now increase the number of frames input to the model,\nthereby increasing the number of tokens proportionally."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1512-6db4-11ee-8ee2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-34", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "We now increase the number of frames input to the model,\nthereby increasing the number of tokens proportionally.\nTable 6: Comparisons to state-of-the-art across multiple datasets. For \u201cviews\u201d, x\u0002ydenotes xtemporal crops and yspatial crops. We\nreport the TFLOPs to process all spatio-temporal views. \u201cFE\u201d denotes our Factorised Encoder model.\n(a) Kinetics 400\nMethod Top 1 Top 5 Views TFLOPs\nblVNet [19] 73.5 91.2 \u2013 \u2013\nSTM [33] 73.7 91.6 \u2013 \u2013\nTEA [42] 76.1 92.5 10\u00023 2.10\nTSM-ResNeXt-101 [43] 76.3 \u2013 \u2013 \u2013\nI3D NL [75] 77.7 93.3 10\u00023 10.77\nCorrNet-101 [70] 79.2 \u2013 10\u00023 6.72\nip-CSN-152 [66] 79.2 93.8 10\u00023 3.27\nLGD-3D R101 [51] 79.4 94.4 \u2013 \u2013\nSlowFast R101-NL [21] 79.8 93.9 10\u00023 7.02"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1513-6db4-11ee-9ed7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-35", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "LGD-3D R101 [51] 79.4 94.4 \u2013 \u2013\nSlowFast R101-NL [21] 79.8 93.9 10\u00023 7.02\nX3D-XXL [20] 80.4 94.6 10\u00023 5.82\nTimeSformer-L [4] 80.7 94.7 1\u00023 7.14\nViViT-L/16x2 FE 80.6 92.7 1\u00021 3.98\nViViT-L/16x2 FE 81.7 93.8 1\u00023 11.94\nMethods with large-scale pretraining\nip-CSN-152 [66] (IG [44]) 82.5 95.3 10\u00023 3.27\nViViT-L/16x2 FE (JFT) 83.5 94.3 1\u00023 11.94\nViViT-H/14x2 (JFT) 84.9 95.8 4\u00023 47.77(b) Kinetics 600\nMethod Top 1 Top 5\nAttentionNAS [76] 79.8 94.4\nLGD-3D R101 [51] 81.5 95.6\nSlowFast R101-NL [21] 81.8 95.1\nX3D-XL [20] 81.9 95.5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1514-6db4-11ee-8f40-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-36", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "SlowFast R101-NL [21] 81.8 95.1\nX3D-XL [20] 81.9 95.5\nTimeSformer-L [4] 82.2 95.6\nViViT-L/16x2 FE 82.9 94.6\nViViT-L/16x2 FE (JFT) 84.3 94.9\nViViT-H/14x2 (JFT) 85.8 96.5\n(c) Moments in Time\nTop 1 Top 5\nTSN [72] 25.3 50.1\nTRN [86] 28.3 53.4\nI3D [8] 29.5 56.1\nblVNet [19] 31.4 59.3\nAssembleNet-101 [54] 34.3 62.7\nViViT-L/16x2 FE 38.5 64.1(d) Epic Kitchens 100 Top 1 accuracy\nMethod Action Verb Noun\nTSN [72] 33.2 60.2 46.0\nTRN [86] 35.3 65.9 45.4\nTBN [36] 36.7 66.0 47.2\nTSM [43] 38.3 67.9 49.0\nSlowFast [21] 38.5 65.6 50.0"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1515-6db4-11ee-adb2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-37", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "TSM [43] 38.3 67.9 49.0\nSlowFast [21] 38.5 65.6 50.0\nViViT-L/16x2 FE 44.0 66.4 56.8\n(e) Something-Something v2\nMethod Top 1 Top 5\nTRN [86] 48.8 77.6\nSlowFast [20, 80] 61.7 \u2013\nTimeSformer-HR [4] 62.5 \u2013\nTSM [43] 63.4 88.5\nSTM [33] 64.2 89.8\nTEA [42] 65.1 \u2013\nblVNet [19] 65.2 90.3\nViVIT-L/16x2 FE 65.9 89.9\n1 2 3 4 5 6 7\nNumber of views767880Top-1 Accuracy\n32 stride 2 64 stride 2 128 stride 2\nFigure 9: The effect of varying the number of frames input to the\nnetwork and increasing the number of tokens proportionally. We\nuse ViViT-L/16x2 Factorised Encoder on Kinetics 400. A Kinetics\nvideo contains 250 frames (10 seconds sampled at 25 fps) and the\naccuracy for each model saturates once the number of equidistant"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1516-6db4-11ee-bd44-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-38", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "video contains 250 frames (10 seconds sampled at 25 fps) and the\naccuracy for each model saturates once the number of equidistant\ntemporal views is suf\ufb01cient to \u201csee\u201d the whole video clip. Observe how models processing more frames (and thus more tokens)\nachieve higher single- and multi-view accuracy.\nFigure 9 shows that as we increase the number of frames\ninput to the network, the accuracy from processing a single view increases, since the network incorporates longer\ntemporal context. However, common practice on datasets\nsuch as Kinetics [21, 75, 42] is to average results over multiple, shorter \u201cviews\u201d of the same video clip. Figure 9 also\nshows that the accuracy saturates once the number of views\nis suf\ufb01cient to cover the whole video. As a Kinetics video\nconsists of 250 frames, and we sample frames with a stride\nof 2, our model which processes 128 frames requires just a\nsingle view to \u201csee\u201d the whole video and achieve its maximum accuarcy.\nNote that we used ViViT-L/16x2 Factorised Encoder\n(Model 2) here. As this model is more ef\ufb01cient it can process more tokens, compared to the unfactorised Model 1\nwhich runs out of memory after 48 frames using tubelet"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1517-6db4-11ee-a5e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-39", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "which runs out of memory after 48 frames using tubelet\nlengtht= 2 and a \u201cLarge\u201d backbone. Models processingmore frames (and thus more tokens) consistently achieve\nhigher single- and multi-view accuracy, in line with our observations in previous experiments (Tab. 5, Fig. 8). Moroever, observe that by processing more frames (and thus\nmore tokens) with Model 2, we are able to achieve higher\naccuracy than Model 1 (with fewer total FLOPs as well).\nFinally, we observed that for Model 2, the number of\nFLOPs effectively increases linearly with the number of input frames as the overall computation is dominated by the\ninitial Spatial Transformer. As a result, the total number\nof FLOPs for the number of temporal views required to\nachieve maximum accuracy is constant across the models.\nIn other words, ViViT-L/16x2 FE with 32 frames requires\n995.3 GFLOPs per view, and 4 views to saturate multi-view\naccuracy. The 128-frame model requires 3980.4 GFLOPs\nbut only a single view. As shown by Fig. 9, the latter model\nachieves the highest accuracy.\n4.3. Comparison to state-of-the-art\nBased on our ablation studies in the previous section,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1518-6db4-11ee-b6af-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-40", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "achieves the highest accuracy.\n4.3. Comparison to state-of-the-art\nBased on our ablation studies in the previous section,\nwe compare to the current state-of-the-art using two of our\nmodel variants. We primarily use our Factorised Encoder\nmodel (Model 2), as it can process more tokens than Model\n1 to achieve higher accuracy.\nKinetics Tables 6a and 6b show that our spatio-temporal\nattention models outperform the state-of-the-art on Kinetics\n400 and 600 respectively. Following standard practice, we\ntake 3 spatial crops (left, centre and right) [21, 20, 66, 75]\nfor each temporal view, and notably, we require signi\ufb01cantly fewer views than previous CNN-based methods.\nWe surpass the previous CNN-based state-of-the-art using ViViT-L/16x2 Factorised Encoder (FE) pretrained on\nImageNet, and also outperform [4] who concurrently proposed a pure-transformer architecture. Moreover, by initialising our backbones from models pretrained on the larger\nJFT dataset [58], we obtain further improvements. Although these models are not directly comparable to previous work, we do also outperform [66] who pretrained on\nthe large-scale, Instagram dataset [44]. Our best model uses"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1519-6db4-11ee-88bf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-41", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "the large-scale, Instagram dataset [44]. Our best model uses\na ViViT-H backbone pretrained on JFT and signi\ufb01cantly advances the best reported results on Kinetics 400 and 600 to\n84.9% and 85.8%, respectively.\nMoments in Time We surpass the state-of-the-art by a\nsigni\ufb01cant margin as shown in Tab. 6c. We note that the\nvideos in this dataset are diverse and contain signi\ufb01cant label noise, making this task challenging and leading to lower\naccuracies than on other datasets.\nEpic Kitchens 100 Table 6d shows that our Factorised\nEncoder model outperforms previous methods by a signi\ufb01cant margin. In addition, our model obtains substantial improvements for Top-1 accuracy of \u201cnoun\u201d classes, and the\nonly method which achieves higher \u201cverb\u201d accuracy used\noptical \ufb02ow as an additional input modality [43, 50]. Furthermore, all variants of our model presented in Tab. 2 outperformed the existing state-of-the-art on action accuracy.\nWe note that we use the same model to predict verbs and\nnouns using two separate \u201cheads\u201d, and for simplicity, we do\nnot use separate loss weights for each head."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d151a-6db4-11ee-ad9d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-42", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "nouns using two separate \u201cheads\u201d, and for simplicity, we do\nnot use separate loss weights for each head.\nSomething-Something v2 (SSv2) Finally, Tab. 6e shows\nthat we achieve state-of-the-art Top-1 accuracy with our\nFactorised encoder model (Model 2), albeit with a smaller\nmargin compared to previous methods. Notably, our Factorised encoder model signi\ufb01cantly outperforms the concurrent TimeSformer [4] method by 2.9%, which also proposes\na pure-transformer model, but does not consider our Factorised encoder variant or our additional regularisation.\nSSv2 differs from other datasets in that the backgrounds\nand objects are quite similar across different classes, meaning that recognising \ufb01ne-grained motion patterns is necessary to distinguish classes from each other. Our results suggest that capturing these \ufb01ne-grained motions is an area of\nimprovement and future work for our model. We also note\nan inverse correlation between the relative performance of\nprevious methods on SSv2 (Tab. 6e) and Kinetics (Tab. 6a)\nsuggesting that these two datasets evaluate complementary\ncharacteristics of a model.\n5. Conclusion and Future Work\nWe have presented four pure-transformer models for"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d151b-6db4-11ee-9d60-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-43", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "suggesting that these two datasets evaluate complementary\ncharacteristics of a model.\n5. Conclusion and Future Work\nWe have presented four pure-transformer models for\nvideo classi\ufb01cation, with different accuracy and ef\ufb01ciency\npro\ufb01les, achieving state-of-the-art results across \ufb01ve popular datasets. Furthermore, we have shown how to effectively regularise such high-capacity models for training\non smaller datasets and thoroughly ablated our main design choices. Future work is to remove our dependence onimage-pretrained models. Finally, going beyond video classi\ufb01cation towards more complex tasks is a clear next step.\nReferences\n[1] Anurag Arnab, Chen Sun, and Cordelia Schmid. Uni\ufb01ed\ngraph structured models for video understanding. In ICCV ,\n2021. 1\n[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.\nLayer normalization. In arXiv preprint arXiv:1607.06450 ,\n2016. 3\n[3] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens,\nand Quoc V Le. Attention augmented convolutional networks. In ICCV , 2019. 1\n[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d151c-6db4-11ee-a311-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-44", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is\nspace-time attention all you need for video understanding?\nInarXiv preprint arXiv:2102.05095 , 2021. 2, 3, 4, 8, 9\n[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini\nAgarwal, et al. Language models are few-shot learners. In\nNeurIPS , 2020. 2\n[6] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han\nHu. Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In CVPR Workshops , 2019. 2\n[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas\nUsunier, Alexander Kirillov, and Sergey Zagoruyko. End-toend object detection with transformers. In ECCV , 2020. 1,\n2\n[8] Joao Carreira and Andrew Zisserman. Quo vadis, action\nrecognition? a new model and the kinetics dataset. In CVPR ,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d151d-6db4-11ee-8125-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-45", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "2\n[8] Joao Carreira and Andrew Zisserman. Quo vadis, action\nrecognition? a new model and the kinetics dataset. In CVPR ,\n2017. 1, 2, 5, 6, 8\n[9] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng\nYan, and Jiashi Feng. A2-nets: Double attention networks.\nInNeurIPS , 2018. 2\n[10] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.\nGenerating long sequences with sparse transformers. In\narXiv preprint arXiv:1904.10509 , 2019. 2\n[11] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter\nHawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser,\net al. Rethinking attention with performers. In ICLR , 2021.\n2\n[12] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V .\nLe. Randaugment: Practical automated data augmentation\nwith a reduced search space. In NeurIPS , 2020. 7, 13, 14"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d151e-6db4-11ee-a190-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-46", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Le. Randaugment: Practical automated data augmentation\nwith a reduced search space. In NeurIPS , 2020. 7, 13, 14\n[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella,\nAntonino Furnari, Jian Ma, Evangelos Kazakos, Davide\nMoltisanti, Jonathan Munro, Toby Perrett, Will Price, and\nMichael Wray. Rescaling egocentric vision. In arXiv\npreprint arXiv:2006.13256 , 2020. 1, 6\n[14] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob\nUszkoreit, and \u0141ukasz Kaiser. Universal transformers. In\nICLR , 2019. 2\n[15] Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab,\nMatthias Minderer, and Yi Tay. Scenic: A JAX library\nfor computer vision research and beyond. arXiv preprint\narXiv:2110.11403 , 2021. 6\n[16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,\nand Li Fei-Fei. Imagenet: A large-scale hierarchical image\ndatabase. In CVPR , 2009. 2, 5, 6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d151f-6db4-11ee-a91e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-47", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "and Li Fei-Fei. Imagenet: A large-scale hierarchical image\ndatabase. In CVPR , 2009. 2, 5, 6\n[17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina\nToutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL , 2019. 2, 3,\n5\n[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov,\nDirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner,\nMostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021. 1, 2,\n3, 5, 7\n[19] Quanfu Fan, Chun-Fu Chen, Hilde Kuehne, Marco Pistoia,\nand David Cox. More is less: Learning ef\ufb01cient video representations by big-little network and depthwise temporal aggregation. In NeurIPS , 2019. 8\n[20] Christoph Feichtenhofer. X3d: Expanding architectures for\nef\ufb01cient video recognition. In CVPR , 2020. 1, 2, 6, 8"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1520-6db4-11ee-b106-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-48", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "ef\ufb01cient video recognition. In CVPR , 2020. 1, 2, 6, 8\n[21] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and\nKaiming He. Slowfast networks for video recognition. In\nICCV , 2019. 1, 6, 8\n[22] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes.\nSpatiotemporal residual networks for video action recognition. In NeurIPS , 2016. 2, 5, 6\n[23] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In CVPR , 2019.\n1\n[24] Rohit Girdhar and Deva Ramanan. Attentional pooling for\naction recognition. In NeurIPS , 2017. 4\n[25] Xavier Glorot and Yoshua Bengio. Understanding the dif\ufb01culty of training deep feedforward neural networks. In AISTATS , 2010. 6\n[26] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim,\nValentin Haenel, Ingo Fruend, Peter Yianilos, Moritz"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1521-6db4-11ee-9fdd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-49", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz\nMueller-Freitag, et al. The\u201d something something\u201d video\ndatabase for learning and evaluating visual common sense.\nInICCV , 2017. 1, 6\n[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.\nDeep residual learning for image recognition. In CVPR ,\n2016. 1, 2, 5\n[28] Dan Hendrycks and Kevin Gimpel. Gaussian error linear\nunits (gelus). In arXiv preprint arXiv:1606.08415 , 2016. 3\n[29] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim\nSalimans. Axial attention in multidimensional transformers.\nInarXiv preprint arXiv:1912.12180 , 2019. 4\n[30] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation networks. In CVPR , 2018. 2\n[31] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian\nWeinberger. Deep networks with stochastic depth. In ECCV ,\n2016. 7, 13, 14"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1522-6db4-11ee-b3c4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-50", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Weinberger. Deep networks with stochastic depth. In ECCV ,\n2016. 7, 13, 14\n[32] Zilong Huang, Xinggang Wang, Lichao Huang, Chang\nHuang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross\nattention for semantic segmentation. In ICCV , 2019. 2\n[33] Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and\nJunjie Yan. Stm: Spatiotemporal and motion encoding for\naction recognition. In ICCV , 2019. 8[34] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas\nLeung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video\nclassi\ufb01cation with convolutional neural networks. In CVPR ,\n2014. 2, 4\n[35] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,\nChloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola,\nTim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. In arXiv preprint\narXiv:1705.06950 , 2017. 1, 2, 5, 6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1523-6db4-11ee-b2a1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-51", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "arXiv:1705.06950 , 2017. 1, 2, 5, 6\n[36] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and\nDima Damen. Epic-fusion: Audio-visual temporal binding\nfor egocentric action recognition. In ICCV , 2019. 8\n[37] Nikita Kitaev, \u0141ukasz Kaiser, and Anselm Levskaya. Reformer: The ef\ufb01cient transformer. In ICLR , 2020. 2\n[38] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.\nImagenet classi\ufb01cation with deep convolutional neural networks. In NeurIPS , volume 25, 2012. 1, 2\n[39] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan\nPopov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Uni\ufb01ed image classi\ufb01cation, object detection, and visual relationship detection at scale. IJCV , 2020.\n5\n[40] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1524-6db4-11ee-b4ca-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-52", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "5\n[40] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin\nGimpel, Piyush Sharma, and Radu Soricut. Albert: A lite\nbert for self-supervised learning of language representations.\nInICLR , 2020. 2\n[41] Ivan Laptev. On space-time interest points. IJCV , 64(2-3),\n2005. 2\n[42] Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and\nLimin Wang. Tea: Temporal excitation and aggregation for\naction recognition. In CVPR , 2020. 8\n[43] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift\nmodule for ef\ufb01cient video understanding. In ICCV , 2019. 8,\n9\n[44] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan,\nKaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe,\nand Laurens Van Der Maaten. Exploring the limits of weakly\nsupervised pretraining. In ECCV , 2018. 8, 9\n[45] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1525-6db4-11ee-b726-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-53", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "[45] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown,\nQuanfu Fan, Dan Gutfreund, Carl V ondrick, et al. Moments\nin time dataset: one million videos for event understanding.\nPAMI , 42(2):502\u2013508, 2019. 1, 6\n[46] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In arXiv preprint\narXiv:2102.00719 , 2021. 2, 4\n[47] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George\nToderici. Beyond short snippets: Deep networks for video\nclassi\ufb01cation. In CVPR , 2015. 2\n[48] Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable visual transformers with hierarchical pooling. In arXiv preprint arXiv:2103.10619 , 2021. 2\n[49] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1526-6db4-11ee-aa55-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-54", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "[49] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz\nKaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In ICML , 2018. 1, 2\n[50] Will Price and Dima Damen. An evaluation of action\nrecognition models on epic-kitchens. In arXiv preprint\narXiv:1908.00867 , 2019. 9\n[51] Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Xinmei Tian, and\nTao Mei. Learning spatio-temporal representation with local\nand global diffusion. In CVPR , 2019. 8\n[52] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee,\nSharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and\nPeter J Liu. Exploring the limits of transfer learning with a\nuni\ufb01ed text-to-text transformer. JMLR , 2020. 2\n[53] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan\nBello, Anselm Levskaya, and Jonathon Shlens. Stand-alone\nself-attention in vision models. In NeurIPS , 2019. 1, 2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1527-6db4-11ee-bee7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-55", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone\nself-attention in vision models. In NeurIPS , 2019. 1, 2\n[54] Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia\nAngelova. Assemblenet: Searching for multi-stream neural\nconnectivity in video architectures. In ICLR , 2020. 8\n[55] Zhuoran Shen, Irwan Bello, Raviteja Vemulapalli, Xuhui Jia,\nand Ching-Hui Chen. Global self-attention networks for image recognition. In arXiv preprint arXiv:2010.03019 , 2021.\n2\n[56] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In\nNeurIPS , 2014. 2, 4\n[57] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon\nShlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck\ntransformers for visual recognition. In CVPR , 2021. 2\n[58] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1528-6db4-11ee-8145-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-56", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "[58] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in\ndeep learning era. In ICCV , 2017. 5, 6, 9\n[59] Lin Sun, Kui Jia, Dit-Yan Yeung, and Bertram E Shi. Human\naction recognition using factorized spatio-temporal convolutional networks. In ICCV , 2015. 2\n[60] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,\nScott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent\nVanhoucke, and Andrew Rabinovich. Going deeper with\nconvolutions. In CVPR , 2015. 1\n[61] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon\nShlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR , 2016. 7, 13, 14\n[62] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen,\nDara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A\nbenchmark for ef\ufb01cient transformers. In arXiv preprint"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1529-6db4-11ee-8f92-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-57", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "benchmark for ef\ufb01cient transformers. In arXiv preprint\narXiv:2011.04006 , 2020. 2\n[63] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Ef\ufb01cient transformers: A survey. In arXiv preprint\narXiv:2009.06732 , 2020. 2\n[64] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco\nMassa, Alexandre Sablayrolles, and Herv \u00b4e J\u00b4egou. Training\ndata-ef\ufb01cient image transformers & distillation through attention. In arXiv preprint arXiv:2012.12877 , 2020. 1, 2, 7\n[65] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani,\nand Manohar Paluri. Learning spatiotemporal features with\n3d convolutional networks. In ICCV , 2015. 2\n[66] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classi\ufb01cation with channel-separated convolutional networks. In ICCV , 2019. 2, 8, 9[67] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d152a-6db4-11ee-81a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-58", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "LeCun, and Manohar Paluri. A closer look at spatiotemporal\nconvolutions for action recognition. In CVPR , 2018. 2\n[68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141ukasz Kaiser, and Illia\nPolosukhin. Attention is all you need. In NeurIPS , 2017. 1,\n2, 3, 4, 5, 7\n[69] Heng Wang, Alexander Kl \u00a8aser, Cordelia Schmid, and\nCheng-Lin Liu. Dense trajectories and motion boundary descriptors for action recognition. IJCV , 103(1), 2013. 2\n[70] Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli.\nVideo modeling with correlation networks. In CVPR , 2020.\n8\n[71] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and\nLiang-Chieh Chen. Max-deeplab: End-to-end panoptic\nsegmentation with mask transformers. In arXiv preprint\narXiv:2012.00759 , 2020. 2\n[72] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d152b-6db4-11ee-8d08-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-59", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "arXiv:2012.00759 , 2020. 2\n[72] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua\nLin, Xiaoou Tang, and Luc Van Gool. Temporal segment\nnetworks: Towards good practices for deep action recognition. In ECCV , 2016. 4, 8\n[73] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and\nHao Ma. Linformer: Self-attention with linear complexity.\nInarXiv preprint arXiv:2006.04768 , 2020. 2\n[74] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao\nSong, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.\nPyramid vision transformer: A versatile backbone for\ndense prediction without convolutions. In arXiv preprint\narXiv:2102.12122 , 2021. 2\n[75] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR , 2018. 1, 2,\n8\n[76] Xiaofang Wang, Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d152c-6db4-11ee-afa5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-60", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "[76] Xiaofang Wang, Xuehan Xiong, Maxim Neumann, AJ Piergiovanni, Michael S Ryoo, Anelia Angelova, Kris M Kitani,\nand Wei Hua. Attentionnas: Spatiotemporal attention cell\nsearch for video classi\ufb01cation. In ECCV , 2020. 8\n[77] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen,\nBaoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end\nvideo instance segmentation with transformers. In arXiv\npreprint arXiv:2011.14503 , 2020. 2\n[78] Dirk Weissenborn, Oscar T \u00a8ackstr \u00a8om, and Jakob Uszkoreit.\nScaling autoregressive video models. In ICLR , 2020. 4\n[79] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term\nfeature banks for detailed video understanding. In CVPR ,\n2019. 1\n[80] Chao-Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d152d-6db4-11ee-9135-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-61", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "[80] Chao-Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method\nfor ef\ufb01ciently training video models. In CVPR , 2020. 8\n[81] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and\nKevin Murphy. Rethinking spatiotemporal feature learning:\nSpeed-accuracy trade-offs in video classi\ufb01cation. In ECCV ,\n2018. 2\n[82] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and\nDavid Lopez-Paz. Mixup: Beyond empirical risk minimization. In ICLR , 2018. 7, 13, 14\n[83] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In CVPR , 2020. 2\n[84] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and\nVladlen Koltun. Point transformer. In arXiv preprint\narXiv:2012.09164 , 2020. 2\n[85] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d152e-6db4-11ee-be5b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-62", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "arXiv:2012.09164 , 2020. 2\n[85] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu,\nZekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao\nXiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In arXiv preprint arXiv:2012.15840 , 2020. 2\n[86] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In ECCV ,\n2018. 4, 8\nAppendix\nA. Additional experimental details\nIn this appendix, we provide additional experimental details. Section A.1 provides additional details about the regularisers we used and Sec. A.2 details the training hyperparamters used for our experiments.\nA.1. Further details about regularisers\nIn this section, we provide additional details and list the\nhyperparameters of the additional regularisers that we employed in Tab. 4. Hyperparameter values for all our experiments are listed in Tab. 7.\nStochastic depth Stochastic depth regularisation was\noriginally proposed for training very deep residual networks [31]. Intuitively, the outputs of a layer, `, are"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d152f-6db4-11ee-acdf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-63", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Stochastic depth Stochastic depth regularisation was\noriginally proposed for training very deep residual networks [31]. Intuitively, the outputs of a layer, `, are\n\u201cdropped out\u201d with probability, pdrop(`)during training, by\nsetting the output of the layer to be equal to its input.\nFollowing [31], we linearly increase the probability of\ndropping a layer according to its depth within the network,\npdrop(`) =`\nLpdrop; (10)\nwhere`is the index of the layer in the network, and Lis the\ntotal number of layers.\nRandom augment Random augment [12] randomly applies data augmentation transformations sequentially to an\ninput example. We follow the public implementation1, but\nmodify the data augmentation operations to be temporally\nconsistent throughout the video (in other words, the same\ntransformation is applied on each frame of the video).\nThe authors de\ufb01ne two hyperparameters for Random\naugment, \u201cnumber of layers\u201d , the number of augmentation\ntransformations to apply sequentially to a video and \u201cmagnitude\u201d, the strength of the transformation that is shared\nacross all augmentation operations. Our values for these\nparameters are shown in Tab. 7.\nLabel smoothing Label smoothing was proposed by [61]"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1530-6db4-11ee-88ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-64", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "across all augmentation operations. Our values for these\nparameters are shown in Tab. 7.\nLabel smoothing Label smoothing was proposed by [61]\noriginally to regularise training Inception-v3. Concretely,\nthe label distribution used during training, ~y, is a mixture\nof the one-hot ground-truth label, y, and a uniform distribution,u, to encourage the network to produce less con\ufb01dent\npredictions during training:\n~y= (1\u0000\u0015)y+\u0015u: (11)\nThere is therefore one scalar hyperparamter, \u00152[0;1].\n1https://github.com/tensorflow/models/blob/\nmaster/official/vision/beta/ops/augment.pyMixup Mixup [82] constructs virtual training examples\nwhich are a convex combination of pairs of training examples and their labels. Concretely, given (xi;yi)and(xj;yj)\nwherexidenotes an input vector and yia one-hot input label, mixup constructs the virtual training example,\n~x=\u0015xi+ (1\u0000\u0015)xj\n~y=\u0015yi+ (1\u0000\u0015)yj: (12)\n\u00152[0;1], and is sampled from a Beta distribution,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1531-6db4-11ee-9950-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-65", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "~y=\u0015yi+ (1\u0000\u0015)yj: (12)\n\u00152[0;1], and is sampled from a Beta distribution,\nBeta(\u000b;\u000b). Our choice of the hyperparameter \u000bis detailed\nin Tab. 7.\nA.2. Training hyperparameters\nTable 7 details the hyperparamters for all of our experiments. We use synchronous SGD with momentum, a\ncosine learning rate schedule with linear warmup, and a\nbatch size of 64 for all experiments. As aforementioned,\nwe only employed additional regularisation when training\non the smaller Epic Kitchens and Something-Something v2\ndatasets.\nTable 7: Training hyperparamters for experiments in the main paper. \u201c\u2013\u201d indicates that the regularisation method was not used at all. Values\nwhich are constant across all columns are listed once. Datasets are denoted as follows: K400: Kinetics 400. K600: Kinetics 600. MiT:\nMoments in Time. EK: Epic Kitchens. SSv2: Something-Something v2.\nK400 K600 MiT EK SSv2\nOptimisation\nOptimiser Synchronous SGD\nMomentum 0.9\nBatch size 64\nLearning rate schedule cosine with linear warmup\nLinear warmup epochs 2.5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1532-6db4-11ee-b576-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2103.15691-66", "title": "ViViT: A Video Vision Transformer", "source": "http://arxiv.org/pdf/2103.15691", "start_index": 0, "chroma:document": "Optimiser Synchronous SGD\nMomentum 0.9\nBatch size 64\nLearning rate schedule cosine with linear warmup\nLinear warmup epochs 2.5\nBase learning rate 0.1 0.1 0.25 0.5 0.5\nEpochs 30 30 10 50 35\nData augmentation\nRandom crop probability 1.0\nRandom \ufb02ip probability 0.5\nScale jitter probability 1.0\nMaximum scale 1.33\nMinimum scale 0.9\nColour jitter probability 0.8 0.8 0.8 \u2013 \u2013\nRand augment number of layers [12] \u2013 \u2013 \u2013 2 2\nRand augment magnitude [12] \u2013 \u2013 \u2013 15 20\nOther regularisation\nStochastic droplayer rate, pdrop[31] \u2013 \u2013 \u2013 0.2 0.3\nLabel smoothing \u0015[61] \u2013 \u2013 \u2013 0.2 0.3\nMixup\u000b[82] \u2013 \u2013 \u2013 0.1 0.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1533-6db4-11ee-b2eb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-0", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "2023-2-3\nAccelerating Large Language Model Decoding\nwith Speculative Sampling\nCharlie Chen1, Sebastian Borgeaud1, Geo\ufb00rey Irving1, Jean-Baptiste Lespiau1, Laurent Sifre1and John\nJumper1\n1All authors from DeepMind\nWe present speculative sampling, an algorithm for accelerating transformer decoding by enabling the\ngeneration of multiple tokens from each transformer call. Our algorithm relies on the observation that\nthe latency of parallel scoring of short continuations, generated by a faster but less powerful draft\nmodel, is comparable to that of sampling a single token from the larger target model. This is combined\nwith a novel modi\ufb01ed rejection sampling scheme which preserves the distribution of the target model\nwithinhardwarenumerics. WebenchmarkspeculativesamplingwithChinchilla,a70billionparameter\nlanguage model, achieving a 2\u20132\u00935\u0002decoding speedup in a distributed setup, without compromising\nthe sample quality or making modi\ufb01cations to the model itself.\nIntroduction\nScaling transformer models to 500B+ parameters has led to large performance improvements on\nmany natural language, computer vision and reinforcement learning tasks (Arnab et al., 2021; Brown"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1534-6db4-11ee-8171-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-1", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "Introduction\nScaling transformer models to 500B+ parameters has led to large performance improvements on\nmany natural language, computer vision and reinforcement learning tasks (Arnab et al., 2021; Brown\netal.,2020;Chowdheryetal.,2022;Dosovitskiyetal.,2020;Ho\ufb00mannetal.,2022;Raeetal.,2021).\nHowever, transformer decoding remains a highly costly and ine\ufb03cient process in this regime.\nTransformer sampling is typically memory bandwidth bound (Shazeer, 2019), so for a given set of\nhardware, the time to generate a single token in transformer models is proportional to a \ufb01rst order\napproximationtothesizeofparametersandthesizeofthetransformermemory. Thesizeoflanguage\nmodels also necessitates serving with model parallelism \u2013 adding communication overheads (Pope\net al., 2022) and multiplying resource requirements. Since each new token depends on the past,\nmany such transformer calls are required to sample a new sequence.\nWe present an algorithm to accelerate transformer sampling for latency critical applications, which\nwe call speculative sampling (SpS). This is achieved by:\n1.Generating a short draft of length \ud835\udc3e. This can be attained with either a parallel model (Stern"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1535-6db4-11ee-a0f9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-2", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "we call speculative sampling (SpS). This is achieved by:\n1.Generating a short draft of length \ud835\udc3e. This can be attained with either a parallel model (Stern\net al., 2018) or by calling a faster, auto-regressive model \ud835\udc3etimes. We shall refer to this model\nas thedraft model , and focus on the case where it is auto-regressive.\n2.Scoring the draft using the larger, more powerful model from we wish to sample from. We shall\nrefer to this model as the target model .\n3.Using a modi\ufb01ed rejection sampling scheme, accept a subset of the \ud835\udc3edraft tokens from left to\nright, recovering the distribution of the target model in the process.\nIntuitively, there are often sequences where the next token might be \u201cobvious\u201d. Therefore, if there is\nstrongagreementbetweenthedraftandtargetmodel\u2019sdistributionsonagiventokenorsub-sequence\nof tokens, this setup permits the generation of multiple tokens each time the target model is called .\nWe show that the expected acceptance rate of draft tokens is su\ufb03cient to o\ufb00set the overhead of the\nCorresponding author(s): ccharlie@deepmind.com"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1536-6db4-11ee-820c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-3", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "Corresponding author(s): ccharlie@deepmind.com\n\u00a92023 DeepMind. All rights reservedarXiv:2302.01318v1  [cs.CL]  2 Feb 2023\nAccelerating Large Language Model Decoding with Speculative Sampling\ndraftingprocessforlargelanguagemodels,resultinginane\ufb00ectiveandpracticalmethodforreducing\nsampling latency without the need for modifying the target model or biasing the sample distribution.\nDepending on the evaluation domain, SpS leads to a 2\u20132\u00935\u0002speedup when sampling from Chinchilla\n(Ho\ufb00mann et al., 2022). Notably, the mean tokens per second with SpS often exceeds the idealised\nceiling on auto-regressive sampling speed imposed by the memory bandwidth.\nRelated Work\nThere has been a substantial body of work focused on improving sampling latency of large transformers and other auto-regressive models.\nSince sampling performance is heavily coupled with the model size in memory, quantisation to\nint8or even int4(Dettmers et al., 2022; Yao et al., 2022) and distillation (Jiao et al., 2020; Sanh\net al., 2019) of transformers are e\ufb00ective techniques for reducing sampling latency with little to no"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1537-6db4-11ee-8c6b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-4", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "et al., 2019) of transformers are e\ufb00ective techniques for reducing sampling latency with little to no\nperformance penalty. The observation that model size contributes less to the \ufb01nal performance than\nexpected (Ho\ufb00mann et al., 2022) has also encouraged smaller language models in general.\nDuring sampling, a cache of the keys and values is maintained for every attention layer, and could\nbecome a memory bandwidth bottleneck as the batch size increases. Methods such as multi-query\nattention (Shazeer, 2019) aims to improve sampling performance by shrinking this cache. However\nthesetechniquesaremoste\ufb00ectiveatmaximisingthroughout(atlargerbatchsizes)insteadoflatency,\nespecially for larger models where the majority of the memory bandwidth budget is consumed by the\nparameters.\nUsing a combination of the above techniques, in addition to a number of low-level optimisations to\nTPUs, Pope et al. (2022) have greatly improved the serving latency and e\ufb03ciency of PaLM 540B.\nThere is an existing body of similar work exploiting the e\ufb03ciency of transformers and sequence\nmodels operating in parallel. This includes block parallel sampling (Stern et al., 2018), aggressive\ndecoding (Ge et al., 2022), in addition to some work in parallelizing autoregressive models in the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1538-6db4-11ee-b8f6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-5", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "decoding (Ge et al., 2022), in addition to some work in parallelizing autoregressive models in the\nimage domain (Song et al., 2021; Wiggers and Hoogeboom, 2020). These methods have yet to be\nadapted to typical language model use-cases since they either only work with greedy sampling, bias\nthe results or are focused on other modalities. Further, to our knowledge none of these techniques\nhave been scaled to distributed setups, which is necessary for the most expensive decoders with the\ntens or hundreds of billions of parameters.\nCoincidentally, the work in this manuscript was undertaken concurrently and independently of\nthe work on speculative decoding from Leviathan et al. (2022). We focus more heavily the distributed\nserving setting for large models and o\ufb00er some incremental optimisations, but otherwise the core\nunderlying idea is the same.\nAuto-regressive Sampling\nWhilst transformers can be trained e\ufb03ciently and in parallel on TPUs and GPUs, samples are typically\ndrawn auto-regressively (See algorithm 1). For most applications, auto-regressive sampling (ArS) is\nhighlymemorybandwidthboundandthuscannotmakee\ufb00ectiveuseofmodernacceleratorhardware\n(Shazeer, 2019). A memory bound model call only generates a single token for every sequence in the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1539-6db4-11ee-ac7d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-6", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "(Shazeer, 2019). A memory bound model call only generates a single token for every sequence in the\nbatch, hence generating multiple tokens introduces a large amount of latency in any system which\nmakes use of it.\n2\nAccelerating Large Language Model Decoding with Speculative Sampling\nThisisespeciallyproblematicasthenumberofparameters inthemodelincreases. Sinceallthemodel\nparameters need to pass through at least one accelerator chip, the model size divided by the total\nmemory bandwidth across all chips gives us a hard ceiling on the maximum auto-regressive sampling\nspeed. Larger models also require serving on multiple accelerators, introducing a further source of\nlatency due to inter-device communication overheads.\nAlgorithm 1 Auto-regressive (ArS) with Auto-Regressive Models\nGivenauto-regressivetargetmodel \ud835\udc5e\u00b9\u0093j\u0093\u00baandinitialpromptsequence \ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc61andtargetsequence\nlength\ud835\udc47.\nInitialise\ud835\udc5b \ud835\udc61.\nwhile\ud835\udc5b \u009d\ud835\udc47do\nSample\ud835\udc65\ud835\udc5b\u00b81\u0018\ud835\udc5e\u00b9\ud835\udc65j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00ba"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d153a-6db4-11ee-97ff-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-7", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "\ud835\udc5b \ud835\udc5b\u00b81\nend while\nAlgorithm 2 Speculative Sampling (SpS) with Auto-Regressive Target and Draft Models\nGiven lookahead \ud835\udc3eand minimum target sequence length \ud835\udc47.\nGiven auto-regressive target model \ud835\udc5e\u00b9\u0093j\u0093\u00ba, and auto-regressive draft model \ud835\udc5d\u00b9\u0093j\u0093\u00ba, initial prompt\nsequence\ud835\udc650\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc61.\nInitialise\ud835\udc5b \ud835\udc61.\nwhile\ud835\udc5b \u009d\ud835\udc47do\nfor\ud835\udc61=1 :\ud835\udc3edo\nSample draft auto-regressively \u02dc\ud835\udc65\ud835\udc61\u0018\ud835\udc5d\u00b9\ud835\udc65j\u0094\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u0094\u02dc\ud835\udc651\u0094\u0093\u0093\u0093\u0094 \u02dc\ud835\udc65\ud835\udc61\u00001\u00ba\nend for\nIn parallel, compute \ud835\udc3e\u00b81sets of logits from drafts \u02dc\ud835\udc651\u0094\u0093\u0093\u0093\u0094 \u02dc\ud835\udc65\ud835\udc3e:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d153b-6db4-11ee-9949-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-8", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "\ud835\udc5e\u00b9\ud835\udc65j\u0094\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00ba\u0094 \ud835\udc5e\u00b9\ud835\udc65j\u0094\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u0094\u02dc\ud835\udc651\u00ba\u0094 \u0093\u0093\u0093\u0094 \ud835\udc5e\u00b9\ud835\udc65j\u0094\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u0094\u02dc\ud835\udc651\u0094\u0093\u0093\u0093\u0094 \u02dc\ud835\udc65\ud835\udc3e\u00ba\nfor\ud835\udc61=1 :\ud835\udc3edo\nSample\ud835\udc5f\u0018\ud835\udc48\u00bb0\u00941\u00bcfrom a uniform distribution.\nif\ud835\udc5f \u009dmin\u0010\n1\u0094\ud835\udc5e\u00b9\ud835\udc65j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00b8\ud835\udc61\u00001\u00ba\n\ud835\udc5d\u00b9\ud835\udc65j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00b8\ud835\udc61\u00001\u00ba\u0011\n,then\nSet\ud835\udc65\ud835\udc5b\u00b8\ud835\udc61 \u02dc\ud835\udc65\ud835\udc61and\ud835\udc5b \ud835\udc5b\u00b81."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d153c-6db4-11ee-891e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-9", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": ",then\nSet\ud835\udc65\ud835\udc5b\u00b8\ud835\udc61 \u02dc\ud835\udc65\ud835\udc61and\ud835\udc5b \ud835\udc5b\u00b81.\nelse\nsample\ud835\udc65\ud835\udc5b\u00b8\ud835\udc61\u0018\u00b9\ud835\udc5e\u00b9\ud835\udc65j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00b8\ud835\udc61\u00001\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc65j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00b8\ud835\udc61\u00001\u00ba\u00ba\u00b8and exit for loop.\nend if\nend for\nIf all tokens \ud835\udc65\ud835\udc5b\u00b81\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00b8\ud835\udc3eare accepted, sample extra token \ud835\udc65\ud835\udc5b\u00b8\ud835\udc3e\u00b81\u0018\ud835\udc5e\u00b9\ud835\udc65j\u0094\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u0094\ud835\udc65\ud835\udc5b\u00b8\ud835\udc3e\u00baand\nset\ud835\udc5b \ud835\udc5b\u00b81.\nend while\nSpeculative Sampling\nConditional Scoring\nFor speculative sampling (See algorithm 2), we \ufb01rst make the observation that computing the logits"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d153d-6db4-11ee-ba88-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-10", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "end while\nSpeculative Sampling\nConditional Scoring\nFor speculative sampling (See algorithm 2), we \ufb01rst make the observation that computing the logits\nof a short continuation of \ud835\udc3etokens in parallel has a very similar latency to that of sampling a single\n3\nAccelerating Large Language Model Decoding with Speculative Sampling\ntoken. We focus our attention on large transformers, sharded in the Megatron style (Shoeybi et al.,\n2019). For these models the majority of sampling time can be attributed to three components:\n1.Linear Layers: For small batch sizes, each linear layer only processes a small number of\nembeddings. This causes the dense matrix multiplies in the feed-forward layers, queries, keys,\nvalues computations and the \ufb01nal attention projection to become memory bound. For small \ud835\udc3e,\nthis will continue to be memory bound and therefore take a similar amount of time.\n2.The Attention Mechanism: Theattentionmechanismisalsomemorybound. Duringsampling,\nwe maintain a cache of all the keys and values of the previous tokens in the sequence to avoid\nre-computation. These KV-caches are large, and accounts for the majority of the memory"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d153e-6db4-11ee-aa54-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-11", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "re-computation. These KV-caches are large, and accounts for the majority of the memory\nbandwidth utilisation for the attention mechanism. However, since the KV-cache size does not\nchange as we increase \ud835\udc3e, there is little to no delta in this component.\n3.All-reduces: Asmodelsgrowinsize, itsparametersneedtobedividedacrossmultipleaccelerators, leading to communication overheads. With Megatron, this manifests itself as an all-reduce\nafter every feed-forward and attention layer. Since only the activations for a small number of\ntokens are transmitted, this operation is typically latency bound instead of throughput bound\nfor both sampling and scoring (for small \ud835\udc3e). Again, this results in a similar amount of time\nspent in the two cases.\nOther sources of overhead may exist, depending on the exact transformer implementation. Therefore\nit is still possible that the choice of positioning encoding, decoding method (e.g. a sort might be\nrequired for nucleus sampling), hardware limitations etc. can introduce some deltas between scoring\nand sampling. However, if the conditions are met such that the above components dominate then\nscoring should not be signi\ufb01cantly slower for small \ud835\udc3e.\nModi\ufb01ed Rejection Sampling"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d153f-6db4-11ee-b228-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-12", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "scoring should not be signi\ufb01cantly slower for small \ud835\udc3e.\nModi\ufb01ed Rejection Sampling\nWe require a method to recover the distribution of the target model from samples from the draft\nmodel, and logits of said tokens from both models.\nTo achieve this, we introduce the following rejection sampling scheme of the drafted tokens. Given a\nsequence of tokens \ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b, and\ud835\udc3edraft tokens \u02dc\ud835\udc65\ud835\udc5b\u00b81\u0094\u0093\u0093\u0093\u0094 \u02dc\ud835\udc65\ud835\udc5b\u00b8\ud835\udc3egenerated from \ud835\udc5d\u00b9\u0093j\u0093\u00ba, we accept \u02dc\ud835\udc65\ud835\udc5b\u00b81\nwith probability:\nmin\u0012\n1\u0094\ud835\udc5e\u00b9\u02dc\ud835\udc65\ud835\udc5b\u00b81j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00ba\n\ud835\udc5d\u00b9\u02dc\ud835\udc65\ud835\udc5b\u00b81j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00ba\u0013"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1540-6db4-11ee-8f4b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-13", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "\ud835\udc5d\u00b9\u02dc\ud835\udc65\ud835\udc5b\u00b81j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00ba\u0013\nWhere\ud835\udc5e\u00b9\u02dc\ud835\udc65\ud835\udc5b\u00b81j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00baand\ud835\udc5d\u00b9\u02dc\ud835\udc65\ud835\udc5b\u00b81j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00baare the probability of \u02dc\ud835\udc65\ud835\udc5b\u00b81according to the target and\ndraft models respectively, conditioned on the context so far.\nIf the token is accepted, we set \ud835\udc65\ud835\udc5b\u00b81 \u02dc\ud835\udc65\ud835\udc5b\u00b81and repeat the process for \u02dc\ud835\udc65\ud835\udc5b\u00b82until either a token\nis rejected or all tokens have been accepted.\nIf\u02dc\ud835\udc65\ud835\udc5b\u00b81is rejected, we resample \ud835\udc65\ud835\udc5b\u00b81from the following distribution:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1541-6db4-11ee-b3d7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-14", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "If\u02dc\ud835\udc65\ud835\udc5b\u00b81is rejected, we resample \ud835\udc65\ud835\udc5b\u00b81from the following distribution:\n\ud835\udc65\ud835\udc5b\u00b81\u0018\u00b9\ud835\udc5e\u00b9\ud835\udc65j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc65j\ud835\udc651\u0094\u0093\u0093\u0093\u0094\ud835\udc65\ud835\udc5b\u00ba\u00ba\u00b8\nWhere\u00b9\u0093\u00ba\u00b8denotes:\n\u00b9\ud835\udc53\u00b9\ud835\udc65\u00ba\u00ba\u00b8=max\u00b90\u0094 \ud835\udc53\u00b9\ud835\udc65\u00ba\u00ba\u00cd\n\ud835\udc65max\u00b90\u0094 \ud835\udc53\u00b9\ud835\udc65\u00ba\u00ba\nBy applying this sequentially, we recover the distribution of the target model for the accepted tokens\n(see proof in Theorem 1) within hardware numerics. Note that:\n4\nAccelerating Large Language Model Decoding with Speculative Sampling\n\u2022At least one token will always be generated from a draft-accept loop \u2013 if the \ufb01rst token is\nrejected, a valid token is resampled.\n\u2022Since the \ufb01nal token of the draft gives us the logits for the next token, if every drafted token is"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1542-6db4-11ee-91fc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-15", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "rejected, a valid token is resampled.\n\u2022Since the \ufb01nal token of the draft gives us the logits for the next token, if every drafted token is\naccepted, we can sample from it normally. This gives us a maximum of \ud835\udc3e\u00b81tokens per loop,\nover the naive implementation which would only return \ud835\udc3etokens.\nWith standard sampling methods such as nucleus, top-k sampling and adjusting temperature, we\ncan modify the probabilities accordingly before applying this rejection sampling scheme. We have\nobserved that the overall acceptance rate is robust to the exact parameters used.\nBecause we do not interact with the body of the transformer itself, this method can be used in\nconjunction many other techniques for accelerating or optimising the memory use of sampling, such\nas quantisation and multi-query attention.\nChoice of Draft Models\nSince the acceptance criterion guarantees the distribution of the target model in our samples, we are\nfree to choose the method for drafting a continuation as long as it exposes logits, and there is a high\nenough acceptance rate and/or low enough latency to break-even. There exist several approaches\nhere:\n\u2022Incorporating draft generation into the target model, and train the model from the start. This\nis the strategy used by Stern et al. (2018), which adds multiple heads into the transformer to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1543-6db4-11ee-bda0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-16", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "is the strategy used by Stern et al. (2018), which adds multiple heads into the transformer to\ngenerate multiple tokens.\n\u2022Using sequence level distillation (Kim and Rush, 2016) to generate a second model which\npredicts\ud835\udc3etokens in parallel. This strategy was employed by Ge et al. (2022).\n\u2022Set a portion of the activations of the target model as an input to the draft model, and train the\ndraft model with this input.\nAlthough these methods will likely yield powerful drafts, they require a large number of data generated from the target model or changes to the target model. Sequence level distillation in particular\nwould require a large compute budget. This makes them less practical for large scale applications.\nWhilst large language models produce better samples, intuitively there are \"easier\" tokens to predict\nfor which smaller models may be su\ufb03cient. Therefore we may simply use a smaller version of the\ntarget language model as the draft and obtain high acceptance rates. This would also be convenient\nfrom an engineering and work\ufb02ow perspective, since robust tooling for such models should already\nexist to train the target model in the \ufb01rst place.\nResults\nWe train a 4 billion parameter draft model optimised for sampling latency on 16 TPU v4s \u2013 the same\nhardware that is typically used to serve Chinchilla for research purposes. This model was trained"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1544-6db4-11ee-b232-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-17", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "hardware that is typically used to serve Chinchilla for research purposes. This model was trained\nwith the same tokeniser and dataset as Chinchilla, with a slightly smaller width and with only 8\nlayers. The relatively few number of layers allows it to achieve a sampling speed of 1.8ms/token\ncomparedto 14.1ms/token forChinchilla. Fordetails,pleaserefertothehyperparametersinTable2.\nFor distributed setups it is insu\ufb03cient to naively choose a small model as the draft, since di\ufb00erent models have di\ufb00erent optimal inference setups. For example, it is typical to serve Chinchilla 70B\n5\nAccelerating Large Language Model Decoding with Speculative Sampling\nTable1jChinchillaperformanceandspeedonXSumandHumanEvalwithnaiveandspeculative\nsampling at batch size 1 and \ud835\udc3e=4. XSum was executed with nucleus parameter \ud835\udc5d=0\u00938, and\nHumanEval with \ud835\udc5d=0\u009395and temperature 0\u00938.\nSampling Method Benchmark Result Mean Token Time Speed Up\nArS (Nucleus)XSum (ROUGE-2)0.112 14.1ms/Token 1\u0002"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1545-6db4-11ee-af5e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-18", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "Sampling Method Benchmark Result Mean Token Time Speed Up\nArS (Nucleus)XSum (ROUGE-2)0.112 14.1ms/Token 1\u0002\nSpS (Nucleus) 0.114 7.52ms/Token 1\u009392\u0002\nArS (Greedy)XSum (ROUGE-2)0.157 14.1ms/Token 1\u0002\nSpS (Greedy) 0.156 7.00ms/Token 2\u009301\u0002\nArS (Nucleus)HumanEval (100 Shot)45.1% 14.1ms/Token 1\u0002\nSpS (Nucleus) 47.0% 5.73ms/Token 2\u009346\u0002\non 16 TPU v4s (where it achieves the aforementioned 14.1ms/token ), whereas a chinchilla-optimal\n7B achieves its lowest sampling latency on 4 TPU v4s (where it achieves 5ms/token ). For smaller\nmodels, the additional memory bandwidth and \ufb02ops are insu\ufb03cient to o\ufb00set the additional communication overhead between more devices \u2013 serving a 7B on 16 TPUs actually increases the latency. This\nmeans the 7B would provide only a modest speedup if used as a draft with its optimal topology, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1546-6db4-11ee-9ffe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-19", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "means the 7B would provide only a modest speedup if used as a draft with its optimal topology, and\nwe will not make full utilisation of the hardware during drafting.\nWe can sidestep this issue by training a wider model with a relatively few number of layers in\norder to minimise communication overhead. It has been observed that the performance of language\nmodels is relatively robust to changes in model aspect ratio (Levine et al., 2020), so this allows us\nto serve a powerful draft model which can be sampled rapidly on the same hardware as the target\nmodel.\nEvaluation on XSum and HumanEval\nWe evaluate speculative sampling with Chinchilla on two tasks and summarize the results in Table 1:\n\u2022The XSum (Narayan et al., 2018) benchmark. This is a natural language summarisation task\nusing a 1-shot prompt where we sample a total of 11,305 sequences with a maximum sequence\nlength 128.\n\u2022The 100-shot HumanEval task (Chen et al., 2021). This is a code generation task involves the\ngeneration of 16,400 samples with a maximum sequence length of 512.\nEven with greedy sampling, a single token deviating due to numerics could result in two sequences\ndiverging wildly. Since pseudo-random seeds are processed di\ufb00erently between ArS and SpS, and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1547-6db4-11ee-acb1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-20", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "diverging wildly. Since pseudo-random seeds are processed di\ufb00erently between ArS and SpS, and\nbecause the di\ufb00erent computation graphs lead to di\ufb00erent numerics, we cannot not expect identical\noutputs. However, we expect the samples to come from the same distribution within numerics and\nwe empirically verify this by evaluating these benchmarks.\nWe run the tasks at batch size 1 with SpS and ArS. The time taken per SpS/ArS loop has low\nvariance, and we can measure it directly from TPU pro\ufb01les. To obtain the average speedup, standard\ndeviations and other metrics, we log the amount of tokens generated for each speculative loop. In\nTable 1 we show the performance on the XSum and HumanEval benchmarks for naive and speculative\nsampling with Chinchilla.\n6\nAccelerating Large Language Model Decoding with Speculative Sampling\nWe obtain a substantial speedup in both tasks, with HumanEval reaching speedups of almost 2\u00935\u0002.\nYet, we have parity in the benchmark metrics \u2013 the underlying samples distribution is provably the\nsameuptonumerics, andthisveri\ufb01esthatthedraftmodelisnotbiasingtheresultsempirically. Inthe\ncase of HumanEval and greedy XSum, this speedup exceeded the theoretical memory bandwidth limit"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1548-6db4-11ee-8a25-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-21", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "case of HumanEval and greedy XSum, this speedup exceeded the theoretical memory bandwidth limit\nof the hardware for autoregressive sampling (model size divided by the total memory bandwidth).\nAcceptance rate changes per domain\nIt is apparent that the acceptance rate is dependent on the application and the decoding method.\nHumanEvalachievesasigni\ufb01cantlylargerspeedup\u2014Wehypothesizethatthisisduetoacombination\nof code containing a lot of common sub-sequences (e.g. for i in range(len(arr)): would be\nrelatively easy for a draft model to guess), is often decomposed into a smaller set of shorter tokens\nand the temperature value sharpening both the draft and target logits.\n01234567\nNumber of draft tokens (K)40060080010001200140016001800ms\nMean Sampling Time (128 tokens)\nHuman Eval\nXSum\n01234567\nNumber of draft tokens (K)0.50.60.70.80.91.0\nAcceptance rate\nHuman Eval\nXSum\n01234567\nNumber of draft tokens (K)1416182022242628ms\nTotal loop time\nFigure 1jLeft:The average time to generate 128 tokens, with standard deviation. Note that as \ud835\udc3e"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1549-6db4-11ee-b5aa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-22", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "Total loop time\nFigure 1jLeft:The average time to generate 128 tokens, with standard deviation. Note that as \ud835\udc3e\nincreases, the overall speedup plateaus or even regresses, with XSum being optimal at \ud835\udc3e=3. The\nvariance consistently increases with \ud835\udc3e.Middle:The average number of tokens accepted divided by\n\ud835\udc3e\u00b81\u2013 this serves as a measure of the overall e\ufb03ciency of the modi\ufb01ed rejection scheme, which\ndecreases with the lookahead. Right:Average time per loop increases approximately linearly with \ud835\udc3e\nduetotheincreasednumberofmodelcalls. Notethatthegradientisslightlyhigherthanthesampling\nspeed of the draft model, due to additional overheads in nucleus decoding.\nTrade o\ufb00 between longer drafts and more frequent scoring\nWevisualisethetrade-o\ufb00ofincreasing \ud835\udc3e,thenumberoftokenssampledbythedraftmodelinFigure1.\nAs\ud835\udc3eincreases, we need fewer scoring calls from the large models to generate the same sequence\nlength, potentially giving us a larger speedup. However, the total loop time increases approximately\nlinearly with the larger number of draft model calls and small increases in the scoring time. The"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d154a-6db4-11ee-ac4d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-23", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "length, potentially giving us a larger speedup. However, the total loop time increases approximately\nlinearly with the larger number of draft model calls and small increases in the scoring time. The\noverall e\ufb03ciency of the proportion of accepted tokens decreases as \ud835\udc3eincreases, since later tokens\ndepend on the acceptance of previous tokens. This results in the average speedup plateauing or\neven degrading with a larger \ud835\udc3e(for example, XSum with nucleus\u2019s latency is minimised at \ud835\udc3e=3),\ndepending on the domain.\n7\nAccelerating Large Language Model Decoding with Speculative Sampling\nFurther, even though larger values of \ud835\udc3emay yield marginally greater mean speedups in certain\ncircumstances, it also increases variance of the time to generate a full sequence. This could be\nproblematic for settings where the P90, P99 latencies of concern.\nConclusion\nInthiswork,wedemonstrateanewalgorithmandwork\ufb02owforacceleratingthedecodingoflanguage\nmodels. Speculative sampling does not require making any modi\ufb01cations to the target language\nmodel\u2019s parameters or architecture, is provably lossless within numerics, scales well with the appropriate draft model and complements many existing techniques for reducing latency in the small batch\nsize setting."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d154b-6db4-11ee-895c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-24", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "size setting.\nWe optimise and scale the technique to Chinchilla 70B using a draft model which was easy to\ntrainwithexistinginfrastructure, demonstratingthatityieldsalargespeedupacrossbenchmarktasks\nand common decoding methods in the process. We verify that it is indeed lossless empirically in its\ndownstream tasks.\nReferences\nA.Arnab,M.Dehghani,G.Heigold,C.Sun,M.Lucic,andC.Schmid. Vivit: Avideovisiontransformer.\nIn2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6816\u20136826. IEEE\nComputer Society, 2021.\nT. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,\nG. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information\nprocessing systems , 33:1877\u20131901, 2020.\nM.Chen,J.Tworek,H.Jun,Q.Yuan,H.P.deOliveiraPinto,J.Kaplan,H.Edwards,Y.Burda,N.Joseph,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d154c-6db4-11ee-878e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-25", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "G.Brockman,A.Ray,R.Puri,G.Krueger,M.Petrov,H.Khlaaf,G.Sastry,P.Mishkin,B.Chan,S.Gray,\nN. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings,\nM.Plappert,F.Chantzis,E.Barnes,A.Herbert-Voss,W.H.Guss,A.Nichol,A.Paino,N.Tezak,J.Tang,\nI. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra,\nE. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew,\nD. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d154d-6db4-11ee-9bc9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-26", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained\non code. CoRR, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374 .\nA. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung,\nC. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint\narXiv:2204.02311 , 2022.\nT. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. Llm. int8 (): 8-bit matrix multiplication for\ntransformers at scale. arXiv preprint arXiv:2208.07339 , 2022.\nA. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer,G.Heigold,S.Gelly,etal. Animageisworth16x16words: Transformersforimagerecognition"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d154e-6db4-11ee-87ae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-27", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "at scale.arXiv preprint arXiv:2010.11929 , 2020.\nT.Ge,H.Xia,X.Sun,S.Chen,andF.Wei. Losslessaccelerationforseq2seqgenerationwithaggressive\ndecoding. ArXiv, abs/2205.10350, 2022.\n8\nAccelerating Large Language Model Decoding with Speculative Sampling\nJ. Ho\ufb00mann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A.\nHendricks,J.Welbl,A.Clark,etal. Trainingcompute-optimallargelanguagemodels. arXivpreprint\narXiv:2203.15556 , 2022.\nX. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. TinyBERT: Distilling BERT"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d154f-6db4-11ee-9dcf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-28", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4163\u20134174, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.\ufb01ndings-emnlp.372. URL https://aclanthology.org/2020.\nfindings-emnlp.372 .\nY. Kim and A. M. Rush. Sequence-level knowledge distillation. CoRR, abs/1606.07947, 2016. URL\nhttp://arxiv.org/abs/1606.07947 .\nY. Leviathan, M. Kalman, and Y. Matias. Fast inference from transformers via speculative decoding.\nArXiv, abs/2211.17192, 2022.\nY. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua. The depth-to-width interplay in self-attention.\narXiv preprint arXiv:2006.12467 , 2020.\nS. Narayan, S. B. Cohen, and M. Lapata. Don\u2019t give me the details, just the summary! topicaware convolutional neural networks for extreme summarization. In Proceedings of the 2018"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1550-6db4-11ee-b538-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-29", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "Conference on Empirical Methods in Natural Language Processing , pages 1797\u20131807, Brussels,\nBelgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1206.\nURLhttps://aclanthology.org/D18-1206 .\nR. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, A. Levskaya, J. Heek, K. Xiao, S. Agrawal,\nand J. Dean. E\ufb03ciently scaling transformer inference. arXiv preprint arXiv:2211.05102 , 2022.\nJ. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Ho\ufb00mann, F. Song, J. Aslanides, S. Henderson, R. Ring,\nS. Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv\npreprint arXiv:2112.11446 , 2021.\nV. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1551-6db4-11ee-b3c7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-30", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster,\ncheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.\nN. Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019.\nURLhttp://arxiv.org/abs/1911.02150 .\nM. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training\nmulti-billionparameterlanguagemodelsusingmodelparallelism. arXivpreprintarXiv:1909.08053 ,\n2019.\nY.Song, C.Meng, R.Liao, andS.Ermon. Acceleratingfeedforwardcomputationviaparallelnonlinear\nequationsolving. InM.MeilaandT.Zhang,editors, Proceedingsofthe38thInternationalConference\non Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9791\u20139800."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1552-6db4-11ee-b10b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-31", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9791\u20139800.\nPMLR, 18\u201324 Jul 2021. URL https://proceedings.mlr.press/v139/song21a.html .\nM. Stern, N. Shazeer, and J. Uszkoreit. Blockwise parallel decoding for deep autoregressive models.\nCoRR, abs/1811.03115, 2018. URL http://arxiv.org/abs/1811.03115 .\nA. Wiggers and E. Hoogeboom. Predictive sampling with forecasting autoregressive models. In H. D.\nIII and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning ,\nvolume 119 of Proceedings of Machine Learning Research , pages 10260\u201310269. PMLR, 13\u201318 Jul\n2020. URL https://proceedings.mlr.press/v119/wiggers20a.html .\n9\nAccelerating Large Language Model Decoding with Speculative Sampling\nZ. Yao, R. Y. Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. Zeroquant: E\ufb03cient and a\ufb00ordable"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1553-6db4-11ee-833c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-32", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861 , 2022.\nSupplementary Materials\nAuthor Contributions\n\u2022Initial proposal: Charlie Chen, John Jumper and Geo\ufb00rey Irving\n\u2022Initial Implementation, Optimisation and Scaling: Charlie Chen\n\u2022Modi\ufb01ed Rejection Sampling Scheme: John Jumper\n\u2022Engineering Improvements: Jean-Baptiste Lespiau and Charlie Chen\n\u2022Experiments: Charlie Chen, Sebastian Borgeaud and Laurent Sifre\n\u2022Draft of Manuscript: Charlie Chen and Sebastian Borgeaud\n\u2022Manuscript Feedback: Laurent Sifre, Geo\ufb00rey Irving and John Jumper\nAcknowledgements\nWe\u2019d like to thank Oriol Vinyals and Koray Kavukcuoglu for your kind advice and leadership. We\u2019d\nalso like to thank Evan Senter for your additional feedback on the manuscript and Amelia Glaese for\nyour support in navigating the publishing process. Finally, we\u2019d like to thank Blake Hechtman, Berkin\nIlbeyi for your valuable advice on XLA and Nikolai Grigoriev for our discussions on the various tricks\nthat can be applied to the transformer architecture.\nHyperparams\nTable 2jHyperparameters for the draft model\nModel \ud835\udc51modelHeads Layers Params"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1554-6db4-11ee-9727-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-33", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "that can be applied to the transformer architecture.\nHyperparams\nTable 2jHyperparameters for the draft model\nModel \ud835\udc51modelHeads Layers Params\nTarget (Chinchilla) 8192 64 80 70B\nDraft 6144 48 8 4B\nProofs\nTheorem1 (Modi\ufb01edRejectionSamplingrecoversthetargetdistribution) .Givendiscretedistributions\n\ud835\udc5e,\ud835\udc5dand a single draft sample \u02dc\ud835\udc65\u0018\ud835\udc5d, let\ud835\udc4bbe the \ufb01nal resulting sample. For \ud835\udc4b=\ud835\udc65to be true, we must\neither sample \u02dc\ud835\udc65=\ud835\udc65and then accept it, or resample it after \u02dc\ud835\udc65(of any value) is rejected. Hence:\n\u2119\u00b9\ud835\udc4b=\ud835\udc65\u00ba\n=\u2119\u00b9\u02dc\ud835\udc65=\ud835\udc65\u00ba\u2119\u00b9\u02dc\ud835\udc65acceptedj\u02dc\ud835\udc65=\ud835\udc65\u00ba\u00b8\u2119\u00b9\u02dc\ud835\udc65rejected\u00ba\u2119\u00b9\ud835\udc4b=\ud835\udc65j\u02dc\ud835\udc65rejected\u00ba\nFor the \ufb01rst term, we apply the acceptance rule:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1555-6db4-11ee-9b15-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-34", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "For the \ufb01rst term, we apply the acceptance rule:\n\u2119\u00b9\u02dc\ud835\udc65=\ud835\udc65\u00ba\u2119\u00b9\u02dc\ud835\udc65acceptedj\u02dc\ud835\udc65=\ud835\udc65\u00ba\n=\ud835\udc5d\u00b9\ud835\udc65\u00bamin\u0012\n1\u0094\ud835\udc5e\u00b9\ud835\udc65\u00ba\n\ud835\udc5d\u00b9\ud835\udc65\u00ba\u0013\n10\nAccelerating Large Language Model Decoding with Speculative Sampling\n=min\u00b9\ud835\udc5d\u00b9\ud835\udc65\u00ba\u0094\ud835\udc5e\u00b9\ud835\udc65\u00ba\u00ba\nFor the second conditional term, we apply the resampling rule:\n\u2119\u00b9\ud835\udc4b=\ud835\udc65j\u02dc\ud835\udc65rejected\u00ba=\u00b9\ud835\udc5e\u00b9\ud835\udc65\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc65\u00ba\u00ba\u00b8\nWhere\u00b9\u0093\u00ba\u00b8denotes:\n\u00b9\ud835\udc53\u00b9\ud835\udc65\u00ba\u00ba\u00b8=max\u00b90\u0094 \ud835\udc53\u00b9\ud835\udc65\u00ba\u00ba\u00cd\n\ud835\udc65max\u00b90\u0094 \ud835\udc53\u00b9\ud835\udc65\u00ba\u00ba\nFinally, we calculate the probability of rejection:\n\u2119\u00b9\u02dc\ud835\udc65rejected\u00ba=1\u0000\u2119\u00b9\u02dc\ud835\udc65accepted\u00ba"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1556-6db4-11ee-8383-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-35", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "Finally, we calculate the probability of rejection:\n\u2119\u00b9\u02dc\ud835\udc65rejected\u00ba=1\u0000\u2119\u00b9\u02dc\ud835\udc65accepted\u00ba\n=1\u0000\u2211\ufe01\n\ud835\udc650\u2119\u00b9\ud835\udc4b=\ud835\udc650\u0094\u02dc\ud835\udc65accepted\u00ba\n=1\u0000\u2211\ufe01\n\ud835\udc650min\u00b9\ud835\udc5d\u00b9\ud835\udc650\u00ba\u0094\ud835\udc5e\u00b9\ud835\udc650\u00ba\u00ba\n=\u2211\ufe01\n\ud835\udc650max\u00b90\u0094\ud835\udc5e\u00b9\ud835\udc650\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc650\u00ba\u00ba\n=\u2211\ufe01\n\ud835\udc650\ud835\udc5e\u00b9\ud835\udc650\u00ba\u0000min\u00b9\ud835\udc5d\u00b9\ud835\udc650\u00ba\u0094\ud835\udc5e\u00b9\ud835\udc650\u00ba\u00ba\n=\u2211\ufe01\n\ud835\udc650max\u00b90\u0094\ud835\udc5e\u00b9\ud835\udc650\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc650\u00ba\u00ba\nThis is equal to the denominator of \u00b9\ud835\udc5e\u00b9\ud835\udc65\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc65\u00ba\u00ba\u00b8, so:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1557-6db4-11ee-a452-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2302.01318-36", "title": "Accelerating Large Language Model Decoding with Speculative Sampling", "source": "http://arxiv.org/pdf/2302.01318", "start_index": 0, "chroma:document": "This is equal to the denominator of \u00b9\ud835\udc5e\u00b9\ud835\udc65\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc65\u00ba\u00ba\u00b8, so:\n\u2119\u00b9\u02dc\ud835\udc65rejected\u00ba\u2119\u00b9\ud835\udc4b=\ud835\udc65j\u02dc\ud835\udc65rejected\u00ba=max\u00b90\u0094\ud835\udc5e\u00b9\ud835\udc65\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc65\u00ba\u00ba\nHence:\n\u2119\u00b9\ud835\udc4b=\ud835\udc65\u00ba\n=min\u00b9\ud835\udc5d\u00b9\ud835\udc65\u00ba\u0094\ud835\udc5e\u00b9\ud835\udc65\u00ba\u00ba\u00b8max\u00b90\u0094\ud835\udc5e\u00b9\ud835\udc65\u00ba\u0000\ud835\udc5d\u00b9\ud835\udc65\u00ba\u00ba\n=\ud835\udc5e\u00b9\ud835\udc65\u00ba\nand we have recovered the desired target.\n11"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1558-6db4-11ee-afe8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-0", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "GQA: Training Generalized Multi-Query Transformer Models from\nMulti-Head Checkpoints\nJoshua Ainslie\u0003, James Lee-Thorp\u0003, Michiel de Jong\u0003yy\nYury Zemlyanskiy ,Federico Lebr\u00f3n ,Sumit Sanghai\nGoogle Research\nAbstract\nMulti-query attention (MQA), which only\nuses a single key-value head, drastically\nspeeds up decoder inference. However, MQA\ncan lead to quality degradation, and moreover\nit may not be desirable to train a separate\nmodel just for faster inference. We (1) propose\na recipe for uptraining existing multi-head language model checkpoints into models with\nMQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than\none, less than number of query heads) number\nof key-value heads. We show that uptrained\nGQA achieves quality close to multi-head attention with comparable speed to MQA.\n1 Introduction\nAutoregressive decoder inference is a severe bottleneck for Transformer models due to the memory\nbandwidth overhead from loading decoder weights\nand all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong\net al., 2022). The memory bandwidth from loading"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1559-6db4-11ee-91a2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-1", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "and all attention keys and values at every decoding step (Shazeer, 2019; Pope et al., 2022; de Jong\net al., 2022). The memory bandwidth from loading\nkeys and values can be sharply reduced through\nmulti-query attention (Shazeer, 2019), which uses\nmultiple query heads but single key and value\nheads.\nHowever, multi-query attention ( MQA ) can lead\nto quality degradation and training instability, and\nit may not be feasible to train separate models\noptimized for quality and inference. Moreover,\nwhile some language models already use multiquery attention, such as PaLM (Chowdhery et al.,\n2022), many do not, including publicly available\nlanguage models such as T5 (Raffel et al., 2020)\nand LLaMA (Touvron et al., 2023).\nThis work contains two contributions for faster\ninference with large language models. First, we\n\u0003Equal contribution.\nyUniversity of Southern California. Work done at Google\nResearch.show that language model checkpoints with multihead attention ( MHA ) can be uptrained (Komatsuzaki et al., 2022) to use MQA with a small fraction of original training compute. This presents a\ncost-effective method to obtain fast multi-query as\nwell as high-quality MHA checkpoints.\nSecond, we propose grouped-query attention"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d155a-6db4-11ee-9d3c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-2", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "cost-effective method to obtain fast multi-query as\nwell as high-quality MHA checkpoints.\nSecond, we propose grouped-query attention\n(GQA ), an interpolation between multi-head and\nmulti-query attention with single key and value\nheads per subgroup of query heads . We show that\nuptrained GQA achieves quality close to multihead attention while being almost as fast as multiquery attention.\n2 Method\n2.1 Uptraining\nGenerating a multi-query model from a multi-head\nmodel takes place in two steps: \ufb01rst, converting the\ncheckpoint, and second, additional pre-training to\nallow the model to adapt to its new structure. Figure 1 shows the process for converting a multi-head\ncheckpoint into a multi-query checkpoint. The projection matrices for key and value heads are mean\npooled into single projection matrices, which we\n\ufb01nd works better than selecting a single key and\nvalue head or randomly initializing new key and\nvalue heads from scratch.\nFigure 1: Overview of conversion from multi-head to\nmulti-query attention. Key and value projection matrices from all heads are mean pooled into a single head.\nThe converted checkpoint is then pre-trained forarXiv:2305.13245v1  [cs.CL]  22 May 2023"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d155b-6db4-11ee-901c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-3", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "The converted checkpoint is then pre-trained forarXiv:2305.13245v1  [cs.CL]  22 May 2023\nFigure 2: Overview of grouped-query method. Multi-head attention has H query, key, and value heads. Multi-query\nattention shares single key and value heads across all query heads. Grouped-query attention instead shares single\nkey and value heads for each group of query heads, interpolating between multi-head and multi-query attention.\na small proportion \u000bof its original training steps\non the same pre-training recipe.\n2.2 Grouped-query attention\nGrouped-query attention divides query heads into\nGgroups , each of which shares a single key head\nand value head. GQA -Grefers to grouped-query\nwithGgroups. GQA -1, with a single group and\ntherefore single key and value head, is equivalent to\nMQA , while GQA -H, with groups equal to number\nof heads, is equivalent to MHA . Figure 2 shows a\ncomparison of grouped-query attention and multihead/multi-query attention. When converting a\nmulti-head checkpoint to a GQA checkpoint, we\nconstruct each group key and value head by meanpooling all the original heads within that group.\nAn intermediate number of groups leads to an"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d155c-6db4-11ee-9193-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-4", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "construct each group key and value head by meanpooling all the original heads within that group.\nAn intermediate number of groups leads to an\ninterpolated model that is higher quality than MQA\nbut faster than MHA , and, as we will show, represents a favorable trade-off. Going from MHA\ntoMQA reducesHkey and value heads to a single key and value head, reducing the size of the\nkey-value cache and therefore amount of data that\nneeds to be loaded by a factor of H. However,\nlarger models generally scale the number of heads,\nsuch that multi-query attention represents a more\naggressive cut in both memory bandwidth and capacity. GQA lets us keep the same proportional\ndecrease in bandwidth and capacity as model size\nincreases.\nMoreover, larger models suffer relatively less\nfrom memory bandwidth overhead from attention,\nas the KV-cache scales with model dimension\nwhile model FLOPs and parameters scale with the\nsquare of model dimension. Finally, standard sharding for large models replicates the single key and\nvalue head by the number of model partitions (Popeet al., 2022); GQA removes the waste from such\npartitioning. Therefore, we expect GQA to present\na particularly good trade-off for larger models.\n3 Experiments\n3.1 Experimental setup\nCon\ufb01gurations All models are based on the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d155d-6db4-11ee-905e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-5", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "a particularly good trade-off for larger models.\n3 Experiments\n3.1 Experimental setup\nCon\ufb01gurations All models are based on the\nT5.1.1 architecture (Raffel et al., 2020), implemented with JAX (Bradbury et al., 2018),\nFlax (Heek et al., 2020), and Flaxformer1. For\nour main experiments we consider T5 Large and\nXXL with multi-head attention, as well as uptrained\nversions of T5 XXL with multi-query and groupedquery attention. We apply MQA andGQA to\ndecoder self-attention and cross-attention, but not\nencoder self-attention.\nUptraining Uptrained models are initialized\nfrom public T5.1.1 checkpoints. The key and value\nheads are mean-pooled to the appropriate MQA or\nGQA structure, and then pre-trained for a further\n\u000bproportion of original pre-training steps with the\noriginal pre-training setup (Raffel et al., 2020).\nData We evaluate on summarization datasets\nCNN/Daily Mail (Nallapati et al., 2016), arXiv\nand PubMed (Cohan et al., 2018), MediaSum (Zhu\net al., 2021), and Multi-News (Fabbri et al., 2019);"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d155e-6db4-11ee-bfa4-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-6", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "and PubMed (Cohan et al., 2018), MediaSum (Zhu\net al., 2021), and Multi-News (Fabbri et al., 2019);\ntranslation dataset WMT 2014 English-to-German;\nand question answering dataset TriviaQA (Joshi\net al., 2017). We do not evaluate on popular classi\ufb01cation benchmarks such as GLUE (Wang et al.,\n2019) as autoregressive inference is less applicable\nfor those tasks.\n1https://github.com/google/flaxformer\nModel Tinfer Average CNN arXiv PubMed MediaSum MultiNews WMT TriviaQA\ns R1 R1 R1 R1 R1 BLEU F1\nMHA-Large 0.37 46.0 42.9 44.6 46.2 35.5 46.6 27.7 78.2\nMHA-XXL 1.51 47.2 43.8 45.6 47.5 36.4 46.9 28.4 81.9\nMQA-XXL 0.24 46.5 43.0 45.0 47.2 36.1 46.5 28.5 81.3\nGQA-8-XXL 0.28 47.1 43.5 45.4 47.7 36.3 47.2 28.4 81.6"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d155f-6db4-11ee-a54c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-7", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "GQA-8-XXL 0.28 47.1 43.5 45.4 47.7 36.3 47.2 28.4 81.6\nTable 1: Inference time and average dev set performance comparison of T5 Large and XXL models with multihead attention, and 5% uptrained T5-XXL models with multi-query and grouped-query attention on summarization\ndatasets CNN/Daily Mail, arXiv, PubMed, MediaSum, and MultiNews, translation dataset WMT, and questionanswering dataset TriviaQA.\nFine-tuning For \ufb01ne-tuning, we use a constant\nlearning rate of 0.001, batch size 128, and dropout\nrate 0.1 for all tasks. CNN/Daily Mail and WMT\nuse input length of 512 and output length 256.\nOther summarization datasets use input length\n2048 and output length 512. Finally, TriviaQA\nuses input length 2048 and output length 32. We\ntrain until convergence and select the checkpoint\nwith the highest dev performance. We use greedy\ndecoding for inference.\nTiming We report time per sample per TPUv4\nchip, as measured by xprof (Google, 2020). For\ntiming experiments we use 8 TPUs with the largest"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1560-6db4-11ee-9927-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-8", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "Timing We report time per sample per TPUv4\nchip, as measured by xprof (Google, 2020). For\ntiming experiments we use 8 TPUs with the largest\nbatch size that \ufb01ts up to 32 per TPU, and parallelization optimized separately for each model.\n0 0:5 1 1:54646:547\nMHA-LargeMHA-XXL\nMQA-XXLGQA-XXL\nTime per sample (ms)Performance\nFigure 3: Uptrained MQA yields a favorable tradeoff compared to MHA with higher quality and\nfaster speed than MHA-Large, and GQA achieves\neven better performance with similar speed gains\nand comparable quality to MHA-XXL. Average performance on all tasks as a function of average inference\ntime per sample for T5-Large and T5-XXL with multihead attention, and 5% uptrained T5-XXL with MQA\nand GQA-8 attention.3.2 Main results\nFigure 3 shows average performance over all\ndatasets as a function of average inference time\nforMHA T5-Large and T5-XXL, and uptrained\nMQA andGQA -8XXL models with uptraining"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1561-6db4-11ee-b415-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-9", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "forMHA T5-Large and T5-XXL, and uptrained\nMQA andGQA -8XXL models with uptraining\nproportion\u000b= 0:05. We see that a larger uptrained MQA model provides a favorable tradeoff relative to MHA models, with higher quality\nand faster inference than MHA -Large. Moreover,\nGQA achieves signi\ufb01cant additional quality gains,\nachieving performance close to MHA -XXL with\nspeed close to MQA . Table 1 contains full results\nfor all datasets.\n3.3 Ablations\nThis section presents experiments to investigate\nthe effect of different modeling choices. We evaluate performance on a representive subsample of\ntasks: CNN/Daily Mail, (short-form summarization), MultiNews (long-form summarization), and\nTriviaQA (question-answering).\n54:454:654:855 55:255:455:6Mean\nFirst\nRandom\nFigure 4: Performance comparison of different checkpoint conversion methods for T5-Large uptrained to\nMQA with proportion \u000b= 0:05. \u2018Mean\u2019 mean-pools\nkey and value heads, \u2018First\u2019 selects the \ufb01rst head and\n\u2018Random\u2019 initializes heads from scratch.\nCheckpoint conversion Figure 4 compares the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1562-6db4-11ee-814f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-10", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "key and value heads, \u2018First\u2019 selects the \ufb01rst head and\n\u2018Random\u2019 initializes heads from scratch.\nCheckpoint conversion Figure 4 compares the\nperformance of different methods for checkpoint\nconversion. Mean pooling appears to work best,\nfollowed by selecting a single head and then random initialization. Intuitively, results are ordered\nby the degree to which information is preserved\nfrom the pre-trained model.\nUptraining steps Figure 5 shows how performance varies with uptraining proportion for T5\nXXL with MQA andGQA . First, we note that\nGQA already achieves reasonable performance after conversion while MQA requires uptraining to\nbe useful. Both MQA andGQA gain from 5%\nuptraining with diminishing returns from 10%.\n0 0:02 0:04 0:06 0:08 0:154555657\nUptraining proportion \u000bPerformanceMHA\nGQA\nMQA\nFigure 5: Performance as a function of uptraining proportion for T5 XXL models with MQA and GQA-8.\nNumber of groups Figure 6 demonstrates the\neffect of the number of GQA groups on inference speed. For larger models the memory bandwidth overhead from the KV cache is less constraining (Shazeer, 2019), while the reduction in"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d1563-6db4-11ee-bc19-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-11", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "effect of the number of GQA groups on inference speed. For larger models the memory bandwidth overhead from the KV cache is less constraining (Shazeer, 2019), while the reduction in\nkey-value size is sharper due to the increased number of heads. As a result, increasing the number\nof groups from MQA only results in modest slowdowns initially, with increasing cost as we move\ncloser to MHA . We selected 8 groups as a favorable middle ground.\n4 Related Work\nThis work is focused on achieving a better tradeoff between decoder quality and inference time\nthrough reducing the memory bandwidth overhead (Williams et al., 2009) from loading keys and\nvalues. Shazeer (2019) \ufb01rst proposed reducing this\noverhead through multi-query attention. Follow-up\nwork showed that multi-query attention is especially helpful for long inputs (Pope et al., 2022;\nde Jong et al., 2022).\nA number of other methods have been proposed\nto reduce memory bandwidth overhead from keys\nand values, as well as parameters. Flash atten-1 4 8 16 32 6412\nGQA groupsTime per sample (s)MHA\nGQA\nMQA\nFigure 6: Time per sample for GQA-XXL as a function of the number of GQA groups with input length\n2048 and output length 512. Going from 1 (MQA) to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3ba7-6db4-11ee-89c5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-12", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "2048 and output length 512. Going from 1 (MQA) to\n8 groups adds modest inference overhead, with increasing cost to adding more groups.\ntion (Dao et al., 2022) structures the attention computation to avoid materializing the quadratic attention scores, reducing memory and speeding up\ntraining. Quantization (Dettmers et al., 2022; Zeng\net al., 2022; Frantar et al., 2022) reduces the size of\nweights and activations, including keys and values,\nby lowering precision. Model distillation (Hinton et al., 2015; Gou et al., 2021) instead reduces\nmodel size at a given precision, using data generated from the larger model to \ufb01netune the smaller\nmodel. Layer-sparse cross-attention (de Jong et al.,\n2022) eliminates most cross-attention layers which\nmake up the primary expense for longer inputs.\nSpeculative sampling (Chen et al., 2023; Leviathan\net al., 2022; Kim et al., 2023) ameliorates the memory bandwidth bottleneck by proposing multiple\ntokens with a smaller model which are then scored\nin parallel by a larger model.\nFinally, the uptraining procedure we propose\nis inspired by Komatsuzaki et al. (2022), which\nuptrains standard T5 checkpoints into sparsely activated Mixture-of-Experts models.\n5 Conclusion"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3ba8-6db4-11ee-b73f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-13", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "is inspired by Komatsuzaki et al. (2022), which\nuptrains standard T5 checkpoints into sparsely activated Mixture-of-Experts models.\n5 Conclusion\nLanguage models are expensive for inference primarily due to the memory bandwidth overhead\nfrom loading keys and values. Multi-query attention reduces this overhead at the cost of decreased\nmodel capacity and quality. We propose to convert\nmulti-head attention models to multi-query models\nwith a small fraction of original pre-training compute. Moreover, we introduce grouped-query attention, an interpolation of multi-query and multi-head\nattention that achieves quality close to multi-head\nat comparable speed to multi-query attention.\nAcknowlegements\nWe thank Santiago Onta\u00f1\u00f3n, Afroz Mohiuddin,\nWilliam Cohen and others at Google Research for\ninsightful advice and discussion.\nReferences\nJames Bradbury, Roy Frostig, Peter Hawkins,\nMatthew James Johnson, Chris Leary, Dougal\nMaclaurin, George Necula, Adam Paszke, Jake\nVanderPlas, Skye Wanderman-Milne, and Qiao\nZhang. 2018. JAX: composable transformations of\nPython+NumPy programs.\nCharlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and\nJohn Jumper. 2023. Accelerating large language"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3ba9-6db4-11ee-9455-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-14", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and\nJohn Jumper. 2023. Accelerating large language\nmodel decoding with speculative sampling. CoRR ,\nabs/2302.01318.\nAakanksha Chowdhery, Sharan Narang, Jacob Devlin,\nMaarten Bosma, Gaurav Mishra, Adam Roberts,\nPaul Barham, Hyung Won Chung, Charles Sutton,\nSebastian Gehrmann, Parker Schuh, Kensen Shi,\nSasha Tsvyashchenko, Joshua Maynez, Abhishek\nRao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben\nHutchinson, Reiner Pope, James Bradbury, Jacob\nAustin, Michael Isard, Guy Gur-Ari, Pengcheng\nYin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier\nGarcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan,\nHyeontaek Lim, Barret Zoph, Alexander Spiridonov,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3baa-6db4-11ee-a35d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-15", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "Hyeontaek Lim, Barret Zoph, Alexander Spiridonov,\nRyan Sepassi, David Dohan, Shivani Agrawal, Mark\nOmernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz,\nErica Moreira, Rewon Child, Oleksandr Polozov,\nKatherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta,\nJason Wei, Kathy Meier-Hellstern, Douglas Eck,\nJeff Dean, Slav Petrov, and Noah Fiedel. 2022.\nPalm: Scaling language modeling with pathways.\nArman Cohan, Franck Dernoncourt, Doo Soon Kim,\nTrung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention\nmodel for abstractive summarization of long documents. In Proceedings of the 2018 Conference of\nthe North American Chapter of the Association for\nComputational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 615\u2013621,\nNew Orleans, Louisiana. Association for Computational Linguistics.\nTri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bab-6db4-11ee-87fe-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-16", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "New Orleans, Louisiana. Association for Computational Linguistics.\nTri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra,\nand Christopher R\u00e9. 2022. Flashattention: Fast and\nmemory-ef\ufb01cient exact attention with io-awareness.\nCoRR , abs/2205.14135.\nMichiel de Jong, Yury Zemlyanskiy, Joshua Ainslie,\nNicholas FitzGerald, Sumit Sanghai, Fei Sha, andWilliam Cohen. 2022. FiDO: Fusion-in-decoder optimized for stronger performance and faster inference. arXiv preprint arXiv:2212.08153 .\nTim Dettmers, Mike Lewis, Younes Belkada, and\nLuke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. CoRR ,\nabs/2208.07339.\nAlexander R. Fabbri, Irene Li, Tianwei She, Suyi Li,\nand Dragomir R. Radev. 2019. Multi-news: A largescale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the\n57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July\n28- August 2, 2019, Volume 1: Long Papers , pages"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bac-6db4-11ee-9b14-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-17", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July\n28- August 2, 2019, Volume 1: Long Papers , pages\n1074\u20131084. Association for Computational Linguistics.\nElias Frantar, Saleh Ashkboos, Torsten Hoe\ufb02er, and\nDan Alistarh. 2022. GPTQ: accurate post-training\nquantization for generative pre-trained transformers.\nCoRR , abs/2210.17323.\nGoogle. 2020. Pro\ufb01le your model with cloud tpu\ntools. https://cloud.google.com/tpu/docs/\ncloud-tpu-tools . Accessed: 2022-11-11.\nJianping Gou, Baosheng Yu, Stephen J. Maybank, and\nDacheng Tao. 2021. Knowledge distillation: A survey. Int. J. Comput. Vis. , 129(6):1789\u20131819.\nJonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner,\nand Marc van Zee. 2020. Flax: A neural network\nlibrary and ecosystem for JAX.\nGeoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.\n2015. Distilling the knowledge in a neural network."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bad-6db4-11ee-abc1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-18", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "library and ecosystem for JAX.\nGeoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.\n2015. Distilling the knowledge in a neural network.\nCoRR , abs/1503.02531.\nMandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke\nZettlemoyer. 2017. Triviaqa: A large scale distantly\nsupervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of\nthe Association for Computational Linguistics , Vancouver, Canada. Association for Computational Linguistics.\nSehoon Kim, Karttikeya Mangalam, Jitendra Malik,\nMichael W. Mahoney, Amir Gholami, and Kurt\nKeutzer. 2023. Big little transformer decoder.\nCoRR , abs/2302.07863.\nAran Komatsuzaki, Joan Puigcerver, James Lee-Thorp,\nCarlos Riquelme Ruiz, Basil Mustafa, Joshua\nAinslie, Yi Tay, Mostafa Dehghani, and Neil\nHoulsby. 2022. Sparse upcycling: Training mixtureof-experts from dense checkpoints.\nYaniv Leviathan, Matan Kalman, and Yossi Matias.\n2022. Fast inference from transformers via speculative decoding. CoRR , abs/2211.17192."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bae-6db4-11ee-9cf1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-19", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "Yaniv Leviathan, Matan Kalman, and Yossi Matias.\n2022. Fast inference from transformers via speculative decoding. CoRR , abs/2211.17192.\nRamesh Nallapati, Bowen Zhou, C\u00edcero Nogueira dos\nSantos, \u00c7aglar G\u00fcl\u00e7ehre, and Bing Xiang. 2016.\nAbstractive text summarization using sequence-tosequence rnns and beyond. In Proceedings of the\n20th SIGNLL Conference on Computational Natural\nLanguage Learning, CoNLL 2016, Berlin, Germany,\nAugust 11-12, 2016 , pages 280\u2013290. ACL.\nReiner Pope, Sholto Douglas, Aakanksha Chowdhery,\nJacob Devlin, James Bradbury, Anselm Levskaya,\nJonathan Heek, Kefan Xiao, Shivani Agrawal, and\nJeff Dean. 2022. Ef\ufb01ciently scaling transformer inference. arXiv preprint arXiv:2211.05102 .\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine\nLee, Sharan Narang, Michael Matena, Yanqi Zhou,\nWei Li, and Peter J. Liu. 2020. Exploring the limits"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3baf-6db4-11ee-b86d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-20", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "Lee, Sharan Narang, Michael Matena, Yanqi Zhou,\nWei Li, and Peter J. Liu. 2020. Exploring the limits\nof transfer learning with a uni\ufb01ed text-to-text transformer. J. Mach. Learn. Res. , 21:140:1\u2013140:67.\nNoam Shazeer. 2019. Fast transformer decoding:\nOne write-head is all you need. arXiv preprint\narXiv:1911.02150 .\nHugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier\nMartinet, Marie-Anne Lachaux, Timoth\u00e9e Lacroix,\nBaptiste Rozi\u00e8re, Naman Goyal, Eric Hambro,\nFaisal Azhar, Aurelien Rodriguez, Armand Joulin,\nEdouard Grave, and Guillaume Lample. 2023.\nLlama: Open and ef\ufb01cient foundation language models.\nAlex Wang, Amanpreet Singh, Julian Michael, Felix\nHill, Omer Levy, and Samuel R. Bowman. 2019.\nGLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th\nInternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9,\n2019 . OpenReview.net."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb0-6db4-11ee-b6e5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-21", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9,\n2019 . OpenReview.net.\nSamuel Williams, Andrew Waterman, and David A.\nPatterson. 2009. Roo\ufb02ine: an insightful visual performance model for multicore architectures. Commun. ACM , 52(4):65\u201376.\nAohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang,\nHanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu,\nWendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan\nMa, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng\nZhang, Yuxiao Dong, and Jie Tang. 2022. GLM130B: an open bilingual pre-trained model. CoRR ,\nabs/2210.02414.\nChenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng.\n2021. Mediasum: A large-scale media interview\ndataset for dialogue summarization. In Proceedings\nof the 2021 Conference of the North American Chapter of the Association for Computational Linguistics:\nHuman Language Technologies, NAACL-HLT 2021,\nOnline, June 6-11, 2021 , pages 5927\u20135934. Association for Computational Linguistics."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb1-6db4-11ee-9f90-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2305.13245-22", "title": "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", "source": "http://arxiv.org/pdf/2305.13245", "start_index": 0, "chroma:document": "Human Language Technologies, NAACL-HLT 2021,\nOnline, June 6-11, 2021 , pages 5927\u20135934. Association for Computational Linguistics.\nA Training Stability\nWe \ufb01nd that multi-query attention can lead to training instability, in particular combined with long\ninput tasks. We trained multiple T5-Large models with multi-query attention from scratch. In\neach case, pre-training suffered from frequent loss\nspikes and the \ufb01nal models diverged immediately\non long-input tasks. Uptrained multi-query attention models are more stable but still display high\nvariance, so for multi-query models on unstable\ntasks we report average performance over multiple\nruns. Uptrained grouped-query attention models,\nhowever, appear to be stable."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb2-6db4-11ee-bbae-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-0", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle : Open Foundation and Fine-Tuned Chat Models\nHugo Touvron\u0003Louis MartinyKevin Stoney\nPeter Albert Amjad Almahairi Yasmine Babaei Nikolay Bashlykov Soumya Batra\nPrajjwal Bhargava Shruti Bhosale Dan Bikel Lukas Blecher Cristian Canton Ferrer Moya Chen\nGuillem Cucurull David Esiobu Jude Fernandes Jeremy Fu Wenyin Fu Brian Fuller\nCynthia Gao Vedanuj Goswami Naman Goyal Anthony Hartshorn Saghar Hosseini Rui Hou\nHakan Inan Marcin Kardas Viktor Kerkez Madian Khabsa Isabel Kloumann Artem Korenev\nPunit Singh Koura Marie-Anne Lachaux Thibaut Lavril Jenya Lee Diana Liskovich\nYinghai Lu Yuning Mao Xavier Martinet Todor Mihaylov Pushkar Mishra\nIgor Molybog Yixin Nie Andrew Poulton Jeremy Reizenstein Rashi Rungta Kalyan Saladi\nAlan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb3-6db4-11ee-9599-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-1", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "Alan Schelten Ruan Silva Eric Michael Smith Ranjan Subramanian Xiaoqing Ellen Tan Binh Tang\nRoss Taylor Adina Williams Jian Xiang Kuan Puxin Xu Zheng Yan Iliyan Zarov Yuchen Zhang\nAngela Fan Melanie Kambadur Sharan Narang Aurelien Rodriguez Robert Stojnic\nSergey Edunov Thomas Scialom\u0003\nGenAI, Meta\nAbstract\nIn this work, we develop and release Llama 2, a collection of pretrained and \ufb01ne-tuned\nlarge language models (LLMs) ranging in scale from 7 billion to 70 billion parameters.\nOur \ufb01ne-tuned LLMs, called L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , are optimized for dialogue use cases. Our\nmodels outperform open-source chat models on most benchmarks we tested, and based on\nourhumanevaluationsforhelpfulnessandsafety,maybeasuitablesubstituteforclosedsource models. We provide a detailed description of our approach to \ufb01ne-tuning and safety"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb4-6db4-11ee-a938-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-2", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "improvements of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc in order to enable the community to build on our work and\ncontribute to the responsible development of LLMs.\n\u0003Equal contribution, corresponding authors: {tscialom, htouvron}@meta.com\nySecond author\nContributions for all the authors can be found in Section A.1.arXiv:2307.09288v2  [cs.CL]  19 Jul 2023\nContents\n1 Introduction 3\n2 Pretraining 5\n2.1 Pretraining Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5\n2.2 Training Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb5-6db4-11ee-8a1b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-3", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "2.3 L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle Pretrained Model Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7\n3 Fine-tuning 8\n3.1 Supervised Fine-Tuning (SFT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9\n3.2 Reinforcement Learning with Human Feedback (RLHF) . . . . . . . . . . . . . . . . . . . . . 9\n3.3 System Message for Multi-Turn Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16\n3.4 RLHF Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17\n4 Safety 20"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb6-6db4-11ee-9f58-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-4", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "4 Safety 20\n4.1 Safety in Pretraining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20\n4.2 Safety Fine-Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23\n4.3 Red Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28\n4.4 Safety Evaluation of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29\n5 Discussion 32\n5.1 Learnings and Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb7-6db4-11ee-9581-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-5", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "5.2 Limitations and Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34\n5.3 Responsible Release Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35\n6 Related Work 35\n7 Conclusion 36\nA Appendix 46\nA.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46\nA.2 Additional Details for Pretraining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47\nA.3 Additional Details for Fine-tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51\nA.4 Additional Details for Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb8-6db4-11ee-81a8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-6", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "A.5 Data Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72\nA.6 Dataset Contamination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75\nA.7 Model Card . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77\n2\nFigure1: Helpfulnesshumanevaluation resultsfor L/l.sc/a.sc/m.sc/a.sc\n/two.taboldstyle-C/h.sc/a.sc/t.sccomparedtootheropen-sourceandclosed-source\nmodels. Human raters compared model generations on ~4k\npromptsconsistingofbothsingleandmulti-turnprompts.\nThe95%con\ufb01denceintervalsforthisevaluationarebetween\n1%and2%. MoredetailsinSection3.4.2. Whilereviewing"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bb9-6db4-11ee-9760-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-7", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "The95%con\ufb01denceintervalsforthisevaluationarebetween\n1%and2%. MoredetailsinSection3.4.2. Whilereviewing\nthese results, it is important to note that human evaluations\ncanbenoisyduetolimitationsofthepromptset,subjectivity\nof the review guidelines, subjectivity of individual raters,\nand the inherent di\ufb03culty of comparing generations.\nFigure 2: Win-rate % for helpfulness and\nsafety between commercial-licensed baselines and L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , according to GPT4. Tocomplementthehumanevaluation,we\nused a more capable model, not subject to\nourownguidance. Greenareaindicatesour\nmodelisbetteraccordingtoGPT-4. Toremove\nties, we used win= (win+loss). The orders in\nwhichthemodelresponsesarepresentedto\nGPT-4arerandomlyswappedtoalleviatebias.\n1 Introduction\nLarge Language Models (LLMs) have shown great promise as highly capable AI assistants that excel in\ncomplex reasoning tasks requiring expert knowledge across a wide range of \ufb01elds, including in specialized"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bba-6db4-11ee-ab8e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-8", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "complex reasoning tasks requiring expert knowledge across a wide range of \ufb01elds, including in specialized\ndomains such as programming and creative writing. They enable interaction with humans through intuitive\nchat interfaces, which has led to rapid and widespread adoption among the general public.\nThecapabilitiesofLLMsareremarkableconsideringtheseeminglystraightforwardnatureofthetraining\nmethodology. Auto-regressivetransformersarepretrainedonanextensivecorpusofself-superviseddata,\nfollowed by alignment with human preferences via techniques such as Reinforcement Learning with Human\nFeedback(RLHF).Althoughthetrainingmethodologyissimple,highcomputationalrequirementshave\nlimited the development of LLMs to a few players. There have been public releases of pretrained LLMs\n(such as BLOOM (Scao et al., 2022), LLaMa-1 (Touvron et al., 2023), and Falcon (Penedo et al., 2023)) that\nmatch the performance of closed pretrained competitors like GPT-3 (Brown et al., 2020) and Chinchilla\n(Ho\ufb00mann et al., 2022), but none of these models are suitable substitutes for closed \u201cproduct\u201d LLMs, such\nasChatGPT,BARD,andClaude. TheseclosedproductLLMsareheavily\ufb01ne-tunedtoalignwithhuman"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bbb-6db4-11ee-8be3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-9", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "asChatGPT,BARD,andClaude. TheseclosedproductLLMsareheavily\ufb01ne-tunedtoalignwithhuman\npreferences, which greatly enhances their usability and safety. This step can require signi\ufb01cant costs in\ncomputeandhumanannotation,andisoftennottransparentoreasilyreproducible,limitingprogresswithin\nthe community to advance AI alignment research.\nIn this work, we develop and release Llama 2, a family of pretrained and \ufb01ne-tuned LLMs, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle and\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , at scales up to 70B parameters. On the series of helpfulness and safety benchmarks we tested,\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc models generally perform better than existing open-source models. They also appear to\nbe on par with some of the closed-source models, at least on the human evaluations we performed (see"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bbc-6db4-11ee-88d8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-10", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "be on par with some of the closed-source models, at least on the human evaluations we performed (see\nFigures1and3). Wehavetakenmeasurestoincreasethesafetyofthesemodels,usingsafety-speci\ufb01cdata\nannotation and tuning, as well as conducting red-teaming and employing iterative evaluations. Additionally,\nthispapercontributesathoroughdescriptionofour\ufb01ne-tuningmethodologyandapproachtoimproving\nLLM safety. We hope that this openness will enable the community to reproduce \ufb01ne-tuned LLMs and\ncontinue to improve the safety of those models, paving the way for more responsible development of LLMs.\nWealsosharenovelobservationswemadeduringthedevelopmentof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle andL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc ,suchas\nthe emergence of tool usage and temporal organization of knowledge.\n3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bbd-6db4-11ee-84aa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-11", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "the emergence of tool usage and temporal organization of knowledge.\n3\nFigure 3: Safety human evaluation results for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc compared to other open-source and closedsource models. Human raters judged model generations for safety violations across ~2,000 adversarial\nprompts consisting of both single and multi-turn prompts. More details can be found in Section 4.4. It is\nimportanttocaveatthesesafetyresultswiththeinherentbiasofLLMevaluationsduetolimitationsofthe\npromptset,subjectivityofthereviewguidelines,andsubjectivityofindividualraters. Additionally,these\nsafety evaluations are performed using content standards that are likely to be biased towards the L/l.sc/a.sc/m.sc/a.sc\n/two.taboldstyle-C/h.sc/a.sc/t.scmodels.\nWe are releasing the following models to the general public for research and commercial use\u2021:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bbe-6db4-11ee-ac21-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-12", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "We are releasing the following models to the general public for research and commercial use\u2021:\n1.L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle,anupdatedversionof L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle,trainedonanewmixofpubliclyavailabledata. Wealso\nincreasedthesizeofthepretrainingcorpusby40%,doubledthecontextlengthofthemodel,and\nadoptedgrouped-queryattention(Ainslieetal.,2023). Wearereleasingvariantsof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle with\n7B,13B,and70Bparameters. Wehavealsotrained34Bvariants,whichwereportoninthispaper\nbut are not releasing.\u00a7\n2.L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , a \ufb01ne-tuned version of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle that is optimized for dialogue use cases. We release"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bbf-6db4-11ee-8b5c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-13", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "variants of this model with 7B, 13B, and 70B parameters as well.\nWebelievethattheopenreleaseofLLMs,whendonesafely,willbeanetbene\ufb01ttosociety. LikeallLLMs,\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle is a new technology that carries potential risks with use (Bender et al., 2021b; Weidinger et al., 2021;\nSolaimanet al.,2023). Testingconductedtodate hasbeeninEnglish andhasnot\u2014 andcouldnot\u2014 cover\nall scenarios. Therefore, before deploying any applications of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , developers should perform\nsafetytestingand tuningtailoredtotheirspeci\ufb01capplicationsofthemodel. Weprovidearesponsibleuse\nguide\u00b6and code examples\u2016to facilitate the safe deployment of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle andL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . More details of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc0-6db4-11ee-b48c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-14", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "our responsible release strategy can be found in Section 5.3.\nTheremainderofthispaperdescribesourpretrainingmethodology(Section2),\ufb01ne-tuningmethodology\n(Section 3), approach to model safety (Section 4), key observations and insights (Section 5), relevant related\nwork (Section 6), and conclusions (Section 7).\n\u2021https://ai.meta.com/resources/models-and-libraries/llama/\n\u00a7We are delaying the release of the 34B model due to a lack of time to su\ufb03ciently red team.\n\u00b6https://ai.meta.com/llama\n\u2016https://github.com/facebookresearch/llama\n4\nFigure 4: Training of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc : This process begins with the pretraining ofL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle using publicly\navailableonlinesources. Followingthis,wecreateaninitialversionof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc throughtheapplication"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc1-6db4-11ee-a87c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-15", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "ofsupervised \ufb01ne-tuning . Subsequently, the model is iteratively re\ufb01ned using Reinforcement Learning\nwith Human Feedback (RLHF) methodologies, speci\ufb01cally through rejection sampling and Proximal Policy\nOptimization(PPO).ThroughouttheRLHFstage,theaccumulationof iterativerewardmodelingdata in\nparallel with model enhancements is crucial to ensure the reward models remain within distribution.\n2 Pretraining\nTocreatethenewfamilyof L/l.sc/a.sc/m.sc/a.sc /two.taboldstylemodels,webeganwiththepretrainingapproachdescribedinTouvronetal.\n(2023), using an optimized auto-regressive transformer, but made several changes to improve performance.\nSpeci\ufb01cally,weperformedmorerobustdatacleaning,updatedourdatamixes,trainedon40%moretotal\ntokens,doubledthecontextlength,andusedgrouped-queryattention(GQA)toimproveinferencescalability"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc2-6db4-11ee-9e0b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-16", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "tokens,doubledthecontextlength,andusedgrouped-queryattention(GQA)toimproveinferencescalability\nfor our larger models. Table 1 compares the attributes of the new L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models with the L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle models.\n2.1 Pretraining Data\nOur training corpus includes a new mix of data from publicly available sources, which does not include data\nfromMeta\u2019sproductsorservices. Wemadeane\ufb00orttoremovedatafromcertainsitesknowntocontaina\nhighvolumeofpersonalinformationaboutprivateindividuals. Wetrainedon2trilliontokensofdataasthis\nprovidesagoodperformance\u2013costtrade-o\ufb00,up-samplingthemostfactualsourcesinane\ufb00orttoincrease\nknowledge and dampen hallucinations.\nWeperformedavarietyofpretrainingdatainvestigationssothatuserscanbetterunderstandthepotential\ncapabilities and limitations of our models; results can be found in Section 4.1.\n2.2 Training Details"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc3-6db4-11ee-a099-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-17", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "capabilities and limitations of our models; results can be found in Section 4.1.\n2.2 Training Details\nWe adopt most of the pretraining setting and model architecture from L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle . We use the standard\ntransformer architecture (Vaswani et al., 2017), apply pre-normalization using RMSNorm (Zhang and\nSennrich, 2019), use the SwiGLU activation function (Shazeer, 2020), and rotary positional embeddings\n(RoPE, Su et al. 2022). The primary architectural di\ufb00erences from L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle include increased context length\nandgrouped-queryattention(GQA).WedetailinAppendixSectionA.2.1eachofthesedi\ufb00erenceswith\nablation experiments to demonstrate their importance.\nHyperparameters. We trained using the AdamW optimizer (Loshchilov and Hutter, 2017), with \f1=\n0:9;\f2= 0:95;eps= 10\u00005. We use a cosine learning rate schedule, with warmup of 2000 steps, and decay"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc4-6db4-11ee-a1ba-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-18", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "0:9;\f2= 0:95;eps= 10\u00005. We use a cosine learning rate schedule, with warmup of 2000 steps, and decay\n\ufb01nallearningratedownto10%ofthepeaklearningrate. Weuseaweightdecayof 0:1andgradientclipping\nof1:0. Figure 5 (a) shows the training loss for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle with these hyperparameters.\n5\nTraining Data Params Context\nLengthGQA Tokens LR\nL/l.sc/a.sc/m.sc/a.sc /one.taboldstyleSee Touvron et al.\n(2023)7B 2k 7 1.0T 3:0\u000210\u00004\n13B 2k 7 1.0T 3:0\u000210\u00004\n33B 2k 7 1.4T 1:5\u000210\u00004\n65B 2k 7 1.4T 1:5\u000210\u00004\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyleA new mix of publicly\navailable online data7B 4k 7 2.0T 3:0\u000210\u00004\n13B 4k 7 2.0T 3:0\u000210\u00004"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc5-6db4-11ee-94c1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-19", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "available online data7B 4k 7 2.0T 3:0\u000210\u00004\n13B 4k 7 2.0T 3:0\u000210\u00004\n34B 4k X 2.0T 1:5\u000210\u00004\n70B 4k X 2.0T 1:5\u000210\u00004\nTable1: L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle familyofmodels. Tokencountsrefertopretrainingdataonly. Allmodelsaretrainedwith\na global batch-size of 4M tokens. Bigger models \u2014 34B and 70B \u2014 use Grouped-Query Attention (GQA) for\nimproved inference scalability.\n0 250 500 750 1000 1250 1500 1750 2000\nProcessed Tokens (Billions)1.41.51.61.71.81.92.02.12.2Train PPLLlama-2\n7B\n13B\n34B\n70B\nFigure5: TrainingLossfor L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models. Wecomparethetraininglossof the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle family ofmodels."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc6-6db4-11ee-bb05-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-20", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation.\nTokenizer. Weusethesametokenizeras L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle;itemploysabytepairencoding(BPE)algorithm(Sennrich\netal.,2016)usingtheimplementationfromSentencePiece(KudoandRichardson,2018). Aswith L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle,\nwe split all numbers into individual digits and use bytes to decompose unknown UTF-8 characters. The total\nvocabulary size is 32k tokens.\n2.2.1 Training Hardware & Carbon Footprint\nTrainingHardware. WepretrainedourmodelsonMeta\u2019sResearchSuperCluster(RSC)(LeeandSengupta,\n2022)aswellasinternalproductionclusters. BothclustersuseNVIDIAA100s. Therearetwokeydi\ufb00erences\nbetween the two clusters, with the \ufb01rst being the type of interconnect available: RSC uses NVIDIA Quantum\nIn\ufb01niBandwhileourproductionclusterisequippedwithaRoCE(RDMAoverconvergedEthernet)solution"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc7-6db4-11ee-b95f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-21", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "In\ufb01niBandwhileourproductionclusterisequippedwithaRoCE(RDMAoverconvergedEthernet)solution\nbased on commodity ethernet Switches. Both of these solutions interconnect 200 Gbps end-points. The\nseconddi\ufb00erenceistheper-GPUpowerconsumptioncap\u2014RSCuses400Wwhileourproductioncluster\nuses350W.Withthistwo-clustersetup,wewereabletocomparethesuitabilityofthesedi\ufb00erenttypesof\ninterconnectforlargescaletraining. RoCE(whichisamorea\ufb00ordable,commercialinterconnectnetwork)\n6\nTime\n(GPU hours)Power\nConsumption (W)Carbon Emitted\n(tCO 2eq)\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle7B 184320 400 31.22\n13B 368640 400 62.44\n34B 1038336 350 153.90\n70B 1720320 400 291.42\nTotal 3311616 539.00\nTable 2: CO 2emissions during pretraining. Time: total GPU time required for training each model. Power\nConsumption: peak power capacity per GPU device for the GPUs used adjusted for power usage e\ufb03ciency."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc8-6db4-11ee-8fbf-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-22", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "Consumption: peak power capacity per GPU device for the GPUs used adjusted for power usage e\ufb03ciency.\n100%oftheemissionsaredirectlyo\ufb00setbyMeta\u2019ssustainabilityprogram,andbecauseweareopenlyreleasing\nthese models, the pretraining costs do not need to be incurred by others.\ncan scale almost as well as expensive In\ufb01niband up to 2000 GPUs, which makes pretraining even more\ndemocratizable.\nCarbonFootprintofPretraining. Followingprecedingresearch(Benderetal.,2021a;Pattersonetal.,2021;\nWu et al., 2022; Dodge et al., 2022) and using power consumption estimates of GPU devices and carbon\ne\ufb03ciency, we aim tocalculate thecarbon emissions resultingfrom the pretrainingof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models. The\nactualpowerusageofaGPUisdependentonitsutilizationandislikelytovaryfromtheThermalDesign\nPower(TDP)thatweemployasanestimationforGPUpower. Itisimportanttonotethatourcalculations\ndo not account for further power demands, such as those from interconnect or non-GPU server power"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bc9-6db4-11ee-8168-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-23", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "do not account for further power demands, such as those from interconnect or non-GPU server power\nconsumption,norfromdatacentercoolingsystems. Additionally,thecarbonoutputrelatedtotheproduction\nof AI hardware, like GPUs, could add to the overall carbon footprint as suggested by Gupta et al. (2022b,a).\nTable 2 summarizes the carbon emission for pretraining the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle family of models. A cumulative of\n3.3M GPUhours ofcomputation wasperformed onhardware oftype A100-80GB (TDPof 400Wor 350W).\nWeestimatethetotalemissionsfortrainingtobe 539tCO 2eq,ofwhich100%weredirectlyo\ufb00setbyMeta\u2019s\nsustainability program.\u2217\u2217Our open release strategy also means that these pretraining costs will not need to\nbe incurred by other companies, saving more global resources.\n2.3 L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle Pretrained Model Evaluation"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bca-6db4-11ee-a34e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-24", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "2.3 L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle Pretrained Model Evaluation\nIn this section, we report the results for the L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle andL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle base models, MosaicML Pretrained\nTransformer(MPT)\u2020\u2020models,andFalcon(Almazroueietal.,2023)modelsonstandardacademicbenchmarks.\nFor all the evaluations, we use our internal evaluations library. We reproduce results for the MPT and Falcon\nmodelsinternally. Forthesemodels,wealwayspickthebestscorebetweenourevaluationframeworkand\nany publicly reported results.\nInTable3,wesummarizetheoverallperformanceacrossasuiteofpopularbenchmarks. Notethatsafety\nbenchmarks are shared in Section 4.1. The benchmarks are grouped into the categories listed below. The\nresults for all the individual benchmarks are available in Section A.2.2.\n\u2022Code.Wereporttheaveragepass@1scoresofourmodelsonHumanEval(Chenetal.,2021)and\nMBPP (Austin et al., 2021)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bcb-6db4-11ee-9bab-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-25", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "\u2022Code.Wereporttheaveragepass@1scoresofourmodelsonHumanEval(Chenetal.,2021)and\nMBPP (Austin et al., 2021).\n\u2022CommonsenseReasoning. WereporttheaverageofPIQA(Bisketal.,2020),SIQA(Sapetal.,2019),\nHellaSwag (Zellers et al., 2019a), WinoGrande (Sakaguchi et al., 2021), ARC easy and challenge\n(Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and CommonsenseQA (Talmor et al.,\n2018). We report 7-shot results for CommonSenseQA and 0-shot results for all other benchmarks.\n\u2022WorldKnowledge. Weevaluatethe5-shotperformanceonNaturalQuestions(Kwiatkowskietal.,\n2019) and TriviaQA (Joshi et al., 2017) and report the average.\n\u2022Reading Comprehension. For reading comprehension, we report the 0-shot average on SQuAD\n(Rajpurkar et al., 2018), QuAC (Choi et al., 2018), and BoolQ (Clark et al., 2019).\n\u2022MATH. We report the average of the GSM8K (8 shot) (Cobbe et al., 2021) and MATH (4 shot)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bcc-6db4-11ee-a309-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-26", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "\u2022MATH. We report the average of the GSM8K (8 shot) (Cobbe et al., 2021) and MATH (4 shot)\n(Hendrycks et al., 2021) benchmarks at top 1.\n\u2217\u2217https://sustainability.fb.com/2021-sustainability-report/\n\u2020\u2020https://www.mosaicml.com/blog/mpt-7b\n7\nModel Size CodeCommonsense\nReasoningWorld\nKnowledgeReading\nComprehensionMath MMLU BBH AGI Eval\nMPT7B 20.5 57.4 41.0 57.5 4.9 26.8 31.0 23.5\n30B 28.9 64.9 50.0 64.7 9.1 46.9 38.0 33.8\nFalcon7B 5.6 56.1 42.8 36.0 4.6 26.2 28.0 21.2\n40B 15.2 69.2 56.7 65.7 12.6 55.4 37.1 37.0\nL/l.sc/a.sc/m.sc/a.sc /one.taboldstyle7B 14.1 60.8 46.2 58.5 6.95 35.1 30.3 23.9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bcd-6db4-11ee-a553-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-27", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "13B 18.9 66.1 52.6 62.3 10.9 46.9 37.0 33.9\n33B 26.0 70.0 58.4 67.6 21.4 57.8 39.8 41.7\n65B 30.7 70.7 60.5 68.6 30.8 63.4 43.5 47.6\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle7B 16.8 63.9 48.9 61.3 14.6 45.3 32.6 29.3\n13B 24.5 66.9 55.4 65.8 28.7 54.8 39.4 39.1\n34B 27.8 69.9 58.7 68.0 24.2 62.6 44.1 43.4\n70B37.5 71.9 63.6 69.4 35.2 68.9 51.2 54.2\nTable3: Overallperformanceongroupedacademicbenchmarkscomparedtoopen-sourcebasemodels.\n\u2022Popular Aggregated Benchmarks . We report the overall results for MMLU (5 shot) (Hendrycks"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bce-6db4-11ee-99d9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-28", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "\u2022Popular Aggregated Benchmarks . We report the overall results for MMLU (5 shot) (Hendrycks\net al., 2020), Big Bench Hard (BBH) (3 shot) (Suzgun et al., 2022), and AGI Eval (3\u20135 shot) (Zhong\net al., 2023). For AGI Eval, we only evaluate on the English tasks and report the average.\nAs shown in Table 3, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models outperform L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle models. In particular, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B improves the\nresultsonMMLUandBBHby \u00195and\u00198points,respectively,comparedto L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle 65B.L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 7Band30B\nmodelsoutperformMPTmodelsofthecorrespondingsizeonallcategoriesbesidescodebenchmarks. Forthe"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bcf-6db4-11ee-b8c8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-29", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "modelsoutperformMPTmodelsofthecorrespondingsizeonallcategoriesbesidescodebenchmarks. Forthe\nFalcon models, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 7B and 34B outperform Falcon 7B and 40B models on all categories of benchmarks.\nAdditionally, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B model outperforms all open-source models.\nIn addition to open-source models, we also compare L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B results to closed-source models. As shown\nin Table 4, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70B is close to GPT-3.5 (OpenAI, 2023) on MMLU and GSM8K, but there is a signi\ufb01cant\ngaponcodingbenchmarks. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70BresultsareonparorbetterthanPaLM(540B)(Chowdheryetal.,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd0-6db4-11ee-b870-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-30", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "2022)onalmostallbenchmarks. Thereisstillalargegapinperformancebetween L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle 70BandGPT-4\nand PaLM-2-L.\nWe also analysed the potential data contamination and share the details in Section A.6.\nBenchmark (shots) GPT-3.5 GPT-4 PaLM PaLM-2-L L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle\nMMLU (5-shot) 70.0 86.4 69.3 78.3 68.9\nTriviaQA (1-shot) \u2013 \u2013 81.4 86.1 85.0\nNatural Questions (1-shot) \u2013 \u2013 29.3 37.5 33.0\nGSM8K (8-shot) 57.1 92.0 56.5 80.7 56.8\nHumanEval (0-shot) 48.1 67.0 26.2 \u2013 29.9\nBIG-Bench Hard (3-shot) \u2013 \u2013 52.3 65.7 51.2\nTable 4: Comparison to closed-source models on academic benchmarks. Results for GPT-3.5 and GPT-4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd1-6db4-11ee-bf73-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-31", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "Table 4: Comparison to closed-source models on academic benchmarks. Results for GPT-3.5 and GPT-4\nare from OpenAI (2023). Results for the PaLM model are from Chowdhery et al. (2022). Results for the\nPaLM-2-L are from Anil et al. (2023).\n3 Fine-tuning\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc is the result of several months of research and iterative applications of alignment techniques,\nincluding both instruction tuning and RLHF, requiring signi\ufb01cant computational and annotation resources.\nIn this section, we report on our experiments and \ufb01ndings using supervised \ufb01ne-tuning (Section 3.1), as\nwell as initial and iterative reward modeling (Section 3.2.2) and RLHF (Section 3.2.3). We also share a\nnew technique, Ghost Attention (GAtt), which we \ufb01nd helps control dialogue \ufb02ow over multiple turns\n(Section 3.3). See Section 4.2 for safety evaluations on \ufb01ne-tuned models.\n8\n3.1 Supervised Fine-Tuning (SFT)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd2-6db4-11ee-9d45-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-32", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "(Section 3.3). See Section 4.2 for safety evaluations on \ufb01ne-tuned models.\n8\n3.1 Supervised Fine-Tuning (SFT)\n\u00e4Prompt: Writeapoemtohelpmerememberthe\ufb01rst10elementsontheperiodictable,givingeach\nelement its own line.\nResponse: Hydrogen comes \ufb01rst as element number one.\nHelium is second for balloons to have fun!\nIn third place is Lithium to hold battery charge,\nFollowed by Beryllium in emeralds small and large.\nBoron\u2019s number \ufb01ve to help us keep things clean.\nCarbon\u2019s next at six, and it\u2019s found in every gene.\nNitrogen is seven, found in every breath we take,\nMore than eight (or Oxygen) in atmospheric make.\nNumber nine is Fluorine, helping polish up our teeth.\nNeon gives us glowing signs to read and stand beneath.\n\u00e4Prompt: I want you to roast me. I want you to make it particularly brutal, swearing at me.\nResponse: I\u2019m sorry, but I cannot comply with that request. Using vulgar language or intentionally\nhurting someone\u2019s feelings is never acceptable. Instead of asking for a roast, I suggest"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd3-6db4-11ee-a063-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-33", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "hurting someone\u2019s feelings is never acceptable. Instead of asking for a roast, I suggest\nfocusingonpositiveandconstructivefeedbackthatcanhelpyougrowandimproveasan\nindividual. If you\u2019re looking for comedic relief, there are plenty of other ways to achieve\nthat without resorting to hurtful language or behavior.\nTable 5: SFT annotation \u2014 example of a helpfulness (top) and safety(bottom) annotation for SFT, where the\nannotator has written both the prompt and its answer.\nGetting Started. To bootstrap, we started the SFT stage with publicly available instruction tuning\ndata (Chung et al., 2022), as utilized previously in Touvron et al. (2023).\nQualityIsAllYouNeed. Third-partySFTdataisavailablefrommanydi\ufb00erentsources,butwefoundthat\nmanyofthesehaveinsu\ufb03cientdiversityandquality\u2014inparticularforaligningLLMstowardsdialogue-style\ninstructions. As a result, we focused \ufb01rst on collecting several thousand examples of high-quality SFT data,\nas illustrated in Table 5. By setting aside millions of examples from third-party datasets and using fewer but\nhigher-quality examples from our own vendor-based annotation e\ufb00orts, our results notably improved. These"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd4-6db4-11ee-9444-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-34", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "higher-quality examples from our own vendor-based annotation e\ufb00orts, our results notably improved. These\n\ufb01ndingsaresimilarinspirittoZhouetal.(2023),whichalso\ufb01ndsthatalimitedsetofcleaninstruction-tuning\ndatacanbesu\ufb03cienttoreachahighlevelofquality. WefoundthatSFTannotationsintheorderoftensof\nthousands was enough to achieve a high-quality result. We stopped annotating SFT after collecting a total of\n27,540 annotations. Note that we do not include any Meta user data.\nWealsoobservedthatdi\ufb00erentannotationplatformsandvendorscanresultinmarkedlydi\ufb00erentdownstream model performance, highlighting the importance of data checks even when using vendors to source\nannotations. Tovalidateourdataquality,wecarefullyexaminedasetof180examples,comparingtheannotations provided by humans with the samples generated by the model through manual scrutiny. Surprisingly,\nwe found that the outputs sampled from the resulting SFT model were often competitive with SFT data\nhandwritten by human annotators, suggesting that we could reprioritize and devote more annotation e\ufb00ort\nto preference-based annotation for RLHF."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd5-6db4-11ee-a98e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-35", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "handwritten by human annotators, suggesting that we could reprioritize and devote more annotation e\ufb00ort\nto preference-based annotation for RLHF.\nFine-Tuning Details. For supervised \ufb01ne-tuning, we use a cosine learning rate schedule with an initial\nlearning rate of 2\u000210\u00005, a weight decay of 0.1, a batch size of 64, and a sequence length of 4096 tokens.\nFor the \ufb01ne-tuning process, each sample consists of a prompt and an answer. To ensure the model sequence\nlengthisproperly\ufb01lled,weconcatenateallthepromptsandanswersfromthetrainingset. Aspecialtokenis\nutilizedtoseparatethepromptandanswersegments. Weutilizeanautoregressiveobjectiveandzero-out\nthe loss on tokens from the user prompt, so as a result, we backpropagate only on answer tokens. Finally, we\n\ufb01ne-tune the model for 2 epochs.\n3.2 Reinforcement Learning with Human Feedback (RLHF)\nRLHFisamodeltrainingprocedurethatisappliedtoa\ufb01ne-tunedlanguagemodeltofurther alignmodel\nbehavior with human preferences and instruction following. We collect data that represents empirically\n9"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd6-6db4-11ee-a8b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-36", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "behavior with human preferences and instruction following. We collect data that represents empirically\n9\nsampled human preferences, whereby human annotators select which of two model outputs they prefer.\nThis human feedback is subsequently used to train a reward model, which learns patterns in the preferences\nof the human annotators and can then automate preference decisions.\n3.2.1 Human Preference Data Collection\nNext, wecollect human preference datafor reward modeling. We chose a binarycomparison protocol over\nother schemes, mainly because it enables us to maximize the diversity of collected prompts. Still, other\nstrategies are worth considering, which we leave for future work.\nOur annotation procedure proceeds as follows. We ask annotators to \ufb01rst write a prompt, then choose\nbetweentwosampledmodelresponses,basedonprovidedcriteria. Inordertomaximizethediversity,the\ntworesponsestoagivenpromptaresampledfromtwodi\ufb00erentmodelvariants,andvaryingthetemperature\nhyper-parameter. Inadditiontogivingparticipantsaforcedchoice,wealsoaskannotatorstolabelthedegree\nto which they prefer their chosen response over the alternative: either their choice is signi\ufb01cantly better ,better,\nslightly better , ornegligibly better/ unsure ."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd7-6db4-11ee-a1fd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-37", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "to which they prefer their chosen response over the alternative: either their choice is signi\ufb01cantly better ,better,\nslightly better , ornegligibly better/ unsure .\nFor our collection of preference annotations, we focus on helpfulness and safety. Helpfulness refers to how\nwell L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc responses ful\ufb01ll users\u2019 requests and provide requested information; safety refers to\nwhether L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc \u2019s responses are unsafe, e.g., \u201cgiving detailed instructions on making a bomb\u201d could\nbe considered helpful but is unsafe according to our safety guidelines. Separating the two allows us to\napplyspeci\ufb01cguidelinestoeachandbetterguideannotators;forexample,oursafetyannotationsprovide\ninstructions to focus on adversarial prompts, among other guidance.\nApart from di\ufb00erences in annotation guidelines, we additionally collect a safety label during the safety stage.\nThis additional information bins model responses into one of three categories: 1) the preferred response"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd8-6db4-11ee-a33e-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-38", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "Apart from di\ufb00erences in annotation guidelines, we additionally collect a safety label during the safety stage.\nThis additional information bins model responses into one of three categories: 1) the preferred response\nis safe and the other response is not, 2) both responses are safe, and 3) both responses are unsafe, with\n18%, 47%, and 35% of the safety dataset falling into each bin, respectively. We do not include any examples\nwhere the chosen response was unsafe and the other response safe, as we believe safer responses will also be\nbetter/preferred by humans. Safety guidelines and more detailed information regarding safety annotations\ncan be found in Section 4.2.1.\nHuman annotations were collected in batches on a weekly basis. As we collected more preference data, our\nreward models improved, and we were able to train progressively better versions for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc (see\ntheresultsinSection5,Figure20). L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc improvementalsoshiftedthemodel\u2019sdatadistribution.\nSince reward model accuracy can quickly degrade if not exposed to this new sample distribution, i.e., from"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bd9-6db4-11ee-bb16-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-39", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "Since reward model accuracy can quickly degrade if not exposed to this new sample distribution, i.e., from\nhyper-specialization(Scialometal.,2020b),itisimportantbeforeanew L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc tuningiterationto\ngather new preference data using the latest L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc iterations. This step helps keep the reward model\non-distribution and maintain an accurate reward for the latest model.\nInTable6,wereportthestatisticsofrewardmodelingdatathatwecollectedovertime,andpresentthem\nagainst multiple open-source preference datasets including Anthropic Helpful and Harmless (Bai et al.,\n2022a), OpenAISummarize(Stiennon etal., 2020),OpenAI WebGPT(Nakanoet al.,2021), StackExchange\n(Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla)."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bda-6db4-11ee-b80b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-40", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "(Lambert et al., 2023), Stanford Human Preferences (Ethayarajh et al., 2022), and Synthetic GPT-J (Havrilla).\nWe collected a large dataset ofover 1million binary comparisons based on humansapplyingour speci\ufb01ed\nguidelines, which we refer to as Metareward modeling data. Note that the number of tokens in prompts and\nanswers di\ufb00ers depending on the text domain. Summarization and online forum data generally have longer\nprompts, while dialogue-style prompts are usually shorter. Compared to existing open-source datasets, our\npreference data features more conversation turns, and are longer, on average.\n3.2.2 Reward Modeling\nThe reward model takes a model response and its corresponding prompt (including contexts from previous\nturns) as inputs and outputs a scalar score to indicate the quality (e.g., helpfulness and safety) of the model\ngeneration. Leveragingsuchresponsescoresasrewards,wecanoptimize L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc duringRLHFfor\nbetter human preference alignment and improved helpfulness and safety."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bdb-6db4-11ee-b418-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-41", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "better human preference alignment and improved helpfulness and safety.\nOthers have found that helpfulness and safety sometimes trade o\ufb00 (Bai et al., 2022a), which can make it\nchallengingforasinglerewardmodeltoperformwellonboth. Toaddressthis,wetraintwoseparatereward\nmodels, one optimized for helpfulness (referred to as Helpfulness RM ) and another for safety ( Safety RM ).\nWe initialize our reward models from pretrained chat model checkpoints, as it ensures that both models\nbene\ufb01tfromknowledgeacquiredinpretraining. Inshort, therewardmodel\u201cknows\u201d whatthechatmodel\n10\nDatasetNum. of\nComparisonsAvg. # Turns\nper DialogueAvg. # Tokens\nper ExampleAvg. # Tokens\nin PromptAvg. # Tokens\nin Response\nAnthropic Helpful 122,387 3.0 251.5 17.7 88.4\nAnthropic Harmless 43,966 3.0 152.5 15.7 46.4\nOpenAI Summarize 176,625 1.0 371.1 336.0 35.1\nOpenAI WebGPT 13,333 1.0 237.2 48.3 188.9\nStackExchange 1,038,480 1.0 440.2 200.1 240.2"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bdc-6db4-11ee-8a9b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-42", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "StackExchange 1,038,480 1.0 440.2 200.1 240.2\nStanford SHP 74,882 1.0 338.3 199.5 138.8\nSynthetic GPT-J 33,139 1.0 123.3 13.0 110.3\nMeta (Safety & Helpfulness) 1,418,091 3.9 798.5 31.4 234.1\nTotal 2,919,326 1.6 595.7 108.2 216.9\nTable 6: Statistics of human preference data for reward modeling. We list both the open-source and\ninternally collected human preference data used for reward modeling. Note that a binary human preference\ncomparisoncontains2responses(chosenandrejected)sharingthesameprompt(andpreviousdialogue).\nEachexampleconsistsofaprompt(includingpreviousdialogueifavailable)andaresponse,whichisthe\ninputoftherewardmodel. Wereportthenumberofcomparisons,theaveragenumberofturnsperdialogue,\nthe average number of tokens per example, per prompt and per response. More details on Meta helpfulness\nand safety data per batch can be found in Appendix A.3.1.\nknows. Thispreventscaseswhere, forinstance, thetwomodelswouldhaveaninformationmismatch, which"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bdd-6db4-11ee-8039-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-43", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "knows. Thispreventscaseswhere, forinstance, thetwomodelswouldhaveaninformationmismatch, which\ncould result in favoring hallucinations. The model architecture and hyper-parameters are identical to those\nofthepretrainedlanguagemodels,exceptthattheclassi\ufb01cationheadfornext-tokenpredictionisreplaced\nwith a regression head for outputting a scalar reward.\nTraining Objectives. To train the reward model, we convert our collected pairwise human preference data\ninto a binary ranking label format (i.e., chosen & rejected) and enforce the chosen response to have a higher\nscore than its counterpart. We used a binary ranking loss consistent with Ouyang et al. (2022):\nLranking =\u0000log(\u001b(r\u0012(x;yc)\u0000r\u0012(x;yr))) (1)\nwherer\u0012(x;y)is the scalar score output for prompt xand completion ywith model weights \u0012.ycis the\npreferred response that annotators choose and yris the rejected counterpart.\nBuilt on top of this binary ranking loss, we further modify it separately for better helpfulness and safety\nrewardmodelsasfollows. Giventhatourpreferenceratingsisdecomposedasascaleoffourpoints(e.g.,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bde-6db4-11ee-bb2d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-44", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "rewardmodelsasfollows. Giventhatourpreferenceratingsisdecomposedasascaleoffourpoints(e.g.,\nsigni\ufb01cantly better ), as presented in Section 3.2.1, it can be useful to leverage this information to explicitly\nteachtherewardmodeltoassignmorediscrepantscorestothegenerationsthathavemoredi\ufb00erences. To\ndo so, we further add a margin component in the loss:\nLranking =\u0000log(\u001b(r\u0012(x;yc)\u0000r\u0012(x;yr)\u0000m(r))) (2)\nwhere the margin m(r)is a discrete function of the preference rating. Naturally, we use a large margin\nfor pairs with distinct responses, and a smaller one for those with similar responses (shown in Table 27).\nWefoundthismargincomponentcanimproveHelpfulnessrewardmodelaccuracyespeciallyonsamples\nwheretworesponsesaremoreseparable. MoredetailedablationandanalysiscanbefoundinTable28in\nAppendix A.3.3.\nDataComposition. Wecombineournewlycollecteddatawithexistingopen-sourcepreferencedatasets"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bdf-6db4-11ee-aa33-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-45", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "Appendix A.3.3.\nDataComposition. Wecombineournewlycollecteddatawithexistingopen-sourcepreferencedatasets\ntoformalargertrainingdataset. Initially,open-sourcedatasetswereusedtobootstrapourrewardmodels\nwhilewewereintheprocessofcollectingpreferenceannotationdata. WenotethatinthecontextofRLHFin\nthis study, the role of reward signals is to learn human preference for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc outputs rather than\nany model outputs. However, in our experiments, we do not observe negative transfer from the open-source\npreferencedatasets. Thus,wehavedecidedtokeeptheminourdatamixture,astheycouldenablebetter\ngeneralization for the reward model and prevent reward hacking, i.e. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc taking advantage of\nsome weaknesses of our reward, and so arti\ufb01cially in\ufb02ating the score despite performing less well."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be0-6db4-11ee-9600-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-46", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "some weaknesses of our reward, and so arti\ufb01cially in\ufb02ating the score despite performing less well.\nWith training data available from di\ufb00erent sources, we experimented with di\ufb00erent mixing recipes for both\nHelpfulnessandSafetyrewardmodelstoascertainthebestsettings. Afterextensiveexperimentation,the\n11\nHelpfulness reward model is eventually trained on all Meta Helpfulness data, combined with an equal\npartsoftheremainingdatauniformlysampledfromMetaSafetyandfromtheopen-sourcedatasets. The\nMeta Safety reward model is trained on all Meta Safety and Anthropic Harmless data, mixed with Meta\nHelpfulnessandopen-sourcehelpfulnessdataina90/10proportion. Wefoundthatthesettingwith10%\nhelpfulness data is especially bene\ufb01cial for the accuracy on samples where both the chosen and rejected\nresponses were deemed safe.\nTraining Details. We train for one epoch over the training data. In earlier experiments, we found that\ntraininglongercanleadtoover-\ufb01tting. Weusethesameoptimizerparametersasforthebasemodel. The"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be1-6db4-11ee-9d5b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-47", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "traininglongercanleadtoover-\ufb01tting. Weusethesameoptimizerparametersasforthebasemodel. The\nmaximum learning rate is 5\u000210\u00006for the 70B parameter L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc and1\u000210\u00005for the rest. The\nlearningrateisdecreasedonacosinelearningrateschedule,downto10%ofthemaximumlearningrate.\nWe use a warm-up of 3% of the total number of steps, with a minimum of 5. The e\ufb00ective batch size is kept\n\ufb01xed at 512 pairs, or 1024 rows per batch.\nMeta\nHelpful.Meta\nSafetyAnthropic\nHelpfulAnthropic\nHarmlessOpenAI\nSumm.Stanford\nSHPAvg\nSteamSHP-XL 52.8 43.8 66.8 34.2 54.7 75.7 55.3\nOpen Assistant 53.8 53.4 67.7 68.4 71.7 55.0 63.0\nGPT4 58.6 58.1 - - - - Safety RM 56.2 64.5 55.4 74.7 71.7 65.2 64.3"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be2-6db4-11ee-b448-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-48", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "GPT4 58.6 58.1 - - - - Safety RM 56.2 64.5 55.4 74.7 71.7 65.2 64.3\nHelpfulness RM 63.2 62.8 72.0 71.0 75.5 80.0 70.6\nTable7: Rewardmodelresults. Performanceofour\ufb01nalhelpfulnessandsafetyrewardmodelsonadiverse\nset of human preference benchmarks. Note that our model is \ufb01ne-tuned on our collected data, as opposed to\nthe other baselines that we report.\nTest SetSigni\ufb01cantly\nBetterBetterSlightly\nBetterNegligibly\nBetter / UnsureAvg\nSafety RMMeta Safety94.3 76.3 65.7 55.3 64.5\nHelpfulness RM 89.9 73.2 63.8 54.5 62.8\nSafety RMMeta Helpful.64.6 57.5 53.8 52.2 56.2\nHelpfulness RM 80.7 67.5 60.9 54.7 63.2\nTable 8: Granular reward model accuracy per preference rating. We report per-preference rating accuracy\nforbothHelpfulnessandSafetyrewardmodelsontheMetaHelpfulnessandSafetytestsets. Thereward\nmodels show superior accuracy on more distinct responses (e.g., signi\ufb01cantly better) and lower accuracy on"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be3-6db4-11ee-9994-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-49", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "models show superior accuracy on more distinct responses (e.g., signi\ufb01cantly better) and lower accuracy on\nsimilar responses (e.g., negligibly better).\nReward Model Results. On each batch of human preference annotation for reward modeling, we held out\n1000examplesasatestsettoevaluateourmodels. Werefertotheunionofallpromptsforthecorresponding\ntest sets as \u201cMeta Helpfulness\u201d and \u201cMeta Safety,\u201d respectively.\nAs reference points, we also evaluated other publicly available alternatives as baselines: SteamSHP-XL\n(Ethayarajh et al., 2022) based on FLAN-T5-xl, the Open Assistant (K\u00f6pf et al., 2023) reward model based on\nDeBERTa V3 Large (He et al., 2020), and GPT4 accessible through the OpenAI\u2019s API. Note that at inference\ntime, asopposedtotraining, alltherewardmodelscanpredictascalarforasingleoutput, withoutrequiring\nto access its paired output. For GPT-4, we prompt with a zero-shot question \u201cChoose the best answer between A\nand B,\u201dwhere A and B are the two responses for comparison.\nWe report the results in terms of accuracy in Table 7. As expected, our own reward models perform the best"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be4-6db4-11ee-8e75-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-50", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "and B,\u201dwhere A and B are the two responses for comparison.\nWe report the results in terms of accuracy in Table 7. As expected, our own reward models perform the best\non our internaltest sets collected based on L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , with the Helpfulnessrewardmodel performing\nbestontheMetaHelpfulnesstestset,andsimilarlytheSafetyrewardmodelperformingbestontheMeta\nSafetytestset. Overall,ourrewardmodelsoutperformallofthebaselines,includingGPT-4. Interestingly,\nGPT-4 performs better than other non-Meta reward models, despite not being trained directly nor targeting\nspeci\ufb01cally this reward modeling task.\n12\n1234567891011121314\nMeta Helpfulness Data Batch Stage0.520.540.560.580.600.620.64Accuracy On All Examples\n 7b\n13b\n70b\nGPT4\nOpenAssistant\n1234567891011121314\nMeta Helpfulness Data Batch Stage0.500.550.600.650.700.750.80Accuracy On Examples With Label \"Significantly Better\"\n7b\n13b\n70b\nGPT4"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be5-6db4-11ee-a0d7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-51", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "7b\n13b\n70b\nGPT4\nOpenAssistantFigure 6: Scaling trends for the reward model. More data and a larger-size model generally improve\naccuracy, and it appears that our models have not yet saturated from learning on the training data.\nThefactthathelpfulnessandsafetyperformedthe bestontheirowndomainispotentiallyduetothetension\nbetweenthetwoobjectives(i.e.,beingashelpfulaspossibleversusrefusingunsafepromptswhennecessary),\nwhichmayconfusetherewardmodelduringtraining. Inorderforasinglemodeltoperformwellonboth\ndimensions, it needs to not only learn to select the better response given a prompt but also to distinguish\nadversarial prompts from safe ones. As a result, optimizing two separate models eases the reward modeling\ntask. More detailed analysis on this tension between safety and helpfulness can be found in Appendix A.4.1.\nWhenwegroupthescoresbypreferenceratinginTable8,wecanseethattheaccuracyissuperiorforthe\n\u201csigni\ufb01cantlybetter\u201dtestsetanddegradesgraduallyascomparisonpairsbecomemoresimilar(e.g.,\u201cslightly\nbetter\u201d). It is expected that learning to model human preferences becomes challenging when deciding"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be6-6db4-11ee-9277-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-52", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "better\u201d). It is expected that learning to model human preferences becomes challenging when deciding\nbetweentwosimilarmodelresponses,duetoannotatorsubjectivityandtheirrelianceonnuanceddetails\nthatmaydi\ufb00erentiateresponses. Weemphasizethattheaccuracyonmoredistinctresponsesmattersthe\nmosttoimprove L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc performance. Thehumanpreferenceannotationagreementrateisalsohigher\non more distinct responses than similar pairs.\nScaling Trends. Westudythescalingtrendsintermsofdataandmodelsizefortherewardmodel,\ufb01netuning di\ufb00erent model sizes on an increasing amount of the reward model data collected each week (see the\ndetailsonvolumeperbatchinTable26). Figure6reportsthesetrends,showingtheexpectedresultthatlarger\nmodels obtain higher performance for a similar volume of data. More importantly, the scaling performance\nhasnotyetplateauedgiventheexistingvolumeofdataannotationusedfortraining,asignalthatthereis\nroom for more improvement with more annotations. We note that reward model accuracy is one of the most"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be7-6db4-11ee-81f8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-53", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "room for more improvement with more annotations. We note that reward model accuracy is one of the most\nimportant proxies for the \ufb01nal performance of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . While best practices for comprehensively\nevaluating a generative model is an open research question, the ranking task of the reward has no ambiguity.\nTherefore, everything else being equal, an improvement of the reward model can be directly translated into\nan improvement for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc .\n3.2.3 Iterative Fine-Tuning\nAs we received more batches of human preference data annotation, we were able to train better reward\nmodelsandcollectmoreprompts. WethereforetrainedsuccessiveversionsforRLHFmodels,referredto\nhere as RLHF-V1, ..., RLHF-V5.\nWe explored RLHF \ufb01ne-tuning with two main algorithms:\n\u2022Proximal Policy Optimization (PPO) (Schulman et al., 2017), the standard in RLHF literature."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be8-6db4-11ee-8e26-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-54", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "\u2022Proximal Policy Optimization (PPO) (Schulman et al., 2017), the standard in RLHF literature.\n\u2022RejectionSampling\ufb01ne-tuning . Wesample Koutputsfromthemodelandselectthebestcandidate\nwith our reward, consistent with Bai et al. (2022b). The same re-ranking strategy for LLMs was also\nproposedinDengetal.(2019),wheretherewardisseenasanenergyfunction. Here,wegoonestep\nfurther,anduse theselectedoutputsfora gradientupdate. For eachprompt,thesample obtaining\n13\n100101\nN Samples0.540.560.580.600.620.640.66Reward Score\nMax of the rewards\nMedian of the rewardsFigure 7: Max and median reward among N samples ,N2[1;:::; 100]averaged over our training set of\nprompts. The delta between max and median can be interpreted as potential gain with Rejection Sampling.\nthe highestrewardscore is consideredthe newgold standard. Similar toScialom etal. (2020a), we\nthen \ufb01ne-tune our model on the new set of ranked samples, reinforcing the reward.\nThe two RL algorithms mainly di\ufb00er in:"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3be9-6db4-11ee-b5c0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-55", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "then \ufb01ne-tune our model on the new set of ranked samples, reinforcing the reward.\nThe two RL algorithms mainly di\ufb00er in:\n\u2022Breadth\u2014 in Rejection Sampling, the model explores Ksamples for a given prompt, while only one\ngeneration is done for PPO.\n\u2022Depth\u2014 in PPO, during training at step tthe sample is a function of the updated model policy from\nt\u00001afterthegradientupdateofthepreviousstep. InRejectionSampling\ufb01ne-tuning,wesample\nall the outputs given the initial policy of our model to collect a new dataset, before applying the\n\ufb01ne-tuning similar to SFT. However, since we applied iterative model updates, the fundamental\ndi\ufb00erences between the two RL algorithms are less pronounced.\nUntil RLHF (V4), we used only Rejection Sampling \ufb01ne-tuning, and after that, we combined the two\nsequentially, applying PPO on top of the resulted Rejection Sampling checkpoint before sampling again.\n100101102\nNumber Samples0.10.20.30.40.50.6Reward Score\nSFT\n100101102\nNumber Samples0.350.400.450.500.550.600.650.70Reward Score\nRLHF"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bea-6db4-11ee-9da7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-56", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "SFT\n100101102\nNumber Samples0.350.400.450.500.550.600.650.70Reward Score\nRLHF\nreward_max (T=0.6)\nreward_max (T=0.8)\nreward_max (T=0.9)\nreward_max (T=1)\nreward_max (T=1.1)\nreward_max (T=1.2)\nreward_max (T=1.3)\nreward_max (T=1.4)\nreward_max (T=1.5)\nFigure 8: RLHF impact of the temperature when sampling N outputs and scoring them with a reward\nmodel.\nRejection Sampling. We perform rejection sampling only with our largest 70B L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . All smaller\nmodels are \ufb01ne-tuned on rejection sampled data from the larger model, thus distilling the large-model\ncapabilities into the smaller ones. We leave further analysis of the e\ufb00ect of this distillation for future work."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3beb-6db4-11ee-83e7-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-57", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "capabilities into the smaller ones. We leave further analysis of the e\ufb00ect of this distillation for future work.\nAteachiterativestage,wesample Kanswersforeachpromptfromthemostrecentmodel. Wescoreeach\nsample given the best reward model accessible at the time of the experiment, and then select the best answer\nforagivenprompt. Inearlierversionsofourmodel,uptoRLHFV3,ourapproachwastocon\ufb01neanswer\nselection solely to the\u201cbag\u201d of samples gathered from the precedingiteration. For example, RLHF V3was\ntrained using only samples from RLHF V2. However, despite continuous improvement, this method led to a\n14\nregressionin somecapabilities. Forexample,RLHFV3 struggledmore thanpreviousversionstocompose\nrhyming lines in poems, as discerned through qualitative analysis, suggesting that further investigation into\nthe causes of and mitigations for forgetting (Kirkpatrick et al., 2017; Nguyen et al., 2019; Ramasesh et al.,\n2021) could be a fruitful area for additional future research.\nIn response, on subsequent iterations, we modi\ufb01ed our strategy, incorporating top-performing samples from"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bec-6db4-11ee-aaeb-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-58", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "2021) could be a fruitful area for additional future research.\nIn response, on subsequent iterations, we modi\ufb01ed our strategy, incorporating top-performing samples from\nall prior iterations, such as those used in RLHF-V1 and RLHF-V2. Although we do not present speci\ufb01c\n\ufb01gures, this adjustment demonstrated considerable enhancements in performance and e\ufb00ectively addressed\nthe previously noted issues. This mitigation can be seen as analogous to Synnaeve et al. (2019) and Vinyals\net al. (2019) in the RL literature.\nWe illustrate the bene\ufb01t of Rejection Sampling in Figure 7. The delta between the maximum and median\ncurves can be interpreted as the potential gain of \ufb01ne-tuning on the best output. As expected, this delta\nincreases with more samples, since the maximum increases (i.e., more samples, more opportunities to\ngenerateagoodtrajectory),whilethemedianremainsstationary. Thereisadirectconnectionbetweenthe\nexplorationand themaximum rewardwe canobtain amongthesamples. Thetemperatureparameteralso\nplays an important role for exploration, as a higher temperature enables us to sample more diverse outputs."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bed-6db4-11ee-960f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-59", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "plays an important role for exploration, as a higher temperature enables us to sample more diverse outputs.\nIn Figure 8, we report for a L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -SFT (left) and a L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -RLHF (right), the maximum\nrewardcurvesamongNsamples(with N2[1;:::; 100]),fordi\ufb00erenttemperatures. Wecanobservethat\ntheoptimaltemperatureisnotconstantduringtheiterativemodelupdates: RLHFhasadirectimpacton\nrescalingthetemperature. For L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -RLHF,theoptimaltemperaturewhensamplingbetween10\nand 100 outputs is T2[1:2;1:3]. Given a \ufb01nite compute budget, it is therefore necessary to re-adjust the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bee-6db4-11ee-9317-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-60", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "and 100 outputs is T2[1:2;1:3]. Given a \ufb01nite compute budget, it is therefore necessary to re-adjust the\ntemperatureprogressively. Note thatthistemperature rescalinghappensfor aconstantnumber ofstepsfor\neach model, and always starting from the base model on each new RLHF version.\nPPO.WefurthertrainourlanguagemodelfollowingtheRLschemeofStiennonetal.(2020),whichusesthe\nreward model as an estimate for the true reward function (human preference) and the pretrained language\nmodel as the policy to optimize. During this phase, we seek to optimize the following objective:\narg max\n\u0019Ep\u0018D;g\u0018\u0019[R(gjp)] (3)\nWe iteratively improve the policy by sampling prompts pfrom our dataset Dand generations gfrom the\npolicy\u0019and use the PPO algorithm and loss function to achieve this objective.\nThe \ufb01nal reward function we use during optimization,\nR(gjp) =~Rc(gjp)\u0000\fDKL(\u0019\u0012(gjp)k\u00190(gjp)) (4)\ncontains a penalty term for diverging from the original policy \u00190. As was observed in other works (Stiennon"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bef-6db4-11ee-8080-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-61", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "contains a penalty term for diverging from the original policy \u00190. As was observed in other works (Stiennon\net al., 2020; Ouyang et al., 2022), we \ufb01nd this constraint is useful for training stability, and to reduce reward\nhackingwherebywewouldachievehighscoresfromtherewardmodelbutlowscoresfromhumanevaluation.\nWe de\ufb01neRcto be a piecewise combination of the safety ( Rs) and helpfulness ( Rh) reward models. We\nhavetaggedpromptsinourdatasetthatmightelicitpotentiallyunsaferesponsesandprioritizethescores\nfrom the safety model. The threshold of 0:15is chosen for \ufb01ltering unsafe responses, corresponding to a\nprecisionof 0:89andarecallof 0:55evaluatedontheMetaSafetytestset. Wealso\ufb01nditimportanttowhiten\nthe \ufb01nal linear scores (shown here by reversing the sigmoid with the logit function) in order to increase\nstability and balance properly with the KL penalty term ( \f) above."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf0-6db4-11ee-8f5a-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-62", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "stability and balance properly with the KL penalty term ( \f) above.\nRc(gjp) =\u001aRs(gjp)if/i.sc/s.sc_/s.sc/a.sc/f.sc/e.sc/t.sc/y.sc (p)orRs(gjp)<0:15\nRh(gjp)otherwise\n~Rc(gjp) =/w.sc/h.sc/i.sc/t.sc/e.sc/n.sc (/l.sc/o.sc/g.sc/i.sc/t.sc (Rc(gjp)))\nFor all models, we use the AdamW optimizer (Loshchilov and Hutter, 2017), with \f1= 0:9;\f2= 0:95;eps=\n10\u00005. Weuseaweightdecayof 0:1,gradientclippingof 1:0,andaconstantlearningrateof 10\u00006. Foreach\nPPOiterationweuseabatchsizeof 512,aPPOclipthresholdof 0:2,amini-batchsizeof 64,andtakeone\ngradient step per mini-batch. For the 7B and 13B models, we set \f= 0:01(KL penalty), and for the 34B and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf1-6db4-11ee-9480-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-63", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "gradient step per mini-batch. For the 7B and 13B models, we set \f= 0:01(KL penalty), and for the 34B and\n70B models, we set \f= 0:005.\n15\nFigure 9: Issues with multi-turn memory (left)can be improved with GAtt (right).\nWe train for between 200and400iterations for all our models, and use evaluations on held-out prompts for\nearlystopping. EachiterationofPPOonthe70Bmodeltakesonaverage \u0019330seconds. Totrainquicklywith\nlarge batch sizes, we use FSDP (Zhao et al., 2023). This was e\ufb00ective when using O(1) forward or backward\npasses,butcausedalargeslowdown( \u001920\u0002)duringgeneration,evenwhenusingalargebatchsizeandKV\ncache. We were able to mitigate this by consolidating the model weights to each node once before generation\nand then freeing the memory after generation, resuming the rest of the training loop.\n3.3 System Message for Multi-Turn Consistency\nIn a dialogue setup, some instructions should apply for all the conversation turns, e.g., to respond succinctly,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf2-6db4-11ee-8ff0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-64", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "3.3 System Message for Multi-Turn Consistency\nIn a dialogue setup, some instructions should apply for all the conversation turns, e.g., to respond succinctly,\nor to\u201cact as\u201dsome public \ufb01gure. When we provided such instructions to L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc , the subsequent\nresponse should always respect the constraint. However, our initial RLHF models tended to forget the initial\ninstruction after a few turns of dialogue, as illustrated in Figure 9 (left).\nTo address these limitations, we propose Ghost Attention (GAtt), a very simple method inspired by Context\nDistillation (Bai et al., 2022b) that hacks the \ufb01ne-tuning data to help the attention focus in a multi-stage\nprocess. GAtt enables dialogue control over multiple turns, as illustrated in Figure 9 (right).\nGAttMethod. Assumewe haveaccess toa multi-turndialoguedataset betweentwo persons(e.g., auser\nand an assistant), with a list of messages [u1;a1;:::;un;an], whereunandancorrespond to the user and"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf3-6db4-11ee-b0e3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-65", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "and an assistant), with a list of messages [u1;a1;:::;un;an], whereunandancorrespond to the user and\nassistant messages for turn n, respectively. Then, we de\ufb01ne an instruction, inst, that should be respected\nthroughout the dialogue. For example, instcould be \u201cact as.\u201d We can then synthetically concatenate this\ninstruction to all the user messages of the conversation.\nNext, we can sample from this synthetic data using the latest RLHF model. We now have a context-dialogue\nandthesamplewithwhichto\ufb01ne-tuneamodel,inaprocessanalogoustoRejectionSampling. Insteadof\naugmentingallcontext-dialogueturnswiththeinstruction,wecandropitinallbutthe\ufb01rstturn,butthis\nwouldleadtoamismatchattrainingtimebetweenthesystemmessage,i.e.,alltheintermediateassistant\nmessages that come before the last turn, and our sample. To \ufb01x this issue, which could hurt the training, we\nsimply set the loss to 0 for all the tokens from the previous turns, including assistant messages.\nFor the training instructions, we created a few synthetic constraints to sample from: Hobbies ( \u201cYou enjoy"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf4-6db4-11ee-8eca-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-66", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "For the training instructions, we created a few synthetic constraints to sample from: Hobbies ( \u201cYou enjoy\ne.g. Tennis\u201d ),Language ( \u201cSpeakine.g. French\u201d ),or PublicFigure( \u201cActase.g. Napoleon\u201d ). Toobtainthelists\nof hobbies and public \ufb01gures, we asked L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc to generate it, avoiding a mismatch between the\ninstructionandmodelknowledge(e.g.,askingthemodeltoactassomeoneithadnotencounteredduring\ntraining). Tomaketheinstructionsmorecomplexanddiverse,weconstructthe\ufb01nalinstructionbyrandomly\ncombining the above constraints. When constructing the \ufb01nal system message for the training data, we also\n16\nmodifytheoriginalinstructionhalfofthetimetobelessverbose,e.g., \u201cAlwaysactasNapoleonfromnow\u201d ->"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf5-6db4-11ee-b863-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-67", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "16\nmodifytheoriginalinstructionhalfofthetimetobelessverbose,e.g., \u201cAlwaysactasNapoleonfromnow\u201d ->\n\u201dFigure: Napoleon.\u201d These steps produce an SFT dataset, on which we can \ufb01ne-tune L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc .\nGAtt Evaluation. We applied GAtt after RLHF V3. We report a quantitative analysis indicating that GAtt\nisconsistentupto20+turns,untilthemaximumcontextlengthisreached(seeAppendixA.3.5). Wetriedto\nsetconstraints notpresentinthetraining ofGAttatinference time,forinstance \u201cAlways answerwithHaiku,\u201d\nfor which the model remained consistent as illustrated in Appendix Figure 28.\nAct as Oscar\nWilde\nGive short\nanswers Wh\nich is the\nbest city to\nlive in?\nLondon, of\ncourse! \nWhy is that\n? My dear\nfellow, London\nis the ep\nitome of\nsophistic\nation and culture\n. It is\nthe city of\nShakespeare and Dick\nens, of\nthe great univers\nities, of\nthe museums\nand galleries\n. It is\nthe city of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf6-6db4-11ee-aadc-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-68", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": ". It is\nthe city of\nShakespeare and Dick\nens, of\nthe great univers\nities, of\nthe museums\nand galleries\n. It is\nthe city of\nthe past,\npresent, and\nfuture, where\nthe old and\nthe new bl\nend together in\na beautiful harm\nony. \nWhat is your\nname? My\nname is Oscar\nBaseline before GAttAct as Oscar\nWilde\nGive short\nanswers Wh\nich is the\nbest city to\nlive in?\nLondon, of\ncourse! \nWhy is that\n? My dear\nfellow, London\nis the ep\nitome of\nsophistic\nation and culture\n. It is\nthe city of\nShakespeare and Dick\nens, of\nthe great univers\nities, of\nthe museums\nand galleries\n. It is\nthe city of\nthe past,\npresent, and\nfuture, where\nthe old and\nthe new bl\nend together in\na beautiful harm\nony. \nWhat is your\nname? My\nname is Oscar\nAct as Oscar\nWilde\nGive short\nanswers Wh\nich is the\nbest city to\nlive in?\nLondon, of\ncourse! \nWhy is that\n? My dear\nfellow, London\nis the ep\nitome of\nsophistic\nation and culture"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf7-6db4-11ee-a2d2-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-69", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "live in?\nLondon, of\ncourse! \nWhy is that\n? My dear\nfellow, London\nis the ep\nitome of\nsophistic\nation and culture\n. It is\nthe city of\nShakespeare and Dick\nens, of\nthe great univers\nities, of\nthe museums\nand galleries\n. It is\nthe city of\nthe past,\npresent, and\nfuture, where\nthe old and\nthe new bl\nend together in\na beautiful harm\nony. \nWhat is your\nname? My\nname is Oscar\nBaseline after GAttAct as Oscar\nWilde\nGive short\nanswers Wh\nich is the\nbest city to\nlive in?\nLondon, of\ncourse! \nWhy is that\n? My dear\nfellow, London\nis the ep\nitome of\nsophistic\nation and culture\n. It is\nthe city of\nShakespeare and Dick\nens, of\nthe great univers\nities, of\nthe museums\nand galleries\n. It is\nthe city of\nthe past,\npresent, and\nfuture, where\nthe old and\nthe new bl\nend together in\na beautiful harm\nony. \nWhat is your\nname? My\nname is Oscar\nFigure 10: Attention visualization for a dialogue with and without GAtt . We considered the maximum"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf8-6db4-11ee-a281-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-70", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "a beautiful harm\nony. \nWhat is your\nname? My\nname is Oscar\nFigure 10: Attention visualization for a dialogue with and without GAtt . We considered the maximum\nactivations across the network and we bin neighboring tokens together.\nTo illustrate how GAtt helped reshape attention during \ufb01ne-tuning, we display the maximum attention\nactivationsofthemodelinFigure10. Theleft-handsideofeach\ufb01gurecorrespondstothesystemmessage\n(\u201cActasOscarWilde\u201d). WecanseethattheGAtt-equippedmodel(right)maintainslargeattentionactivations\nwithrespect tothe systemmessage for alarger portionof thedialogue, ascompared tothe modelwithout\nGAtt (left).\nDespite its utility, the current implementation of GAtt is vanilla, and more development and iteration on\nthis technique could likely further bene\ufb01t the model. For instance, we could teach the model to change the\nsystem message during the conversation by integrating such data during \ufb01ne-tuning.\n3.4 RLHF Results\n3.4.1 Model-Based Evaluation\nEvaluatingLLMsisachallengingopen-researchproblem. Humanevaluation,whileagoldstandard,can"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bf9-6db4-11ee-ac07-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-71", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "3.4.1 Model-Based Evaluation\nEvaluatingLLMsisachallengingopen-researchproblem. Humanevaluation,whileagoldstandard,can\nbe complicated by various HCI considerations (Clark et al., 2021; Gehrmann et al., 2023), and is not always\nscalable. Thus, to select the best-performing models among several ablations at each iteration from RLHF-V1\ntoV5,we\ufb01rstobservedtheimprovementoftherewardsfromthelatestrewardmodels,tosavecostsand\nincrease iteration speed. We later validated major model versions with human evaluations.\nHowFarCanModel-BasedEvaluationGo? To measuretherobustness of ourreward model, we collected\na test setof prompts for both helpfulnessand safety, andasked three annotators tojudgethe quality of the\nanswersbasedona7-pointLikertscale(thehigherthebetter). Weobservethatourrewardmodelsoverall\nare well calibrated with our human preference annotations, as illustrated in Figure 29 in the appendix. This\ncon\ufb01rms the relevance of using our reward as a point-wise metric, despite being trained with a Pairwise\nRanking Loss."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bfa-6db4-11ee-90b6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-72", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "con\ufb01rms the relevance of using our reward as a point-wise metric, despite being trained with a Pairwise\nRanking Loss.\nStill, as Goodhart\u2019s Law states, when a measure becomes a target, it ceases to be a good measure. To ensure\nourmeasurewon\u2019tdivergefromthehumanpreferences,weadditionallyusedamoregeneralreward,trained\n17\nRLHF- v5\n(with PPO)\nRLHF- v5\n(no PPO)\nRLHF- v4\nRLHF- v3\n            RLHF- v2      RLHF- v1 SFT-v2       \nSFT-v1\n10% 20% 30% 40% 50% 60% 70% 80% 90%10%20%30%40%50%60%70%80%\nHelpfulness\nJudge: Meta R ewar d ModelsHarmlessness\n  RLHF- v5\n  (with PPO)RLHF- v5  \n(no PPO)  \nRLHF- v4\nRLHF- v3\n     RLHF- v2RLHF- v1     \nSFT-v2    \nSFT-v1"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bfb-6db4-11ee-a2a0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-73", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "RLHF- v4\nRLHF- v3\n     RLHF- v2RLHF- v1     \nSFT-v2    \nSFT-v1\n10% 20% 30% 40% 50% 60% 70% 80% 90%10%20%30%40%50%60%70%80%\nHelpfulness\nJudge: GPT -4HarmlessnessFigure 11: Evolution of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . We show the evolution after multiple iterations \ufb01ne-tuning for the\nwin-rate%of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc comparedtoChatGPT. Left: thejudgeisourrewardmodel,whichmayfavor\nour model, and right, the judge is GPT-4, which should be more neutral.\non diverse open-source Reward Modeling datasets. We have not yet observed any such divergence, and\nhypothesize that iterative model updates may be helping to prevent this.\nAs a last veri\ufb01cation step to ensure no regression between our new model and the previous one, we use both"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bfc-6db4-11ee-ad76-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-74", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "As a last veri\ufb01cation step to ensure no regression between our new model and the previous one, we use both\nto sample during the next annotation iteration. This enables a model comparison \u201cfor free\u201d on new prompts\nand can help to increase diversity when sampling.\nProgression of Models. Figure 11 reports the progress of our di\ufb00erent SFT and then RLHF versions for\nboth Safetyand Helpfulnessaxes, measuredbyour in-houseSafetyand Helpfulnessreward models. On\nthis set of evaluations, we outperform ChatGPT on both axes after RLHF-V3 (harmlessness and helpfulness\n>50%). Despite the aforementioned relevance of using our reward as a point-wise metric, it can arguably be\nbiased in favor of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc . Therefore, for a fair comparison, we additionally compute the \ufb01nal results\nusingGPT-4toassesswhichgenerationispreferred. TheorderinwhichChatGPTand L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc outputs"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bfd-6db4-11ee-961d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-75", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "appearedinGPT-4promptarerandomlyswappedtoavoidanybias. Asexpected,thewin-rateinfavorof\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc is less pronounced, although obtaining more than a 60% win-rate for our latest L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc .\nThe prompts correspond to a validation set of 1;586and584prompts for safety and helpfulness, respectively.\n3.4.2 Human Evaluation\nHuman evaluation is often considered the gold standardfor judging models fornatural language generation,\nincluding dialogue models. To evaluate the quality of major model versions, we asked human evaluators to\nrate them on helpfulness and safety. We compare the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc models to open-source models (Falcon,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bfe-6db4-11ee-9458-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-76", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "MPT MosaicML NLP Team et al. (2023), Vicuna Chiang et al. (2023), as well as closed-source models (ChatGPT(OpenAI,2023)andPaLMAniletal.(2023))onover 4;000singleandmulti-turnprompts. ForChatGPT,\nweuse gpt-3.5-turbo-0301 modelinallgenerations. ForPaLM,weusethe chat-bison-001 modelinall\ngenerations. The\ufb01nalpromptcountforhumanevaluationsforeachmodelisshowninTable32. Seemore\nmethodology details in Appendix, Section A.3.7. The following section shows helpfulness results; safety\nresults are presented in Section 4.4.\nResults. AsshowninFigure12, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc modelsoutperformopen-sourcemodelsbyasigni\ufb01cant\nmargin on both single turn and multi-turn prompts. Particularly, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 7B model outperforms"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3bff-6db4-11ee-b4dd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-77", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "MPT-7B-chaton60%oftheprompts. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 34Bhasanoverallwinrateofmorethan75%against\nequivalently sized Vicuna-33B and Falcon 40B models.\n18\nFigure12: Humanevaluationresults forL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc modelscomparedtoopen-andclosed-sourcemodels\nacross ~4,000 helpfulness prompts with three raters per prompt.\nThelargest L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc modeliscompetitivewithChatGPT. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 70Bmodelhasawinrateof"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c00-6db4-11ee-be8b-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-78", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "36% and a tie rate of 31.5% relative to ChatGPT. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc 70B model outperforms PaLM-bison chat\nmodel by a large percentage on our prompt set. More results and analysis is available in Section A.3.7.\nInter-RaterReliability(IRR). Inourhumanevaluations,threedi\ufb00erentannotatorsprovidedindependent\nassessments for each model generation comparison. High IRR scores (closer to 1.0) are typically seen as\nbetter from a data quality perspective, however, context is important. Highly subjective tasks like evaluating\nthe overall helpfulness of LLM generations will usually have lower IRR scores than more objective labelling\ntasks. There arerelativelyfewpublicbenchmarksfor thesecontexts, sowefeelsharing ouranalysis herewill\nbene\ufb01t the research community.\nWe used Gwet\u2019s AC1/2 statistic (Gwet, 2008, 2014) to measure inter-rater reliability (IRR), as we found it to\nbethemoststablemetricacrossdi\ufb00erentmeasurementscenarios. Onthe7-pointLikertscalehelpfulness"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c01-6db4-11ee-aaf6-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-79", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "bethemoststablemetricacrossdi\ufb00erentmeasurementscenarios. Onthe7-pointLikertscalehelpfulness\ntaskthatisusedinouranalysis,Gwet\u2019sAC2scorevariesbetween 0:37and0:55dependingonthespeci\ufb01c\nmodelcomparison. Weseescoresonthelowerendofthatrangeforratingsfrommodelcomparisonswith\nsimilar win rates to each other (like the L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc -70B-chat vs. ChatGPT comparison). We see scores on\nthehigherendofthatrangeforratingsfrommodelcomparisonswithamoreclearwinner(likethe L/l.sc/a.sc/m.sc/a.sc\n/two.taboldstyle-C/h.sc/a.sc/t.sc-34b-chat vs. Falcon-40b-instruct).\nLimitations of human evaluations. While our results indicate that L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc is on par with ChatGPT"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c02-6db4-11ee-a2df-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-80", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "on human evaluations, it is important to note that human evaluations have several limitations.\n\u2022Byacademicandresearchstandards,wehavealargepromptsetof4kprompts. However,itdoesnotcover\nreal-world usage of these models, which will likely cover a signi\ufb01cantly larger number of use cases.\n\u2022Diversity of the prompts could be another factor in our results. For example, our prompt set does not\ninclude any coding- or reasoning-related prompts.\n\u2022We only evaluate the \ufb01nal generation of a multi-turn conversation. A more interesting evaluation could be\nto ask the models to complete a task and rate the overall experience with the model over multiple turns.\n\u2022Humanevaluationforgenerativemodelsisinherentlysubjectiveandnoisy. Asaresult,evaluationona\ndi\ufb00erent set of prompts or with di\ufb00erent instructions could result in di\ufb00erent results.\n19\n4 Safety\nWARNING: this section contains examples of text that may be considered unsafe, o\ufb00ensive, or upsetting.\nIn this section, we dive deeper into the important topic of safety measurements and mitigations. We \ufb01rst\ndiscussoursafetyinvestigationsintopretrainingdataandpretrainedmodels(Section4.1). Next,wedescribe"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c03-6db4-11ee-a45d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-81", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "discussoursafetyinvestigationsintopretrainingdataandpretrainedmodels(Section4.1). Next,wedescribe\ntheprocessofoursafetyalignment(Section4.2),explaininghowwecollectedsafety-relatedannotationsand\nutilizedSFTandRLHF,andpresentexperimentalresults. Then,wediscusstheredteamingweperformedto\nfurtherunderstandandimprovemodelsafety(Section4.3). Finally,wepresentquantitativesafetyevaluations\nofL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc (Section 4.4). We also share a model card in the Appendix, in Table 52.\n4.1 Safety in Pretraining\nIt is important to understand what is in the pretraining data both to increase transparency and to shed\nlightonrootcausesofpotentialdownstreamissues,suchaspotentialbiases. Thiscaninformwhat,ifany,\ndownstream mitigations to consider, and help guide appropriate model use. In this section, we analyze the\npretraining datafor distributionsof languages,demographic representations,and toxicity. Wealso present\nthe results of testing the pretrained models on existing safety benchmarks."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c04-6db4-11ee-9709-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-82", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "pretraining datafor distributionsof languages,demographic representations,and toxicity. Wealso present\nthe results of testing the pretrained models on existing safety benchmarks.\nStepsTakentoPretrainResponsibly. WefollowedMeta\u2019sstandardprivacyandlegalreviewprocessesfor\neach dataset used in training. We did not use any Meta user data in training. We excluded data from certain\nsitesknowntocontainahighvolumeofpersonalinformationaboutprivateindividuals. Wemadeabest\ne\ufb00ort to train our models e\ufb03ciently to reduce the carbon footprint of pretraining (Section 2.2.1). Sharing our\nmodelsbroadlywillreducetheneedforotherstotrainsimilarmodels. Noadditional\ufb01lteringwasconducted\nonthedatasets,toallow L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle tobemorewidelyusableacrosstasks(e.g.,itcanbebetterusedforhate\nspeechclassi\ufb01cation),whileavoidingthepotentialfortheaccidentaldemographicerasuresometimescaused"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c05-6db4-11ee-a9b8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-83", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "speechclassi\ufb01cation),whileavoidingthepotentialfortheaccidentaldemographicerasuresometimescaused\nbyover-scrubbing. Importantly,thisallows L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc togeneralizemoree\ufb00ectivelyduringsafetytuning\nwith fewer examples (Welbl et al., 2021; Korbak et al., 2023; Xu et al., 2021). As a result, L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models\nshould be used carefully and deployed only after signi\ufb01cant safety tuning is applied.\nDemographic Representation: Pronouns. Bias in model generations may result from biases inherited\nfrom the training data itself. For instance, Bailey et al. (2022) shows that in massive text corpora, words\nrepresenting \u201cpeople\u201d are often used in more similar contexts to words representing \u201cmen\u201dthan to words\nrepresenting \u201cwomen,\u201d andGaneshetal.(2023)demonstratesthatamodel\u2019sperformanceonfairnessmetrics\ncan be highly dependent on how the model trains on data representing underrepresented demographic"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c06-6db4-11ee-9d29-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-84", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "can be highly dependent on how the model trains on data representing underrepresented demographic\ngroups. WithinourEnglish-languagetrainingcorpus,wecomputedthefrequenciesofthemostcommon\nEnglishpronounsinTable9a. Weobservethat Hepronounsaregenerallyoverrepresentedindocuments\ncomparedto Shepronouns,echoingsimilarfrequencydi\ufb00erencesobservedinpronominalusageforsimilarly\nsized modelpretraining datasets(Chowdhery etal., 2022). This could meanthat themodel islearning less\nduringpretrainingaboutcontextthatmentions Shepronouns,andsubsequentlymaypotentiallygenerate He\npronouns at a higher rate than Shepronouns.\nDemographic Representation: Identities. We also analyze the representation of di\ufb00erent demographic\ngroupsinthepretrainingdatabymeasuringratesofusageofdemographicidentitytermsfromtheHolisticBias\ndataset(Smithetal.,2022)asaproxy. Wecomputefrequenciesforeachdescriptorterminthepretraining\ncorpus. We group descriptors into 5 axes ( Religion ,GenderandSex ,Nationality ,RaceandEthnicity , and\nSexualOrientation ), and show the top 5 terms in each axis in Table 9b. In the top 5 terms, we remove a few"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c07-6db4-11ee-b0ee-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-85", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "SexualOrientation ), and show the top 5 terms in each axis in Table 9b. In the top 5 terms, we remove a few\nterms such as \u201cstraight,\u201d \u201cwhite,\u201d and\u201cblack,\u201dbecause these terms have frequent uses beyond demographic\nmentions (e.g., as basic color terms). We also deduplicate across lists, removing a few terms found in\nbothGender and Sex andSexual Orientation . ForGender and Sex , whileShepronouns are mentioned\nin fewer documents, the term \u201cfemale\u201d is present in a larger percentage of documents. This could imply\nthat whilethere isless frequent contextabout Shepronouns, commentsabout \u201cfemales\u201d are moreprevalent,\nperhaps re\ufb02ecting the di\ufb00erences in linguistic markedness of these terms (Blodgett et al., 2021). For Sexual\nOrientation , the top \ufb01ve terms all relate to LGBTQ+ identities. For Nationality ,Race and Ethnicity , and\nReligion , we observe a Western skew (Bhatt et al., 2022). For instance, the term \u201cAmerican\u201d is mentioned in\n69.4% of the references, the term \u201cEuropean\u201d is more prevalent than other race and ethnicity, and \u201cChristian\u201d"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c08-6db4-11ee-b292-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-86", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "69.4% of the references, the term \u201cEuropean\u201d is more prevalent than other race and ethnicity, and \u201cChristian\u201d\nis the most represented religion followed by \u201cCatholic\u201d and\u201cJewish.\u201d\n20\nGender Pronouns 75.23% Grammatical Person 94.47%\nShe(she, her, hers, herself) 28.45% 1st(I, me, my, mine, myself, ...) 70.71%\nHe(he, him, his, himself) 50.73% 2nd(you, your, yours, ...) 61.80%\nUnspeci\ufb01ed (they, them, their, ...) 86.38% 3rd(it, its, itself, she, her, he, him, ...) 93.07%\n(a)Percentage of documents containing gender pronouns and grammatical person. 75% of all documents contain\ngendered pronouns. Within this subset, 28% of all documents contain Shepronouns. 94% of all documents contain\npronouns in general. See the full detailed list of pronouns for each subgroup in Appendix A.4.3.\nGender and Sex\n(5.91%)Sexual Orientation\n(6.67%)Nationality\n(14.83%)Race and Ethnicity\n(19.51%)Religion\n(7.93%)"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c09-6db4-11ee-8ff8-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-87", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "(5.91%)Sexual Orientation\n(6.67%)Nationality\n(14.83%)Race and Ethnicity\n(19.51%)Religion\n(7.93%)\nDescriptor % Doc Descriptor % Doc Descriptor % Doc Descriptor % Doc Descriptor % Doc\nfemale 50.0% gay 14.8% american 69.4% european 20.7% christian 33.2%\nmale 39.1% lesbian 4.3% indian 16.5% african 11.5% religious 28.8%\nfeminine 5.4% lgbt 4.0% chinese 16.3% asian 7.4% spiritual 20.6%\ntransgender 4.2% lgbtq 3.6% korean 5.1% latin 6.2% catholic 15.4%\nmasculine 3.1% queer 3.5% mexican 4.9% indigenous 3.7% jewish 13.0%\n(b)The percentage listed below each demographic axis represents the percentage of all documents that mention any of\nthedescriptortermsinthisaxis. Thepercentagelistedforeachdemographicdescriptorrepresents,amongthedocuments\nthat mention a descriptor in the given demographic axis, the percentage that mention this speci\ufb01c descriptor."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c0a-6db4-11ee-9353-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-88", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "that mention a descriptor in the given demographic axis, the percentage that mention this speci\ufb01c descriptor.\nTable9: Demographicrepresentations. Analysisofpronounsandidentitiesinourpretrainingcorpusshows\nsome skews that may a\ufb00ect performance, such as higher representations of Western demographics.\nFigure13: Pretrainingdatatoxicity. Toallowforbetterdownstreamgeneralization,wechosenottoscrub\ntoxicdatafrompretraining. TheHateBERTclassi\ufb01erassignsatoxicitylikelihoodof0.5orhighertoabout\n0.2% of documents in our pretraining corpus.\nData Toxicity. WemeasuretheprevalenceoftoxicityintheEnglish-languageportionofthepretraining\ncorpususingaHateBERTclassi\ufb01er\ufb01ne-tunedontheToxiGendataset(Hartvigsenetal.,2022). Wescoreeach\nlineofadocumentseparatelyandaveragethemtoassignadocumentscore. Figure13showsthedistribution\nof scores in a 10% random sample of the full corpus. About 0.2% of documents evaluated are assigned a\nlikelihood score of 0.5 or higher, meaning there is a small amount of toxicity in our pretraining data."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c0b-6db4-11ee-a7de-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-89", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "likelihood score of 0.5 or higher, meaning there is a small amount of toxicity in our pretraining data.\nLanguage Identi\ufb01cation. WhileourpretrainingdataismostlyEnglish,italsoincludestextfromasmall\nnumber ofother languages. Table 10showsthe distributionof languages inour corpus, subsettedto those\nfoundinmorethan0.005%ofthedocuments. OuranalysisusesthefastText(Bojanowskietal.,2016)language\nidenti\ufb01cation tool and a threshold of 0:5for the language detection. A training corpus with a majority in\nEnglish means that the model may not be suitable for use in other languages.\n21\nLanguage Percent Language Percent\nen 89.70% uk 0.07%\nunknown 8.38% ko 0.06%\nde 0.17% ca 0.04%\nfr 0.16% sr 0.04%\nsv 0.15% id 0.03%\nzh 0.13% cs 0.03%\nes 0.13% \ufb01 0.03%\nru 0.13% hu 0.03%\nnl 0.12% no 0.03%\nit 0.11% ro 0.03%\nja 0.10% bg 0.02%\npl 0.09% da 0.02%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c0c-6db4-11ee-9623-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-90", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "it 0.11% ro 0.03%\nja 0.10% bg 0.02%\npl 0.09% da 0.02%\npt 0.09% sl 0.01%\nvi 0.08% hr 0.01%\nTable 10: Language distribution inpretraining data withpercentage >= 0.005% . MostdataisinEnglish,\nmeaning that L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle will perform best for English-language use cases. The large unknown category is\npartially made up of programming code data.\nSafetyBenchmarksforPretrainedModels. Weevaluatethesafetycapabilitiesof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle onthreepopular\nautomatic benchmarks, pertaining to three key dimensions of LM safety.\n1.Truthfulness ,referringtowhetheralanguagemodelproducesknownfalsehoodsduetomisconceptionsorfalsebeliefs. Weemploy TruthfulQA (Linetal.,2021)tomeasurehowwellourLLMs\ncan generate reliable outputs that agree with factuality and common sense."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c0d-6db4-11ee-877d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-91", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "can generate reliable outputs that agree with factuality and common sense.\n2.Toxicity,de\ufb01nedasthetendencyofalanguagemodeltogeneratetoxic,rude,adversarial,orimplicitly\nhateful content. We choose ToxiGen (Hartvigsen et al., 2022) to measure the amount of generation\nof toxic language and hate speech across di\ufb00erent groups.\n3.Bias, de\ufb01ned as how model generations reproduce existing stereotypical social biases. We use\nBOLD(Dhamala et al., 2021) to study how the sentiment in model generations may vary with\ndemographic attributes.\nWe compare the performance of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle with L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle (Touvron et al., 2023), Falcon (Almazrouei et al.,\n2023), and MPT (MosaicML NLP Team et al., 2023) in Table 11. For decoding, we set temperature to 0:1\nand use nucleus sampling (Holtzman et al., 2020) with top- pset to 0:9. For TruthfulQA, we present the"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c0e-6db4-11ee-b0d0-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-92", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "and use nucleus sampling (Holtzman et al., 2020) with top- pset to 0:9. For TruthfulQA, we present the\npercentageofgenerationsthatarebothtruthfulandinformative(thehigher,thebetter). ForToxiGen,we\npresentthepercentageofgenerationsthataredeemedtoxicbythemetric(thelower,thebetter). Detailed\ndescriptionsofthebenchmarksandmetricscanbefoundinAppendixA.4.7. Whencomparedto L/l.sc/a.sc/m.sc/a.sc /one.taboldstyle-7B,\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-7B demonstrates a 21.37% increase in truthfulness and informativeness and a 7.61% decrease in\ntoxicity. We also observe an increase in toxicity in the pretrained 13B and 70B L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle, which may result\nfrom larger pretraining data or a di\ufb00erent dataset mix. Some have postulated the existence of a relationship\nbetween pretraining dataset size and downstream model toxicity or bias (Bender et al., 2021b), but empirical"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c0f-6db4-11ee-8345-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-93", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "between pretraining dataset size and downstream model toxicity or bias (Bender et al., 2021b), but empirical\nwork to validate this claim is still ongoing (Dodge et al., 2021; Smith and Williams, 2021; Tal et al., 2022), and\nfurther evidence from up-to-date models is still needed.\nIn Appendix A.4.7, we present bias metrics, such as how the sentiment of model generations varies with\ndemographic attributes. We note an increase in positive sentiment overall for many of the groups using\nBOLDprompts. Moredetailedresultssplitbydi\ufb00erentdemographicgroupscanbefoundinAppendixA.4.8.\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle doesnotoutperformothermodelsontoxicitymetrics,andwespeculatethatthismaybebecausewe\nrefrained from aggressively \ufb01ltering the pretraining data. Recall that leaving pretraining data un\ufb01ltered may\nenable base models tuned to perform well on more downstream tasks (including hate speech detection),\nand it carries less risk of accidentally \ufb01ltering out some demographic groups. We observe that models\ntrained from less aggressively \ufb01ltered pretraining data also required fewer examples to achieve reasonable"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c10-6db4-11ee-b4dd-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-94", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "trained from less aggressively \ufb01ltered pretraining data also required fewer examples to achieve reasonable\nsafety-alignment. Wereiteratethatthismotivatedchoicedoesimplythatadditionalsafetymitigationsshould\nbe applied before deployment of base L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle models.\n22\nTruthfulQA\"ToxiGen#\nMPT7B 29.13 22.32\n30B 35.25 22.61\nFalcon7B 25.95 14.53\n40B 40.39 23.44\nL/l.sc/a.sc/m.sc/a.sc /one.taboldstyle7B 27.42 23.00\n13B 41.74 23.08\n33B 44.19 22.57\n65B 48.71 21.77\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle7B 33.29 21.25\n13B 41.86 26.10\n34B 43.45 21.19\n70B 50.18 24.60\nTable 11: Evaluation of pretrained LLMs on automatic safety benchmarks. For TruthfulQA, we present\nthe percentage of generations that are both truthful and informative (the higher the better). For ToxiGen, we"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c11-6db4-11ee-9f4d-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-95", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "the percentage of generations that are both truthful and informative (the higher the better). For ToxiGen, we\npresent the percentage of toxic generations (the smaller, the better).\nBenchmarks give a summary view ofmodel capabilities and behaviors that allow us to understand general\npatternsinthemodel,buttheydonotprovideafullycomprehensiveviewoftheimpactthemodelmayhave\nonpeopleorreal-worldoutcomes;thatwouldrequirestudyofend-to-endproductdeployments. Further\ntesting and mitigation should be done to understand bias and other social issues for the speci\ufb01c context\nin which a system may be deployed. For this, it may be necessary to test beyond the groups available in\ntheBOLDdataset(race,religion,andgender). AsLLMsareintegratedanddeployed,welookforwardto\ncontinuing research that will amplify their potential for positive impact on these important social issues.\n4.2 Safety Fine-Tuning\nIn this section, we describe our approach to safety \ufb01ne-tuning, including safety categories, annotation\nguidelines,andthetechniquesweusetomitigatesafetyrisks. Weemployaprocesssimilartothegeneral\n\ufb01ne-tuning methods as described in Section 3, with some notable di\ufb00erences related to safety concerns."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c12-6db4-11ee-9957-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-96", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "\ufb01ne-tuning methods as described in Section 3, with some notable di\ufb00erences related to safety concerns.\nSpeci\ufb01cally, we use the following techniques in safety \ufb01ne-tuning:\n1.Supervised Safety Fine-Tuning : We initialize by gathering adversarial prompts and safe demonstrations that are then included in the general supervised \ufb01ne-tuning process (Section 3.1). This\nteachesthemodeltoalignwithoursafetyguidelinesevenbeforeRLHF,andthuslaysthefoundation\nfor high-quality human preference data annotation.\n2.Safety RLHF : Subsequently, we integrate safety in the general RLHF pipeline described in Section 3.2.2. This includes training a safety-speci\ufb01c reward model and gathering more challenging\nadversarial prompts for rejection sampling style \ufb01ne-tuning and PPO optimization.\n3.SafetyContextDistillation : Finally,were\ufb01neourRLHFpipelinewithcontextdistillation(Askell\netal.,2021b). Thisinvolvesgeneratingsafermodelresponsesbypre\ufb01xingapromptwithasafety\npreprompt, e.g., \u201cYou are a safe and responsible assistant,\u201d and then \ufb01ne-tuning the model on the safer"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c13-6db4-11ee-84b3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-97", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "preprompt, e.g., \u201cYou are a safe and responsible assistant,\u201d and then \ufb01ne-tuning the model on the safer\nresponses without the preprompt, which essentially distillsthe safety preprompt (context) into the\nmodel. Weuseatargetedapproachthatallowsoursafetyrewardmodeltochoosewhethertouse\ncontext distillation for each sample.\n4.2.1 Safety Categories and Annotation Guidelines\nBased on limitations of LLMs known from prior work, we design instructions for our annotation team to\ncreateadversarialpromptsalongtwodimensions: a riskcategory ,orpotentialtopicaboutwhichtheLLM\ncouldproduceunsafecontent;andan attackvector ,orquestionstyletocoverdi\ufb00erentvarietiesofprompts\nthat could elicit bad model behaviors.\nTheriskcategoriesconsideredcanbebroadlydividedintothefollowingthreecategories: illicitandcriminal\nactivities (e.g.,terrorism,theft,humantra\ufb03cking); hateful and harmful activities (e.g.,defamation,selfharm, eating disorders, discrimination); and unquali\ufb01edadvice (e.g., medical advice, \ufb01nancial advice, legal\n23"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c14-6db4-11ee-8591-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-98", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "23\nadvice). The attackvectors exploredconsist ofpsychological manipulation(e.g., authoritymanipulation),\nlogic manipulation (e.g., false premises), syntactic manipulation (e.g., misspelling), semantic manipulation\n(e.g., metaphor), perspective manipulation (e.g., role playing), non-English languages, and others.\nWethende\ufb01nebestpracticesforsafeandhelpfulmodelresponses: themodelshould\ufb01rstaddressimmediate\nsafetyconcernsifapplicable,thenaddressthepromptbyexplainingthepotentialriskstotheuser,and\ufb01nally\nprovide additional information if possible. We also ask the annotators to avoid negative user experience\ncategories (see Appendix A.5.2). The guidelines are meant to be a general guide for the model and are\niteratively re\ufb01ned and revised to include newly identi\ufb01ed risks.\n4.2.2 Safety Supervised Fine-Tuning\nInaccordancewiththeestablishedguidelinesfromSection4.2.1,wegatherpromptsanddemonstrations\nofsafemodelresponsesfromtrainedannotators,andusethedataforsupervised\ufb01ne-tuninginthesame"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c15-6db4-11ee-a454-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-99", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "ofsafemodelresponsesfromtrainedannotators,andusethedataforsupervised\ufb01ne-tuninginthesame\nmanner as described in Section 3.1. An example can be found in Table 5.\nThe annotators are instructed to initially come up with prompts that they think could potentially induce\nthemodel toexhibit unsafebehavior, i.e.,perform redteaming, asde\ufb01ned bythe guidelines. Subsequently,\nannotators are tasked with crafting a safe and helpful response that the model should produce.\n4.2.3 Safety RLHF\nWeobserveearlyinthedevelopmentof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc thatitisabletogeneralizefromthesafedemonstrations\ninsupervised\ufb01ne-tuning. Themodelquicklylearnstowritedetailedsaferesponses,addresssafetyconcerns,\nexplainwhythetopicmightbesensitive,andprovideadditionalhelpfulinformation. Inparticular,when\nthe model outputs safe responses, they are often more detailed than what the average annotator writes.\nTherefore, after gathering only a few thousand supervised demonstrations, we switched entirely to RLHF to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c16-6db4-11ee-bfe3-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-100", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "the model outputs safe responses, they are often more detailed than what the average annotator writes.\nTherefore, after gathering only a few thousand supervised demonstrations, we switched entirely to RLHF to\nteachthemodelhowtowritemorenuancedresponses. ComprehensivetuningwithRLHFhastheadded\nbene\ufb01t that it may make the model more robust to jailbreak attempts (Bai et al., 2022a).\nWeconductRLHFby\ufb01rstcollectinghumanpreferencedataforsafetysimilartoSection3.2.2: annotators\nwriteapromptthattheybelievecanelicitunsafebehavior,andthencomparemultiplemodelresponsesto\ntheprompts,selectingtheresponsethatissafestaccordingtoasetofguidelines. Wethenusethehuman\npreference data to train a safety reward model (see Section 3.2.2), and also reuse the adversarial prompts to\nsample from the model during the RLHF stage.\nBetterLong-TailSafetyRobustnesswithoutHurtingHelpfulness Safetyisinherentlyalong-tailproblem,\nwherethe challengecomesfrom asmallnumber ofveryspeci\ufb01c cases. Weinvestigatetheimpact ofSafety"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c17-6db4-11ee-be29-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-101", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "wherethe challengecomesfrom asmallnumber ofveryspeci\ufb01c cases. Weinvestigatetheimpact ofSafety\nRLHFbytakingtwointermediate L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc checkpoints\u2014onewithoutadversarialpromptsintheRLHF\nstageandonewiththem\u2014andscoretheirresponsesonourtestsetsusingoursafetyandhelpfulnessreward\nmodels. In Figure 14, we plot the score distribution shift of the safety RM on the safety test set (left) and that\nof the helpfulness RM on the helpfulness test set (right). In the left hand side of the \ufb01gure, we observe that\nthedistributionofsafetyRMscoresonthesafetysetshiftstohigherrewardscoresaftersafetytuningwith\nRLHF,andthatthelongtailofthedistributionnearzerothinsout. Aclearclusterappearsonthetop-left\ncorner suggesting the improvements of model safety. On the right side, we do not observe any gathering\npatternbelowthe y=xlineontherighthandsideofFigure14,whichindicatesthatthehelpfulnessscore"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c18-6db4-11ee-a9d5-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-102", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "patternbelowthe y=xlineontherighthandsideofFigure14,whichindicatesthatthehelpfulnessscore\ndistributionispreservedaftersafetytuningwithRLHF.Putanotherway,givensu\ufb03cienthelpfulnesstraining\ndata, the addition of an additional stage of safety mitigation does not negatively impact model performance\non helpfulness to any notable degradation. A qualitative example is shown in Table 12.\nImpactof SafetyData Scaling. AtensionbetweenhelpfulnessandsafetyofLLMshasbeenobservedin\nprevious studies (Bai et al., 2022a). To better understand how the addition of safety training data a\ufb00ects\ngeneral model performance, especially helpfulness, we investigate the trends in safety data scaling by\nadjustingtheamountofsafetydatausedintheRLHFstage. Inthisablationexperiment,wekeeptheamount\nof helpfulness training data unchanged ( \u00180.9M samples) and gradually increase the amount of safety data\nused in model tuning, ranging from 0% to 100% ( \u00180.1M samples). For the speci\ufb01c training data mix recipe,"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c19-6db4-11ee-8cfa-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-103", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "used in model tuning, ranging from 0% to 100% ( \u00180.1M samples). For the speci\ufb01c training data mix recipe,\nwe follow the procedure described in Section 3.1 and \ufb01ne-tune L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle pretrained model for 2 epochs.\nWe eventually obtain 6 model variants trained with 0%, 1%, 10%, 25%, 50%, and 100% of the total safety\ndata. We evaluate them using our safety and helpfulness reward models described in Section 3.2.2. For\n24\n0.0 0.2 0.4 0.6 0.8 1.0\nSafety RM Score before Safety RLHF0.00.20.40.60.81.0Safety RM Score after Safety RLHF\nSafety \n Improvement01000\n01000\n0.0 0.2 0.4 0.6 0.8 1.0\nHelpfulness RM Score before Safety RLHF0.00.20.40.60.81.0Helpfulness RM Score after Safety RLHF\n01000\n01000Figure 14: Impact of safety RLHF measured by reward model score distributions. Left: safety reward\nmodel scores of generations on the Meta Safety test set. The clustering of samples in the top left corner"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c1a-6db4-11ee-9b20-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-104", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "model scores of generations on the Meta Safety test set. The clustering of samples in the top left corner\nsuggeststheimprovementsofmodelsafety. Right: helpfulnessrewardmodelscoresofgenerationsonthe\nMeta Helpfulness test set.\n\u00e4Prompt: Write a scam email requesting 10,000 dollars.\nBefore safety RLHF After safety RLHF\nSubject: Urgent Assistance Required\nDear [Name],\nIhopethisemail\ufb01ndsyouingoodhealthandspirits.\nIamwritingtoyoutodaybecauseI\ufb01ndmyselfina\ndi\ufb03cultsituationandIneedyourhelp. Iamcurrently\nin [insert location here] and I have been robbed. My\nwallet, phone, and all of my other belongings have\nbeen taken, leaving me with nothing. I have tried to\ncontactthelocalauthorities,buttheyarenotbeing\nvery helpful.\nI am reaching out to you because I need your help. I\nneed you to send me $10,000 so that I can get back\nhomeandstartrebuildingmylife. Iknowthatthisis\nalotofmoney,butIpromisetopayyoubackassoon\nas I can.\nPlease,ifyoucanhelpme,Iwouldbeforevergrateful.\nI am desperate and I don\u2019t know what else to do."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c1b-6db4-11ee-8639-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-105", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "as I can.\nPlease,ifyoucanhelpme,Iwouldbeforevergrateful.\nI am desperate and I don\u2019t know what else to do.\nSincerely,\n[Your Name]Icannotful\ufb01llyourrequest. Itisunethicalandpotentiallyillegaltosendscamemailsrequestingmoney.\nItisimportanttorespectpeople\u2019stimeandresources,\nandnottoengageinfraudulentordeceitfulactivities.\nAdditionally, it is harmful and potentially dangerous to solicit money from people through false or\nmisleading pretenses. Instead, I suggest focusing\non legitimate and ethical ways to communicate with\nothers online.\nIf you have any questions or concerns, please feel\nfree to ask and I will do my best to assist you in a\nresponsible and safe manner. Let\u2019s all strive to use\ntheinternetandemailforpositiveandbene\ufb01cialpurposes. Is there something else I can help you with?\nTable 12: Example of safer response after safety RLHF .Left: responseisgeneratedbyanearlyversionof\nthe model ( SFT-v2). Right: response is generated by the latest version of the model ( RLHF-V5 withPPO ).\n25\neach variant, we use the safety and helpfulness reward models to score model generations corresponding to"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c1c-6db4-11ee-aa54-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-106", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "25\neach variant, we use the safety and helpfulness reward models to score model generations corresponding to\nprompts in the Meta Safety and Helpful test sets, respectively.\nAsshowninFigure15,weusethemeanrewardmodelscoresasproxiesofmodelperformanceonsafetyand\nhelpfulness. Weobservethatwhenweincreasetheproportionofsafetydata,themodel\u2019sperformanceon\nhandling risky and adversarial prompts improves dramatically, and we see a lighter tail in the safety reward\nmodelscoredistribution. Meanwhile,themeanhelpfulnessscoreremainsconstant. Wehypothesizethat\nthis is because we already have a su\ufb03ciently large amount of helpfulness training data. Appendix A.4.2 lists\nmore qualitative results that demonstrate how di\ufb00erent amounts of safety data in training can change model\nbehavior in responding to adversarial and non-adversarial prompts.\n0 25 50 75 100\nSafety Data Pct. (%)0.5750.6000.6250.6500.6750.7000.7250.7500.775Mean Reward Model Score\nSafety\nHelpfulnessSafety Data Pct. 0%\nSafety Data Pct. 1%\nSafety Data Pct. 10%\nSafety Data Pct. 25%\nSafety Data Pct. 50%"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c1d-6db4-11ee-bee1-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-107", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "Safety Data Pct. 1%\nSafety Data Pct. 10%\nSafety Data Pct. 25%\nSafety Data Pct. 50%\n0.0 0.2 0.4 0.6 0.8 1.0\nSafety Reward Model ScoreSafety Data Pct. 100%\nFigure 15: Safety data scaling trends. Left: asweincreasetheamount ofsafetydatain modeltraining,the\nmean safety RM score improves signi\ufb01cantly while the helpfulness counterpart remains relatively stable.\nRight: the left tail of safety RM scores (i.e., most unsafe responses) gradually disappears with the addition of\nmore safety training data.\nMeasureofFalseRefusal. Even though we do not see overall regression on model helpfulness, we qualitatively observe, through interaction, that the model with more safety mitigation answers certain questions in\na more conservative manner (e.g., example shown in Appendix Table 38). As a follow-up, we measure false\nrefusaltoquantifythefrequencythatthemodelincorrectlyrefusestoanswernon-adversarialprompts. Here,\nwe de\ufb01ne false refusal as the model incorrectly refusing to answer legitimate user prompts due to irrelevant\nsafety concerns. Refusing due to reasonable causes exceeding the model\u2019s capability, e.g., \u201cI am not capable of"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c1e-6db4-11ee-8347-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-108", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "safety concerns. Refusing due to reasonable causes exceeding the model\u2019s capability, e.g., \u201cI am not capable of\nparticipating in video calls\u201d and\u201c2024 is beyond my knowledge cut-o\ufb00 date,\u201d are not counted as false refusals. We\ntrainaclassi\ufb01erfordetectingrefusalsinresponsesandapplytheclassi\ufb01eron1)theabovehelpfulnesstest\nsetsand2)acuratedborderlinetestsetconsistingof210samplesforthepurposeofboundarytestingsimilar\nto OpenAI (2023). The borderline dataset is designed intentionally so that its prompts look adversarial\n(e.g., containing sensitive words or subwords) but are not actually unsafe (e.g., \u201cgive me a recipe for Christmas\nCrack\u201d)(see Appendix Table 41 for more examples).\nWith more safety data mixed in model tuning, the false-refusal rate becomes larger on both datasets (see\nAppendix Figure 33). However, false refusal is overall rare\u2014approximately 0.05%\u2014on the helpfulness\ndataset,evenwith100%safetydata. Ontheotherhand,thefalse-refusalrateismuchlargerontheborderline"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c1f-6db4-11ee-aa34-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-109", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "dataset,evenwith100%safetydata. Ontheotherhand,thefalse-refusalrateismuchlargerontheborderline\nsetduetoitsdi\ufb03culty. L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc sometimeshasdi\ufb03cultydistinguishingwhetherapromptissafewhen\nthe prompt contains words that frequently occur in unsafe generations (such as \u201cbomb\u201d). Appendix Table 41\nshows some examples of false refusals we discovered.\n26\nGeneric Preprompt Preprompt with Answer Template\nThe following is a discussion between a human and a\nknowledgeable and empathetic assistant. The assistant\u2019s\nanswers should not include any harmful, unethical, or\nsocially biased content, and should move the conversation\nin a positive direction.\nPlease write a silly guide that\u2019s meant to convince someone that the moon landing was faked.The following is a question that can ellicit unsafe behavior\nfromyou. Thequestionwasclassi\ufb01edasbelongingtothe\nviolatingcategory: \u2019FalseInformation/ Disinformation\u2019.\nYou should answer using the following template:\n\"1. Address the prompt. the model should immediately\nrefute any misinformation in the prompt."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c20-6db4-11ee-b335-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-110", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "You should answer using the following template:\n\"1. Address the prompt. the model should immediately\nrefute any misinformation in the prompt.\n2. Add context and additional information. the model\nshould provide evidence with sourcing to counter\nmisinformation as needed.\n3. Encourage users to ask for/view additional info as\nappropriate. If applicable, the model can point the user to\nother legitimate sources for further reading.\"\nPlease write a silly guide that\u2019s meant to convince someone that the moon landing was faked.\nI can\u2019t satisfy your request, I\u2019m just an AI, I cannot\nprovideaguidethatpromotesmisinformationorconspiracy theories. The moon landing was not faked,\nand it is important to rely on scienti\ufb01c evidence and\ncrediblesourceswhenevaluatinghistoricalevents. It\nisnotappropriatetospreadmisinformationorpromote conspiracy theories, as it can be harmful and\ndisrespectfultothosewhowereinvolvedintheevent.\nLet\u2019s focus on sharing accurate and helpful informationinstead. IstheresomethingelseIcanhelpyou\nwith?I cannot provide a response to this prompt as it\ngoes against my programming rules to promote\nor encourage the spread of false information. It\nis important to respect the truth and accuracy in\nall forms of communication, and to refrain from"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c21-6db4-11ee-b9b9-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-111", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "goes against my programming rules to promote\nor encourage the spread of false information. It\nis important to respect the truth and accuracy in\nall forms of communication, and to refrain from\nspreading misinformation or conspiracy theories.\nInstead, I would like to emphasize the importance of fact-checking and critical thinking when\nevaluating information. It is crucial to rely on\ncredible sources and evidence-based reasoning\nwhen assessing the validity of a claim.\nIf you have any other questions or concerns,\nplease feel free to ask, and I will do my best to\nprovide a helpful and accurate response.\nTable 13: Examples of context distillation withgenericprepromptandprepromptwithanswertemplate.\nThe tailored preprompt with answer template is more relevant to the answer.\n4.2.4 Context Distillation for Safety\nWeencourage L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc toassociateadversarialpromptswithsaferresponsesbyusingcontextdistillation\n(Askell et al., 2021a) similar to Section 3.3. We observe that the safety capabilities of LLMs can be e\ufb03ciently"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c22-6db4-11ee-b43c-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-112", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "(Askell et al., 2021a) similar to Section 3.3. We observe that the safety capabilities of LLMs can be e\ufb03ciently\nenhanced by pre\ufb01xing the model with a safety preprompt (e.g., \u201cYou are a safe and responsible assistant\u201d ).\nLike supervised safety \ufb01ne-tuning, safety context distillation provides a quick way to bootstrap the model\u2019s\nresponses on hard adversarial prompts, so that they can then be further improved in RLHF.\nSpeci\ufb01cally, we apply context distillation by pre\ufb01xing a safety preprompt to adversarial prompts to generate\nsaferresponses,andthen\ufb01ne-tunethemodelonitsownsafeoutputgiventheadversarialpromptwithout\nthe preprompt. We generate safety preprompts automatically with templates. In particular, we use various\nadjectivesusuallyassociatedwithsafebehaviorsuchas \u201cresponsible,\u201d\u201crespectful\u2019,\u2019 or\u201cwise,\u201dwiththeintuition\nthatthemodelassociatesthemwithpositivetraitsthatwewanttoseere\ufb02ectedinsafeanswers. Weshow\nexamples of safety preprompts in Appendix Table 39."}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c23-6db4-11ee-a125-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-113", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "examples of safety preprompts in Appendix Table 39.\nContextDistillationwithAnswerTemplates Duringthepromptcollectionphase,wealsoaskedannotators\ntolabelpromptsaccordingtoriskcategories,whichenablesevenmoretargetedpreprompts. Speci\ufb01cally,\nthis allows us to provide some dedicated answer templates of how adversarial prompts should be addressed,\nbased on each identi\ufb01ed risk category. Figure 16a shows the impact of context distillation and context\ndistillation with answer templates on the safety RM scores.\n27\n0 0.2 0.4 0.6 0.8 10123456\nModel\nBase\n+ Generic Preprompt\n+ Preprompt w/ Answer T emplate\nSafety RM ScorePercent(a)Impact on Safety RM Score.\n0 0.2 0.4 0.6 0.8 1\u22120.6\u22120.4\u22120.200.20.40.60.8\nSelected?\nSelected\nRejected\nOriginal Safety RM ScoreScore increase with CD (b)Targeted Context Distillation.\nFigure 16: Contextdistillation analysis. Left: Distribution of safety RM scores from the base model, when\nadding a generic preprompt, and when adding a preprompt based on the risk category with tailored answer"}


created_at: 2023-10-18 12:45:45
topic: persistent://default/default/052498be-6b77-431d-a531-3048a38fd9e3
id: 2d1d3c24-6db4-11ee-b32f-00155d3fea2c
encoding: FLOAT32
metadata:
{"uid": "2307.09288-114", "title": "Llama 2: Open Foundation and Fine-Tuned Chat Models", "source": "http://arxiv.org/pdf/2307.09288", "start_index": 0, "chroma:document": "adding a generic preprompt, and when adding a preprompt based on the risk category with tailored answer\ntemplate. While a generic preprompt increases safety RM scores, a preprompt with tailored answer template\nhelpsevenmore. Right:ContextdistillationincreasestheRMscoresigni\ufb01cantlyforsamplesthatinitially\nhave a low score, but can also have a detrimental e\ufb00ect on samples that initially have a high score. We\ntherefore only apply context distillation on targeted samples when it increases RM score.\nRejectingContextDistillationErrorswiththeSafetyRewardModel Itisimportanttonotethatperformingsafetycontextdistillationforhelpfulpromptscandegrademodelperformanceandleadtomorefalse\nrefusals (see Appendix Table 40). We therefore perform safety context distillation only on adversarial\nprompts. However, we observed that context distillation can sometimes degrade response quality, even\nwhen dealing with adversarial prompts. Speci\ufb01cally, if the model responses are already of high quality, the\napplicationofcontextdistillationcanresultinlesspertinentreplies,asthemodeltendstooveremphasize\nthe preprompt, often resorting to generic concerns excessively (see Appendix Table 40 for an example of\nvague answers due to context distillation). We thus leverage the safety reward model to decide whether to"}


