\section{Conclusions and Future Work}\label{section:conclusions}
In this paper, we present \emph{GraphVid}, a graph video representations that enable video-processing via graph neural networks. Furthermore, we propose a relational graph convolutional model that suits this representation. Our experimental study demonstrates this model's efficiency in performing video-related tasks while achieving comparable performance to the current state-of-the-art. An interesting avenue for future work is to explore new graph representations of videos, including learnable methods. Additionally, we consider the development of new dedicated graph neural models for processing the unique and dynamic structure of the video-graph as an interesting research direction. Finally, unified models for image and video understanding that disregard temporal edges could be explored in order to take advantage of the amount of data in both worlds.
\section{Experiments}\label{section:experiments}
We validated \emph{GraphVid}\ on 2 human-action-classification benchmarks. The goal of human action classification is to determine the human-involved action that occurs within a video.
The objectives of this empirical study were twofold:
\begin{itemize}
    \item Analyze the impact of the various parameters on the accuracy of the model.
    \item As we first and foremost target efficiency, we sought to examine the resources' consumption of \emph{GraphVid}\ in terms of Floating Point Operations
    (FLOPs). We followed the conventional protocol \cite{feichtenhofer2020x3d}, which uses single-clip FLOPs as a basic unit of computational cost. We show that we are able to achieve a significant improvement in efficiency over previous methods while preserving state-of-the-art performance.

\end{itemize}
\subsection{Setup}

\paragraph{Datasets -}
We use two common datasets for action classification: \textit{Kinetics-400 (K400)} \cite{kay2017kinetics} and \textit{Charades} \cite{sigurdsson2016hollywood}. Kinetics-400 \cite{kay2017kinetics} is a large-scale video dataset released in 2017 that contains 400 classes, with each category consisting of more than 400 videos. It originally had, in total, around 240K, 19K, and 38K videos for training, validation and testing subsets, respectively. Kinetics is gradually shrinking over time due to videos being taken offline, making it difficult to compare against less recent works. We used a dataset containing 208K, 17K and 33K videos for training, validation and test respectively. We report on the most recently available videos. Each video lasts approximately 10 seconds. The Charades dataset \cite{sigurdsson2016hollywood} is composed of 9,848 videos of daily indoor activities, each of an average length of 30 seconds. In total, the dataset contains 66,500 temporal annotations for 157 action classes. In the standard split, there are 7,986 training videos and 1,863 validation videos, sampled at 12 frames per second. We follow prior arts by reporting the Top-1 and Top-5 recognition accuracy for Kinetics-400 and mean average precision (mAP) for Charades.

\begin{figure}[t]
    \centering
    \includegraphics[width=0.65\linewidth]{figures/general_arch.png}
    \caption{The general graph neural network architecture we use in our experiments.}
    \label{fig:general_arch}
\end{figure}

\paragraph{Network Architecture and Training -}
We use GNN variants and feed each of them with our video-graphs. Specifically, we consider Graph Convolutional Networks \cite{kipf2016semi} (GCNs), Graph Attention Networks \cite{velivckovic2017graph} (GATs) and Relational Graph Convolutional Networks \cite{schlichtkrull2018modeling} (RGCNs). The general architecture of our backbones is depicted in Fig. \ref{fig:general_arch}. It consists of $2$ fully-connected (FC) layers with exponential linear unit (ELU) activations that project the node features into a $256D$ feature space. Then come $4$ layers of the corresponding GNN layer (either GCN, GAT or RGCN along with an edge feature concatenation from Eq. \ref{eq:concat_edges}) with a hidden size of 512 with ELU activations, followed by global mean pooling, dropout with a probability of $0.2$ and a linear layer whose output is the predicted logits. For the GAT layers, we use 4 attention heads in each layer, and average the attention heads' results to obtain the desired hidden layer size. For the RGCN layers, we specify 2 relations, which correspond to the spatial and temporal relations, as described in Section \ref{section:methodology}. We use the Adam \cite{kingma2014adam} with a constant learning rate of \text{$1e-3$} for optimization. While choosing this architecture, the core idea was to keep the architecture simple and shallow, while changing the interaction module to better model the relations between parts of the clip.
We divide the videos into clips using a sliding window of 20 frames, using a stride of 2 between consecutive frames and a stride of 10 between clips. In all the experiments, we used a fixed batch size of 200.

\paragraph{Inference -}
At the test phase, we use the same sliding window methodology as in the training. We follow the common practice of processing multiple views of a long video and average per-view logits to obtain the final results. The views are drawn uniformly across the temporal dimension of the video, without spatial cropping. The number of views is determined by the validation dataset.

\paragraph{Implementation Details -} All experiments were run on a Ubuntu 18.04 machine with Intel i9-10920X, 93GB RAM and 2 GeForce RTX 3090 GPUs. Our implementation of \emph{GraphVid}\ is in Python3. To generate superpixels, we use \textit{fast-slic} \cite{fastslic} with the AVX2 instruction set. To train the graph neural models, we use Pytorch-Geometric \cite{fey2019fast}.
We use a fixed seed for SLIC and cache the generated graphs during the first training epochs in order to further reduce the computations. We also store the edge indexes as int16 instead of int64 in order to reduce the memory footprint. Eventually, the memory footprints of the cached datasets is comparable to those of the original ones.

\subsection{Ablation Study}\label{section:ablation}
We conduct an in-depth study on Kinetics-400 to analyze the performance gain contributed by incorporating the different components of \emph{GraphVid}.
\paragraph{Graph Neural Network Variants and Number of Superpixels per Frame -}
We assess the performance of different GNN variants: GCN \cite{kipf2016semi} is trained without edge relations (\emph{i.e.}} \def\Ie{\emph{I.e.}\, temporal and spatial edges are treated via the same weights). GAT \cite{velivckovic2017graph} is trained by employing the attention mechanism for neighborhood aggregation without edge relations. RGCN \cite{schlichtkrull2018modeling} is trained with edge relations, as described in Section \ref{section:model_arch}.

The results of the action classification on K-400 are shown in Figure \ref{fig:n_sp_and_model_variants_ablation}. In this series, the number of views is fixed at $8$, which is the number of views that was found to be most effective for the validation set. For all variants, increasing the number of superpixels per frame ($S$) contributes to the accuracy. We notice a significant improvement in accuracy for the lower range of the number of superpixels, while the accuracy begins to saturate for \text{$S\geq 650$}. Increasing further the number of superpixels leads to bigger inputs, which require more computations. As our goal is to maximize the efficiency, we do not experiment with larger inputs in this section.
\input{graphs/ablation/model_sp_grid}
We further present in Table \ref{table:models_ablation} the models' specifications for $800$ superpixels, which is the best-performing configuration in this series of experiments. Unsurprisingly, the GCN variant requires the least amount of computations. Meanwhile, the RGCN variant requires fewer computations than GAT and achieves a higher level of accuracy. We conclude that it is beneficial to incorporate edge relations when wishing to encode temporal and spatial relations in videos, and that those features are not easily learned by heavy computational models, such as GAT.

\input{tables/models}

\paragraph{Augmentations -}
\input{graphs/ablation/augmentations_grid}
We assessed the impact of augmentations on the performance and their ability to alleviate over-fitting. For this purpose, we chose the best configuration obtained from the previous experiments, that is, RGCN with 800 superpixels per frame, and trained it while adding one augmentation at a time. The results of this series are depicted in Figure \ref{fig:augmentations_grid}. Each graph shows the level of accuracy reached by training the model with one of the parameters that control the augmentation.

We begin with the analysis of the AGEN and AGNN, both relate to the addition of Gaussian noise to the graph components, with the corresponding parameters controlling the standard deviations. Their impact is unnoticeable as these parameters head towards $0$, since lower values reflect the scenarios in which little or no augmentations are applied. Slightly increasing the parameter brings about a gradual improvement in the accuracy, until a turning point is reached, after which the level of accuracy declines until it reaches \text{$\sim \frac{1}{400}$}, which resembles a random classifier. The decrease in accuracy stems from the noise obscuring the original signal, allegedly forcing the classifier to classify ungeneralizable noise.
For RRSE and RRS, the random removal of spatial edges harms the accuracy of the model. This finding leads us to conclude that spatial edges encode meaningful information about relations between the entities. Moreover, slightly removing the nodes contributes to the level of accuracy, reaching a peak at \text{$p_{node}\approx 0.8$}. To conclude, we present the values that lead to the best Top-1 accuracy score in Table \ref{table:augmentations_params}.
\input{tables/aug_params}
\subsection{Comparison to the State-of-the-Art}
\input{graphs/bubbles_grid}
\paragraph{Kinetics-400 -}
We present the K-400 results for our RGCN variant in Table \ref{table:k400_sota} and Figure \ref{fig:k400_relative_sota}, along with comparisons to previous arts, including convolutional-based and transformer-based methods. Our results are denoted RGCN-$d$, where $d$ represents the number of superpixels. Additionally, we use the set of augmentations with the parameters from Table \ref{table:augmentations_params}. First, when the RGCN-800 model is trained with the full set of augmentations (denoted Full-Aug), it achieves a significantly higher Top-1 accuracy than when it is trained without any augmentation (denoted No-Aug) or when each augmentation is applied individually. These results demonstrate the effectiveness of our model and that our augmentations can alleviate overfitting and improve the generalization over the test set. Second, all our RGCNs require orders-of-magnitude fewer computations than the previous arts, as well as more than \text{$\times 10$} fewer parameters.
\input{tables/sota_comparison/k400}
\paragraph{Charades -}
We train RGCN variants with $800$ and $2000$ superpixels with the set of augmentations found in Table \ref{table:augmentations_params}. We also follow prior arts \cite{feichtenhofer2019slowfast,fan2021multiscale} by pre-training on K-400 followed by replacing the last FC layer and fine-tuning on Charades. Table \ref{table:charades_sota} and Figure \ref{fig:charades_relative_sota} show that when our RGCN model is trained with 2000 superpixels, its mAP score is comparable to the current state-of-the-art, but this score is reached with orders-of-magnitude fewer computations and using considerably fewer parameters.
\input{tables/sota_comparison/charades}

\subsection{Video-Graph Generation Run-Time}
\begin{wrapfigure}[15]{r}{0.5\linewidth}
\begin{center}
\input{graphs/samples_generation}
\end{center}
\caption{Time of generation depending on the number of superpixels.}
\label{fig:graph_run_time}
\end{wrapfigure}
The transition into a video-graph representation requires the consideration of the time needed for generating it. In Figure \ref{fig:graph_run_time}, we measured the average time needed using our setup, which include the whole pipeline: \textbf{1.} Superpixels calculation, and \textbf{2.} Graph structure generation, that is, creating edges between adjacent super-pixels and features calculation as described in Section \ref{section:methodology}. Interestingly, the first step is relatively short compared to the second. Apparently, the optimized \textit{fast-slic} \cite{fastslic} performs well, while the search for adjacent superpixels is time consuming. This opens the possibilities of further optimization.

\section{Introduction}\label{section:introduction}
The field of video understanding has gained prominence thanks to the rising popularity of videos, which has become the most common form of data on the web. On each new uploaded video, a variety of tasks can be performed, such as tagging \cite{fernandez2017vits}, human action recognition \cite{pareek2021survey}, anomaly detection \cite{suarez2020survey}, etc. New video-processing algorithms are continuously being developed to automatically organize the web through the flawless accomplishment of the aforementioned tasks.

Nowadays, Deep Neural Networks are the de-facto standard for video understanding \cite{oprea2020review}. However, with every addition of a new element to the training set (that is, a full training video), more resources are required in order to satisfy the enormous computational needs.
On the one hand, the exponential increment in the amount of data raises concerns regarding our ability to handle it in the future. On the other hand, it has also spurred an highly creative research field aimed at finding ways to mitigate this burden.

Among the first-generation of video processing methods were ones geared toward adopting 2D convolution neural networks (CNNs), due to their computational efficiency \cite{simonyan2014two}. Others decomposed 3D convolutions \cite{du2017closer,xie2018rethinking} into simpler operators, or split a complex neural network into an ensemble of
lightweight networks \cite{chen2018multi}. However, video understanding has greatly evolved since then, with the current state-of-the-art methods featuring costly attention mechanisms \cite{arnab2021vivit,girdhar2019video,liu2021video,akbari2021vatt,fan2021multiscale,bertasius2021space,li2021vidtr}. Beyond accuracy, a prominent advantage of the latest generation of methods is that they process raw data, that is, video frames that do not undergo any advanced pre-processing. Meanwhile, pursuing new video representations and incorporating pre-computed features to accelerate training is a promising direction that requires more extensive research.
\newcommand{0.2}{0.2}
\newcommand{1.2in}{1.2in}
\begin{figure}[ht]
    \centering
    \begin{subfigure}[b]{0.4\linewidth}
         \centering
         \includegraphics[width=0.4\linewidth]{figures/guitarist.jpg}
        \caption{Original image}
         \label{fig:original_intro}
     \end{subfigure}
    \begin{subfigure}[b]{0.4\linewidth}
         \centering
         \includegraphics[width=0.4\linewidth]{figures/example_superpixels.png}
        \caption{Mean superpixels}
         \label{fig:superpixels_intro}
     \end{subfigure}
    \caption{A visual comparison between a pixel and a mean-superpixel representation. On the left, the original image is presented. On the right, we present the image formed by generating superpixel regions using SLIC and filling each region with its mean color.}
    \label{fig:superpixels_example}
\end{figure}

Prior to the renaissance of deep learning \cite{lecun2015deep}, much research was done on visual feature generation. Two prominent visual feature generation methods are superpixels\footnote{Superpixel techniques segment an image into regions by considering similarity measures, defined using perceptual features.} and optic-flow\footnote{Optic-flow is the pattern of the apparent motion of an object(s) in the image between two consecutive frames due to the movement of the object or the camera.}. These techniques' ability to encode perceptually meaningful features has greatly contributed to the success of computer vision algorithms. Superpixels provide a convenient, compact representation of images that can be very useful for computationally demanding problems, while optic-flow provides hints about motion. We rely on these methods to construct a novel representation of videos that encodes sufficient information for video understanding: 1) adjacent pixels are grouped together in the form of superpixels, and 2) temporal relations and proximities are expressed via graph connectivity. The example depicted in Figure \ref{fig:superpixels_example} provides an intuition for the sufficiency of superpixel representation for scene understanding. It contains the superpixel regions obtained via SLIC \cite{achanta2010slic}, with each region filled with the mean color. One can clearly discern a person playing a guitar in both images. A different way of depicting the relations between superpixels is a graph with nodes representing superpixels \cite{monti2017geometric,dadsetan2021superpixels,avelar2020superpixel}. Such a representation has the advantage of being invariant to rotations and flips, which obviates the need for further augmentations. We here demonstrate how this representation can reduce the computational requirements for processing videos.

Recent years have seen a surge in the utilization of Graph Neural Networks (GNNs) \cite{kipf2016semi} in tasks that involve images \cite{monti2017geometric,dadsetan2021superpixels,avelar2020superpixel}, audio \cite{dokania2019graph,zhang2019few} and other data forms \cite{wang2018videos,xie2016representation,abadal2021computing}. In this paper, we propose \emph{GraphVid}, a concise graph representation of videos that enables video processing via GNNs. \emph{GraphVid}\ constructs a graph representation of videos that is subsequently processed via a GCN to predict a target. We intend to exploit the power of graphs for efficient video processing. To the best of our knowledge, we are the first to utilize a graph-based representation of videos for efficiency. \emph{GraphVid}\ dramatically reduces the memory footprint of a model, enabling large batch-sizes that translate to better generalization. Moreover, it utilizes models with an order-of-magnitude fewer parameters than the current state-of-the-art models while preserving the predictive power. \textbf{In summary, our contributions are:}
\begin{enumerate}
    \item We present \emph{GraphVid}\ - a simple and intuitive, yet sufficient representation of video clips. This simplicity is crucial for delivering efficiency.
    \item We propose a dedicated GNN for processing the proposed representation. The proposed architecture is compared with conventional GNN models in order to demonstrate the importance of each component of \emph{GraphVid}.
    \item We present 4 types of new augmentations that are directly applied to the video-graph representation. A thorough ablation study of their configurations is preformed in order to demonstrate the contribution of each.
    \item We perform a thorough experimental study, and show that \emph{GraphVid}\ greatly outperforms previous methods in terms of efficiency - first and foremost, the paper utilizes GNNs for efficient video understanding. We show that it successfully
reduces computations while preserving much of the performance of state-of-the-art approaches that utilize computationally demanding models.
\end{enumerate}
\section{\emph{GraphVid}\ - A Video-Graph Representation}\label{section:methodology}
In this section, we introduce the methodology of \emph{GraphVid}. First, we present our method for video-graph representation generation, depicted in Figure \ref{fig:framework} and described in Algorithm \ref{algo:graphvid}. Then, we present our training methodology that utilizes this representation. Finally, we discuss the benefits of \emph{GraphVid}\ and propose several augmentations.

\input{figures/framework}

\subsection{Overview}
In our framework, we deal with video clips that are sequences of $T$ video frames \text{$v\in \mathbb{R}^{T\times~C\times~H\times~W}$}. The goal is to transform $v$ into a graph that is sufficiently informative for further processing. To achieve this, we use SLIC \cite{achanta2010slic} to generate $S$ segmented regions, called \textit{superpixels}, over each frame. We denote each segmented region as $R_{t,i}$, where \text{$t\in [T]$} represents the temporal frame index, and \text{$i\in [S]$} the superpixel-segmented region index. The following is a description of how we utilize the superpixels to construct our video-graph representation.

\paragraph{Graph Elements -}
We define the undirected graph $\mathcal{G}$ as a 3-tuple \text{$\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{R})$}, where \text{$\mathcal{V}=\{R_{t,i} | t\in [T], i\in [S]\}$} is the set of nodes representing the segmented regions, $\mathcal{E}$ is the set of labeled edges (to be defined hereunder) and \text{$\mathcal{R}=\{spatial,temporal\}$} is a set of relations as defined in \cite{schlichtkrull2018modeling}. Each node $R_{t,i}$ is associated with an attribute $R_{t,i}.c\in \mathbb{R}^3$ representing the mean RGB color in that segmented region. Additionally, we refer to $R_{t,i}.y$ and $R_{t,i}.x$ as the coordinates of the superpixel's centroid, which we use to compute the distances between superpixels. These distances, which will later serve as the edge attributes of the graph, are computed by
\begin{equation}
    d^{t_q\to t_p}_{i,j} = \sqrt{\left(\frac{R_{t_q,i}.y - R_{t_p,j}.y}{H}\right)^2 + \left(\frac{R_{t_q,i}.x - R_{t_p,j}.x}{W}\right)^2}.
\end{equation}
Here, \text{$t_q,t_p\in [T]$} denote frame indices, and \text{$i,j\in [S]$} denote superpixel indices generated for the corresponding frames.
The set of edges $\mathcal{E}$ is composed of: \textbf{1)} intra-frame edges (denoted $\mathcal{E}^{spatial}$) - edges between nodes corresponding to superpixels in the same frame. We refer to these as \textit{spatial edges}. \textbf{2)} inter-frame edges (denoted $\mathcal{E}^{temporal}$) - edges between nodes corresponding to superpixels in two sequential frames. We refer to edges as \textit{temporal edges}.
Finally, the full set of edges is \text{$\mathcal{E} = \mathcal{E}^{spatial} \cup \mathcal{E}^{temporal}$}.
Following is a description of how we construct both components.

\paragraph{Spatial Edges -}
In similar to \cite{avelar2020superpixel}, we generate a region-adjacency graph for each frame, with edge attributes describing the distances between superpixel centroids. The notation \text{$\mathcal{E}^{spatial}_t$} refers to the set of the spatial-edges connecting nodes corresponding to superpixels in the frame $t$,
and
\(
    \mathcal{E}^{spatial} = \bigcup_{t=1}^{T}{\mathcal{E}^{spatial}_t}.
\)
Each edge \text{$e_{i,j}^{t}\in \mathcal{E}^{spatial}$} is associated with an attribute that describes the euclidean distance between the two superpixel centroids $i$ and $j$ in frame $t$, that is, $d^{t\to t}_{i,j}$.
These distances provide information about the relations between the superpixels. Additionally, the distances are invariant to rotations and image-flips, which eliminates the need for those augmentations. Note that normalization of the superpixels' centroid coordinates is required in order to obscure information regarding the resolution of frames, which is irrelevant for many tasks, such as action classification. In Figure \ref{fig:spatial_edges}, we demonstrate the procedure of spatial edge generation for a cropped image that results in a partial graph of the whole image. Each superpixel is associated with a node, which is connected via edges to other adjacent nodes (with the distances between the superpixels' centroids serving as edge attributes).
\begin{figure}[!ht]
    \centering
    \includegraphics[width=0.45\linewidth]{figures/spatial_graph.png}
    \caption{Spatial edge generation. First, superpixels are generated. Each superpixel is represented as a node, which is connected via its edges to other such nodes within a frame. Each node is assigned the mean color of the respective segmented region, and each edge is assigned the distances between the superpixel centroids connected by that edge.}
    \label{fig:spatial_edges}
\end{figure}

\paragraph{Temporal Edges -}
In modeling the temporal relations, we aim to connect nodes that tend to describe the same objects in subsequent frames. To do so, we rely on the assumption that in subsequent frames, such superpixels are attributed similar colors and the same spatial proximity. To achieve this, for each superpixel $R_{t,i}$, we construct a neighborhood $\mathcal{N}_{t,i}$ that contains superpixels from its subsequent frame whose centroids have a proximity of at most $d_{proximity}\in (0,1]$ with respect to the euclidean distance. Then, we find the superpixel with the most similar color in this neighborhood. As a result, the $t^{th}$ frame is associated with the set of edges $\mathcal{E}^{temporal}_{t\to t+1}$ that model temporal relations with its subsequent frame, formally:
\begin{equation}\label{eq:neighborhood}
    \mathcal{N}_{t,i} = \{R_{t+1,j} | d^{t\to t+1}_{i,j} < d_{proximity}\},
\end{equation}
\begin{equation}
    neighbor(R_{t,i})=\argmin_{R_{t+1,j}\in \mathcal{N}_{t,i}}{|R_{t,i}.c - R_{t+1,j}.c|_2},
\end{equation}
\begin{equation}
    \mathcal{E}^{temporal}_{t\to t+1} = \{(R_{t,i}, temporal, neighbor(R_{t,i}) | i\in [S]\}.
\end{equation}

Equipped with these definitions, we define the set of temporal edges connecting nodes corresponding to superpixels in frame $t$ to superpixels in frame \text{$t+1$} as the union of the temporal edge sets generated for all the frames:
\(
    \mathcal{E}^{temporal} = \bigcup_{t=1}^{T-1}{\mathcal{E}^{temporal}_{t\to t+1}}
\).
\input{algorithms/graph_generation}
\subsection{Model Architecture}\label{section:model_arch}
In order to model both the spatial and temporal relations between superpixels, our model primarily relies on the Neural Relational Model \cite{schlichtkrull2018modeling}, which is an extension of GCNs \cite{kipf2016semi} to large-scale relational data. In a Neural Relational Model, the propagation model for calculating the forward-pass update of a node, denoted by $v_i$, is defined as
\begin{equation}\small
    h_{i}^{(l+1)}=\sigma \left(\sum_{r\in \mathcal{R}}\sum_{j\in \mathcal{N}_{i}^{r}}{\frac{1}{c_{i,r}} W_{r}^{(l)}h_{j}^{(l)}+W_{0}^{(l)}h_{i}^{(l)}} \right),
\end{equation}
where $\mathcal{N}^r_i$ denotes the set of neighbor indices of node $i$ under relation \text{$r\in \mathcal{R}$} (not to be confused with the notation $\mathcal{N}_{t,i}$ from Eq. \ref{eq:neighborhood}). $c_{i,r}$ is a problem-specific normalization constant that can either be learned or chosen in advance (such as \text{$c_{i,r}=|\mathcal{N}^r_i|)$}. To incorporate edge features, we adapt the approach proposed in \cite{corso2020principal}, that concatenates node and edge attributes as a layer's input, yielding the following:
\begin{equation}\label{eq:concat_edges}\small
    h_{i}^{(l+1)}=\sigma \left(\sum_{r\in \mathcal{R}}\sum_{j\in \mathcal{N}_{i}^{r}}{\frac{1}{c_{i,r}} W_{r}^{(l)}[h_{j}^{(l)},e_{i,j}]+W_{0}^{(l)}h_{i}^{(l)}} \right),
\end{equation}
where $e_{i,j}$ is the feature of the edge connecting nodes \text{$v_i,v_j$}.

\subsection{Augmentations}\label{section:augmentations}
We introduce a few possible augmentations that we found useful for training our model as they improved the generalization.
\paragraph{Additive Gaussian Edge Noise (AGEN) -}
Edge attributes represent distances between superpixel centroids. The coordinates of those centroids may vary due to different superpixel shapes with different centers of mass. To compensate for this, we add a certain amount of noise to each edge attribute. Given a hyper-parameter $\sigma_{edge}$, for each edge attribute $e_{u,v}$ and for each training iteration, we sample a normally distributed variable $z_{u,v}\sim N(0,\sigma_{edge})$ that is added to the edge attribute.

\paragraph{Additive Gaussian Node Noise (AGNN) -}
Node attributes represent the colors of regions in each frame. Similar to edge attributes, the mean color of each segmented region may vary due to different superpixel shapes. To compensate for this, we add a certain amount of noise to each node attribute. Given a hyper-parameter $\sigma_{node}$, for each node attribute $v.c$ of dimension $d_c$ and for each training iteration, we sample a normally distributed variable $z_{v}\sim N_{d_c}(0,\sigma_{node}\cdot I_{d_c})$ that is added to the node attribute.

\paragraph{Random Removal of Spatial Edges (RRSE) -}
This augmentation tends to mimic the regularization effect introduced in DropEdge \cite{rong2019dropedge}. Moreover, since the removal of edges leads to fewer message-passings in a GCN, this also accelerates the training and inference. To perform this, we choose a probability \text{$p_{edge}\in[0,1]$}. Then, each edge $e$ is preserved with a probability of $p_{edge}$.

\paragraph{Random Removal of Superpixels (RRS) -}
SLIC \cite{achanta2010slic} is sensitive to its initialization. Consequently, each video clip may have several graph representations during different training iterations and inference. This can be mitigated by removing a certain amount of superpixels. The outcome is fewer nodes in the corresponding representative graph, as well as fewer edges. Similar to RRSE, we choose a probability \text{$p_{node}\in[0,1]$} so that each superpixel is preserved with a probability of $p_{node}$.

\subsection{Benefits of \textbf{\emph{GraphVid}}}
\paragraph{Invariance -}The absence of coordinates leads to invariance in the spatial dimension. It is evident that such a representation is invariant to rotations and flips since the relations between different parts of the image are solely characterized by distances. This, in turn, obviates the need to perform such augmentations during training.
\paragraph{Efficiency -}We argue that our graph-based representation is more efficient than raw frames. To illustrate this, let $T, C, H$ and $W$ be the dimensions of a clip; that is, the number of frames, number of channels, height and width of a frame, respectively. Correspondingly, the raw representation requires \text{$T\cdot C\cdot H\cdot W$}. To calculate the size of the graph-video, let $S$ be the number of superpixels in a frame. By construction, there are at most \text{$4\cdot S$} edges in each frame because SLIC constraints each to have 4 neighbors. Each edge contains $3$ values, corresponding to the distance on the grid, source and target nodes. Additionally, there are, at most, $S$ edges between every temporal step. This results in \text{$3\cdot (4\cdot S + (T - 1) \cdot S) + C\cdot T\cdot S$} parameters in total. Typically, the second requires much fewer parameters because we choose $S$ so that \text{$S \ll H\cdot W$}.
\paragraph{Prior Knowledge Incorporation -}
Optical-flow and over-segmentation are encoded within the graph-video representation using the inter-frame and intra-frame edges. This incorporates strong prior knowledge within the resultant representation. For example, optical-flow dramatically improved the accuracy in the two-stream methodology that was proposed in \cite{simonyan2014two}. Additionally, over-segmentation using superpixels has been found useful as input features for machine learning models due to the limited loss of important details, accompanied by a dramatic reduction in the expended time by means of reducing the number of elements of the input \cite{proceedings401,dadsetan2021superpixels,avelar2020superpixel}.
\section{Related Work}\label{section:related_work}

\subsection{Deep Learning for Video Understanding}
CNNs have found numerous applications in video processing \cite{mittal2021survey,tran2018closer,yue2015beyond}. These include LSTM-based networks that perform per-frame encoding \cite{srivastava2015unsupervised,ullah2017action,yue2015beyond} and the extension of 2D convolutions to the temporal dimension, \emph{e.g.}} \def\Eg{\emph{E.g.}, 3D
CNNs such as C3D \cite{tran2015learning}, R2D \cite{simonyan2014two} and R(2+1)D \cite{tran2018closer}.

The success of the Transformer model \cite{vaswani2017attention} has led to the development of attention-based models for vision tasks, via self-attention modules that were used to model spatial dependencies in images. NLNet \cite{wang2018non} was the first to employ self-attention in a CNN. With this novel attention mechanism, NLNet is possible to model long-range dependencies between pixels. The next model to be developed was GCNet \cite{cao2019gcnet}, which simplified the NL-module, thanks to its need for fewer parameters and computations, while preserving its performance. A more prominent transition from CNNs to Transformers began with Vision Transformer (ViT) \cite{dosovitskiy2020image}, which prompted research aimed at improving its effectiveness on small datasets, such as Deit \cite{touvron2021training}. Later, vision-transformers were adapted for video tasks \cite{neimark2021video,arnab2021vivit,bertasius2021space,fan2021multiscale,li2021vidtr,liu2021video}, now crowned as the current state-of-the-art that top the leader-boards of this field.

The usage of graph representation in video understanding sparsely took place in the work of Wang \cite{wang2018videos}. They used pre-trained Resnet variants \cite{he2016deep} for generating object bounding boxes of interest on each frame. These bounding boxes are later used for the construction of a spatio-temporal graph that describes how objects change through time, and perform classification on top of the spatio-temporal graph with graph convolutional neural networks \cite{kipf2016semi}. However, we note that the usage of a large backbone for generating object bounding boxes is harmful for performance. We intend to alleviate this by proposing a lighter graph representation. In combination of a dedicated GNN architecture, our representation greatly outperforms \cite{wang2018videos} in all metrics.

\subsection{Superpixel Representation of Visual Data}
Superpixels are groups of perceptually similar pixels that can be used to create visually meaningful entities while heavily reducing the number of primitives for subsequent processing steps \cite{stutz2018superpixels}. The efficiency of the obtained representation has led to the development of many superpixel-generation algorithms for images \cite{stutz2018superpixels}.
This approach was adapted for volumetric data via the construction of supervoxels \cite{papon2013voxel}, which are the trivial extension to depth. These methods were adjusted for use in videos \cite{6247802} by treating the temporal dimension as depth. However, this results in degraded performance, as inherent assumptions regarding neighboring points in the 3D space do not apply to videos with non-negligible motion. Recent approaches especially designed to deal with videos consider the temporal dimensions for generating superpixels that are coherent in time. Xu \emph{et al.}~\cite{10.1007/978-3-642-33783-3_45} proposed a hierarchical graph-based segmentation method. This was followed by the work of Chang \emph{et al.}~\cite{chang2013video}, who suggested that Temporal Superpixels (TSPs) can serve as a representation of videos using temporal superpixels by modeling the flow between frames with a bilateral Gaussian process.

\subsection{Graph Convolutional Neural Networks}
Introduced in \cite{kipf2016semi}, Graph Convolutional Networks (GCNs) have been widely adopted for graph-related tasks \cite{zhang2018network,kumar2020link}.
The basic GCN uses aggregators, such as average and summation, to obtain a node representation given its neighbors. This basic form was rapidly extended to more complex architectures with more sophisticated aggregators. For instance, Graph Attention Networks \cite{velivckovic2017graph} use dot-product-based attention to calculate weights for edges.
Relational GCNs \cite{schlichtkrull2018modeling} add to this framework by also considering multiple edge types, namely, relations (such as temporal and spatial relations), and the aggregating information from each relation via separate weights in a single layer.
Recently, GCNs have been adopted for tasks involving audio \cite{dokania2019graph,zhang2019few} and images \cite{monti2017geometric,dadsetan2021superpixels,avelar2020superpixel}. Following the success of graph models to efficiently perform image-based tasks, we are eager to demonstrate our extension of the image-graph representation to videos.


\section{Introduction}
The  plasma falling and crashing into the Sun after the eruptive flare on 2011 June 7 has been compared to stellar accretion flows \citep{Reale13}. The velocities and densities of impacting fragments are similar. Also in both situations the magnetic field may affect the flows as they fall towards the surface. The impacts during the 2011 June 7 event fell with a range of speeds and densities into a variety of magnetic environments.
They may therefore provide a testbed for accretion onto unresolved stellar objects.

Accretion onto young stars is thought to explain  excesses seen in their infrared \citep{Mendoza66,Rucinski85}, ultraviolet (UV) \citep{Herbig86} and soft X-ray \citep{Kastner02,Schmitt05} emission compared to stars of similar spectral type.
Models of the emission from accretion have made a number of critical assumptions regarding the structure of the accretion flows, the pre-impact sites, and the optical thickness of the heated plasma at the impact site.
For example, simulations often assume uniform plasma columns channelled along one-dimensional (1D) flux tubes \citep{Calvet98, Sacco08, Sacco10} or accretion  along uniform vertical fields \citep{Orlando10, Bonito14}.
 \citet{Reale13} used the spatially resolved solar impacts on June 7 to investigate the applicability of 2D hydrodynamic (HD) accretion models to prominence plasma falling back to the Sun. They investigated fragments with a velocity around 400~km~s$^{-1}$\ and a density $10^{10}$~{cm$^{-3}$}\ plunging through  the solar corona and into chromosphere.
 Similar to the stellar accretion models, the impacting fragments sank deep into the chromosphere where their energy was dissipated creating forward and reverse shocks with high-temperature outflows from the impact region into the surrounding corona.

  To model the emission
  \citet{Reale13} assumed all EUV emission from below the transition region was absorbed and therefore not detected. With this assumption, they concluded that the
observed EUV intensities originated from the 5 to 30\% of the fragment plasma that radiated above the transition region.
A study of stellar accretion revealed that accretion rates derived from X-ray fluxes are considerably lower than those inferred from UV/optical/ infrared fluxes \citep{Curran11}, suggesting that
in stellar systems X-rays may also be absorbed. \citet{Reale13} argue that  absorption at the impact site may  explain the lower accretion rates derived from X-ray observations; however
their models did not account for structure induced by magnetic fields at the impact site.   Magnetic fields may reduce the
 local absorption \citep{Orlando13,Bonito14}, leading \citet{Bonito14} to conclude that the unperturbed, trailing accretion stream rather than absorption at the impact site is a more probable explanation for the low X-ray fluxes.

The solar observations of the June 7 impacts can be used to investigate impact depth and effects of magnetic structuring at the impact site by comparing high resolution EUV  (131, 171, 193, 211~\AA)  and UV (1600 and 1700~\AA)  observations of the impacts because the  EUV and UV emission come from different depths in the atmosphere. The EUV emission is only visible above the transition region because EUV from below is obscured by \ion{H}{i}, \ion{He}{i}, \ion{He}{ii}. On the other hand both the transition region and chromosphere are transparent to UV emission.  Thus the UV intensities give a indication of how much plasma penetrates to the chromosphere. Naively one would expect that since the EUV from deep impacts is absorbed, then the intensity of the UV to EUV (I$_{UV}$/I$_{EUV}$) should increase the deeper the impact. Also according to \citet{Sacco10},  the faster impacts should penetrate deeper into the atmosphere. Consequently one expects the (I$_{UV}$/I$_{EUV}$) to increase with velocity.
By surveying the UV and extreme ultraviolet (EUV) emission from solar impacts of different sizes and velocities, we investigate the conditions controlling the optically thin (UV) to optically thick (EUV) intensity ratios at impact sites on the Sun.

The eruption on 2011 June 7 in NOAA AR 11226 (S22W55) resulted in a fast coronal mass ejection (CME) with velocity about 1250~km~s$^{-1}$\  in interplanetary space. In EUV images,  a dome-shaped CME wave expanded  across the disk with velocities of up to 960~km~s$^{-1}$\ \citep{Li12, Cheng12}. Behind the fast wave, filament plasma was thrown upward, and a lot of it fell back creating impacts across the solar surface.  Four of the impacts into the quiet Sun have been studied by \citet{Gilbert13}. In each of these, the main brightening is due  to dissipation of kinetic energy on impact. The eruption was seen from both the Solar Dynamics Observatory (SDO) and the Solar TErrestial Relations Observatory Ahead (STEREO-A) spacecraft when STEREO-A was 94$^\circ$ ahead of Earth.  \citet{Gilbert13} used images from both these spacecraft to estimate the height-time progress of several ejecta. Although straightforward  in theory, in practice it is difficult  to pick out exactly the same  ejecta from both viewpoints in the image pairs with 2.5~min cadence, especially when the material is close to the Sun. The velocities given by \citet{Gilbert13} were determined roughly 5~min before impact  which means that the actual impact velocity is probably about 50-100~km~s$^{-1}$\ higher, given the solar gravitational acceleration close to the surface, $274 \,\mathrm{m\,s^{-2}}$.

The coronal magnetic field was complex with connections between the three neighbouring active regions in the south \citep{vandriel14} and the active region to the north. As the plasma falls back through the corona, it breaks up due to Rayleigh-Taylor type of instabilities \citep{Innes12} generating long, thin strands with dense clumps at their heads \citep{Carlyle14}.

In this paper we present the results of a survey of the UV and EUV emission from 16 fragments. In the next section, we describe the  image sequences used. The analysis first explains the basis for the impact velocity, then in the next section we present the characteristics of the 16 impacts, and show details of fragments impacting into various environments. The discussion section summarises the results and in particular the conditions for large UV to EUV intensity ratios (I$_{UV}$/I$_{EUV}$).

\section{Observations \label{sec:obs}}
We have determined the light curves of impacts from images obtained by the Atmospheric Imaging Assembly \citep[AIA;][]{Lemen12} onboard SDO in the 1700, 1600, 131, 171, 193, 211~\AA\ channels and thus sample the atmosphere's response over a range of temperatures and hence atmospheric heights.  The  EUV channels were taken with a cadence of 12~s and the UV channels alternated between 1600 and 1700~\AA\ with a 12~s interval between the two channels so the cadence of each UV channel was 24~s.  At the time of the eruption,  304~\AA\ images were taken with a low, 60~s, cadence and were therefore not used in the analysis.
The 1700~\AA\ emission was mostly continuum from the chromosphere with a small contribution from line emission from the transition region. Similarly continuum from the chromosphere was the main contributor to the 1600~\AA\ channel but there may have been a significant contribution from the transition region  due to \ion{C}{iv}\ emission. The 131~\AA\ emission from the impacts was usually weak and mostly due to \ion{Fe}{viii}.  The 171~\AA\ emission, predominantly from \ion{Fe}{ix}, came from slightly cooler plasma (0.63~MK) than the \ion{Fe}{xii}\ emission (1.4~MK) seen in the 193~\AA\ channel and the \ion{Fe}{xiv}\ (1.9~MK) revealed by the 211~\AA\ channel. Impacts in the chromosphere  probably had additional contributions to the EUV channels from  the transition region \citep{Brosius12} as the plasma was rapidly ionising so temperatures derived from the emission ratios of these bright impacts are likely to be unreliable; however the I$_{171}$/I$_{193}$ or the I$_{171}$/I$_{211}$ should give a good indication of the  temperature of plasma around coronal brightenings caused by impacts.

 In all EUV images the cold filament  plasma is seen as dark structures against the background corona due to absorption of H and He photoionising radiation. They are best seen in the well-exposed 171 and 193~\AA\  images. The background is generally smoother in the 193~\AA\ images since it was from the hotter corona, and we used this channel for determining column densities of the impacting fragments. Column densities were derived with the simple one-wavelength method suggested by \citet{Williams13} that uses the relative darkening of the corona due to the filament plasma.

Images from the Extreme Ultraviolet Imager (EUVI) \citep{Wuelser04} at 195~\AA\  on the STEREO-A spacecraft were used in combination with images from  the AIA 193~\AA\ channel to obtain height-time profiles for some of the ejecta. The relatively low cadence of STEREO-A (2.5~min) and the wide separation of the spacecraft (94$^\circ$) made it difficult to unambiguously pick  out the same part of the same ejecta in the AIA/EUVI-A image pairs.
Since the eruption was close to the east limb in the EUVI images, many of the early ejecta falling on the east of the eruption were hidden in the STEREO images due to intervening filament ejecta. The ones best seen from both spacecraft were those from the northern edge and the later, well-separated  ejecta falling from greater heights.

The magnetic fields at the impact sites were obtained from  HMI \citep{Scherrer12} line-of-sight magnetograms. These were  generally weak (<50~G).

\section{Filament eruption}
The filament rise started at 06:28~UT. Material was rapidly ejected into the corona with a distribution of velocities so that the central part rose faster and further than the filament sides. Using AIA and EUVI-A images, the paths  taken by different parts of the eruption could be tracked. For example, in Fig.~\ref{filament_rise},
the ejecta along the northern path (black) had an average speed between 320 and 370~km~s$^{-1}$, and the speed along the southern path (white)  was between 450 and 600~km~s$^{-1}$. The uncertainty is because the ejecta change shape which made it difficult to find exactly the same plasma in consecutive EUVI images, taken 2.5~min apart.

Material from the northern edge is visible throughout its ascent and descent in both AIA and EUVI-A images (Fig.~\ref{filament_images}).  This allowed us to track the height-time profile of an early impact from the northern edge. The height-time path, shown in Fig.~\ref{filament_path},  is well fit by a parabola representing ejecta  leaving the solar surface at 06:25:30~UT with a velocity of 232~km~s$^{-1}$. Note that in this fit the ejection from the surface is 2.5~min earlier than the start time of the filament eruption from 1.07~R$_\odot$.
This is the ejecta responsible for `region 3' studied by \citet{Gilbert13}. They used the last two points about 5 min before impact, at 1.03 and 1.07~R$_\odot$, to deduce a velocity of 146~km~s$^{-1}$, which they took as the impact velocity. This corresponds with our fit;  however the true impact velocity, obtained by extrapolating to the surface,   is about 80~km~s$^{-1}$\ higher.
In this case, it looks as though  the ejecta did follow a ballistic path but it is the only impact  where we were able to track almost the entire path of the ejecta so we cannot say if other ejecta were also ballistic. Some of the higher ejecta appear to have been slowed  by drag forces \citep{Dolei14}. For all impacts we computed the ballistic impact velocity from the flight time, and for those impacts that could be tracked in both AIA and EUVI-A images, the triangulation velocity about 5 min before impact.
As discussed by \citet{Reale13} the velocity may be a critical parameter for understanding the impact emission.

\begin{figure}
	\centering
	\includegraphics[width = 0.9\columnwidth]{filament_rise-eps-converted-to.pdf}
	\caption{AIA 193~\AA\ and EUVI-A195~\AA\ images of the rising filament plasma. The asterisks mark the paths followed by different parts of the filament. The northern (black) part has a velocity of about 350~km~s$^{-1}$, and the southern (white) part about 500~km~s$^{-1}$.}
	\label{filament_rise}
\end{figure}

\begin{figure}
	\centering
	\includegraphics[width = 0.9 \columnwidth]{filament_track-eps-converted-to.pdf}
	\caption{AIA 193~\AA\ and EUVI-A 195~\AA\ images of the descending filament plasma. The asterisks mark the path followed by an early impacting fragment on the northern side of the filament.}
	\label{filament_images}
\end{figure}

\begin{figure}
	\centering
	\includegraphics[width = 0.6 \columnwidth]{filament_accel-eps-converted-to.pdf}
	\caption{The height-time profile of the early fragment shown in Fig.~\ref{filament_images} (dashed curve) fitted with a parabola representing ejecta from the solar surface with a velocity of 232~km~s$^{-1}$, solar  gravitational acceleration of $274~\,\mathrm{m\,s^{-2}}$, and zero drag.}
	\label{filament_path}
\end{figure}

\section{Impacts\label{sec:impacts}}

We  selected 16 impacts. Table~\ref{table1} lists the best estimates for the impact landing time, their position, their maximum intensity in the 171~\AA\ channel integrated over a box of roughly 10" x 10", their column density $2-4$~min before impact from the darkening in the 193~\AA\ channel, the mass of the impacting material, the impacts' velocity assuming ballistic motion and an ejection time from the surface at 06:25:30~UT, their velocity measured from triangulation, and the average magnetic field in an area 10\arcsec\ x 10\arcsec\ around the impact.  Triangulation results in a velocity about $50-100$~km~s$^{-1}$\ less than ballistic. As mentioned already, the difference can be attributed to the fact that the last triangulation position was measured $2-5$~min before impact and the velocity was computed from the last two points in the trajectory.  The magnetic fields were typically quiet-Sun values except for impact 9 which landed in an active region.  Impact 15 landed in a predominantly negative-polarity coronal hole region.

\begin{table*}
\begin{center}
	\begin{tabular}{l c c c c c c c c c}
	\hline
	Impact 	& Time 	&Sun x, y	&I$_{171}$	&Column	 			&Mass		&V (ballist.) 	& V (triang.)	& B+	 	& B- 		\\
			&(UT)	&(arcsec)	&(10$^4$~DN~s$^{-1}$)	& ($10^{18}$cm$^{-2}$)& ($10^{12}$~g)&(km~s$^{-1}$)&(km~s$^{-1}$)		& (G)	& (G)	\\
	\hline
	1$^1$	& 07:03:05 &820, 30	&77			& 3.6$\pm0.2$		&				&232			& 145		&6 		&5 	  	\\
	 2	& 07:03:53&630, -195&16			& 5.2$\pm0.5$		&				&235			 & 140-180	&5		&6		\\
	3	&07:03:53	&720, -100&13			& 7.7$\pm$0.6		&3-4				&235			 & 165		&3		&3 		\\
	4	&07:10:41&650, 30&6.8			& 5.0$\pm$0.5		&1-2				&265			&150-200		&5		&4		\\
	5	&07:12:17&600, -20	&11			& 3.0$\pm$0.5		&				&275			&			&5		&4		\\
	6	&07:16:17	&735, 175	&4.8			&3.2$\pm$0.1		&				&290			&			&6		&6		\\
	7	&07:17:29	&735, 145	&2.8			&3.0$\pm$0.1		&				&290			&			&4		&6		\\
	8	&07:21:05&485, -100&5.9			&3.6$\pm$0.3		&				&305			&			&5		&7		\\
	9	&07:23:29	&620, 245&89			&6.6$\pm$0.4		&2-3				&310			&250			&6		&30		\\
	10	&07:26:17&415, -75	&5.0			&3.0$\pm$0.4		&				&315			&			&5		&4		\\
	11$^2$	&07:28:17	&520, -80	&21			&2.6$\pm$0.3		&				&320			&			&6		&6		\\
	12		&08:07:53	&405, -75	&66			&3.7$\pm$0.1		&				&400			&300			&6		&5		\\
	13$^1$	&08:08:41	&390, -65	&220			&3.7$\pm$0.1		&2-3				&405			&300			&6		&5		\\
	14	&08:50:48	&300, 40	&31			&3.9$\pm$0.2		&2-3				&445			&380			&4		&4		\\
	15	&08:51:31	&315, 120	&49			&<1.5			&				&445			&380			&3		&7		\\
	16	&08:51:55	&415, 65	&10			&2.2$\pm$0.2		&				&450			&380			&3		&3		\\
	\hline
	\end{tabular}
	\caption{Parameters of the selected impacts. I$_{171}$  are the peak impact 171~\AA\ channel intensities. The column densities are an estimate of the neutral hydrogen component 2-4~min  before impact. The mass was computed by integrating the column densities in the falling plasma. The ballistic velocity assumes an ejection time from the surface 06:25:30~UT. The triangulation velocity is the velocity about 5~min before impact. The magnetic field strengths, B+ and B-, are the average positive and negative field  in the region around the impact.
Marked impacts have been analysed by $^1$\citet{Gilbert13}, $^2$\citet{Reale14}.
\label{table1}}
\end{center}
\end{table*}

\subsection{UV and EUV intensities}
 \citet{Gilbert13} have shown that on impact there was generally a rapid increase to peak intensity in all EUV channels simultaneously, followed by a slower intensity decrease. The sharp rise was caused by the initial impact.  The increase in pressure due to the impact subsequently changed the environment into which the trailing fragment plasma was falling. In several events bright EUV emission, looking like a splash, was seen propagating back from the impact site. The characteristics of one of the largest splashes is described later.  In this section we concentrate on emission from the bright impacts, not their surroundings.

To avoid emission from the surroundings, we have chosen compact, short-lived impacts, and obtained light curves in the close vicinity of the impacts.  We typically used boxes of size 10" x 10" but they varied according to the size of the bright emission kernel. When the impact was part of a larger group, the  intensities were taken from a small region centred on the brightest 1600~\AA\ site. If there was no noticeable 1600~\AA\ brightening then the region is centred on the brightest 171~\AA\ patch.
Since the impacts were much brighter than the surroundings and were computed after background subtraction, results are not sensitive to the box size.

Fig.~\ref{composite} shows an example of a fragment group. The group started in the corona as a single finger, the front then broke up into at least 6 smaller fingers. The physics of the break up is still not well understood. It seems to have been influenced by the magnetic field and the interaction of the wind with the partially ionised falling plasma \citep{Innes12}. Each high density finger tip led to a bright circular impact at 1600~\AA.  This is illustrated in Fig.~\ref{composite}(c) where we show a composite of the brightest 1600~\AA\ and the dark,193~\AA\ downflow tracks. Downflow tracks were used by \citet{Reale13} to show the motion of the falling fragments. These tracks are given by the minimum intensity value of each pixel during the downflow.
The 171~\AA\ emission from the impact was more diffuse, covered a larger area and lasted longer than the 1600~\AA. It came both from the corona and the impact site.
The impact, indicated by an arrow and surrounded by a box in (d), is impact 4. The box indicates the region over which the intensities were integrated.

As seen in Table~1, the impacts' 171~\AA\ intensities covered a range  of almost two orders of magnitude and show no simple relation  to either the mass or velocity in the falling plasma.
The intensities are integrated over the box surrounding the impact site and their intensity depends more on the size of the impact site, than on the maximum intensity over the impact site.
For example impact 13 was seven times brighter than  14 although it had the same mass and was slower.  As discussed below, the enhanced EUV intensities were probably due to dissipation of kinetic energy in the transition region or corona  where trapping along low-lying coronal field prevented the fragment from falling as deep in the chromosphere as the HD models of  \citet{Reale13} predict.

The maximum of the impact's intensity in several channels relative to the maximum  171~\AA\ intensities are represented as a column chart in Fig.~\ref{intensities_fig}.
Two early impacts, 2 and 3, stand out because of their much higher-than-average I$_{UV}$/I$_{EUV}$. Both these impacts had an impact velocity between 230 and 250~km~s$^{-1}$\ and landed in quiet-Sun.
A close up of impact 3's 1600 and 171~\AA\ emissions is shown in Fig.~\ref{dyrus_fig}. The 171~\AA\ brightening occurred about 30~s after the 1600~\AA\ brightening.
According to the HD simulations of \citet{Reale13}, impacts with velocity about 250~km~s$^{-1}$\ should not produce significant EUV emission so there must be additional dissipation effects not accounted for in their simulations.

The time delay was typical  for the EUV and UV emissions, as noted by \citet{Reale14}. It was seen in all impacts with roughly circular UV emission kernels. An explanation, given in \citet{Reale14}, is that the 1600~\AA\ is from a second stream of falling fragments that is shock-heated to transition region temperatures by the backflow from the initial impact; however in most of the impacts studied here (e.g. Fig.~\ref{composite}) the plasma accumulates at the head of the fragment and the 1600~\AA\ appeared in front of the downflow tracks so this explanation is unlikely. Quantitative computations showing the piling up of plasma at the head of fragments have been reported by \citet{Carlyle14}. Another problem with the \citet{Reale14} scenario is  that the main peaks in the 1600~\AA\ are almost 100~s after the initial impact (Fig. 1c in \citet{Reale14}) but the observations show only a single peak  from the impact site.
 Also the brightening was seen from the same sites and with similar light curves in both the 1600 and 1700~\AA\ channels (Fig.~\ref{lc_dyrus_fig}), implying that it is chromospheric not transition region emission.
To understand the delay between the EUV and UV requires more detailed models of the dynamics and ionization of the chromosphere's reaction to the impacts, including magnetic effects.

The circular shape of the 1600~\AA\ suggests that the emission was from the essentially uniform chromosphere rather than the more structured corona. Many impacts produced thread-like structures at 171~\AA\ surrounding or connecting the main impact site suggesting that although quiet Sun, the corona may still be  structured by the magnetic field.

There were two impacts with abnormally low I$_{UV}$/I$_{EUV}$; the one  in the active region (9), and one low column-density impact (15). The low column-density impact  (15) was the only one with  a I$_{193}$/I$_{171}$ greater than one. The ratio was around 0.5 for most other impacts.

\begin{figure*}
	\centering
	\includegraphics[width = 0.9\linewidth]{fragments_fig-eps-converted-to.pdf}
	\caption{The relationship between the downflowing plasma and the impacts at 1600~\AA: (a, b) AIA 171~\AA\ images showing the finger and arc structure of the downflowing material; (c)  composite of dark tracks of 193~\AA\ absorption in the fragment tips and the bright 1600~\AA\ impact emission; (d) the impact in 171~\AA. Dense structures in the downflowing plasma impact the chromosphere resulting in circular brightenings in 1600~\AA\ (c). In (d) the arrow points to impact 4 and the black box shows the region used for integrating the fluxes. Panel (c) is the central impact in the movie composite\_1600. The region is indicated by an arrow at the end of the movie.}
	\label{composite}
\end{figure*}

\begin{figure*}
	\centering
	\includegraphics[width = 1 \linewidth]{intensities-eps-converted-to.pdf}
	\caption{Impact intensity in different UV and EUV channels relative to the 171~\AA\ intensity. Note the decrease in I$_{UV}$/I$_{EUV}$  with time and the increase in I$_{193}$/I$_{171}$  for later impacts.  }
	\label{intensities_fig}
\end{figure*}

\begin{figure}
	\centering
	\includegraphics[width = 1.0\columnwidth]{dyrus_mf-eps-converted-to.pdf}
	\caption{The early low-velocity impact 3 with strong UV to EUV: top row 171~\AA: bottom row 1600~\AA.}
	\label{dyrus_fig}
\end{figure}

\begin{figure}
	\centering
	\includegraphics[width = 0.5\columnwidth]{lc_dyrus_fig-eps-converted-to.pdf}
	\caption{UV and EUV light curves of the early low-velocity impact, impact 3, shown in Fig.~\ref{dyrus_fig}. The curves are labelled with their channel wavelength in \AA. }
	\label{lc_dyrus_fig}
\end{figure}

\subsection{Column densities}
Since the kinetic energy of the impact is critical to understand the emission, the impacting mass is an important parameter. We have tested various methods to calculate the mass but are not satisfied with any. The computation of mass is subjective because as the  plasma fell it broke up creating a broad front of fragments (e.g. Fig.~\ref{composite}) so that one needs to decide when to measure the mass. Also if the falling plasma was in the form of long fingers, it was not obvious which part to take as  the impact mass. In several cases, however, it was possible to identify the impacting mass. The estimates given in Table~1 were obtained in these cases by integrating the hydrogen column density of impacting plasma over the fragment area, dividing by the hydrogen abundance, 0.92, and multiplying by the pixel size, 0.19 Mm$^2$, and the mean atomic mass, $2\times10^{-24}$~g.

For all impacts we were able to estimate the neutral hydrogen column density in individual fragments.
It was not possible to determine the line-of-sight width of the fragments so densities could not be calculated,  nevertheless column density may give a rough idea of the relative densities of the fragments.  The column densities were obtained  by taking the ratio of the track image to a background image in the 193~\AA\  channel which gives the evolution of the maximum optical depth at 193~\AA\ in the falling fragment. Then using Equation 4 and Figure 2(a) of \citet{Williams13}, the evolution of the neutral hydrogen column density was calculated. The column density, given in Table~1, is the average hydrogen column density of the highest 150 pixels  2-4 min before impact. The uncertainty  is the corresponding variance. In Fig.~\ref{columndensity}, we show the column-density track of impact 14. A yellow box shows the region from which the highest 150 column densities were selected to compute the column density for the fragment. The actual number of pixels taken was not critical, and we found similar column densities for 30\% more or less pixels.  We note that close to the solar surface the neutral hydrogen column density decreases by at least a factor two, implying ionisation before impact. The ions will be trapped by the coronal magnetic field causing bright EUV emission outside the main impact site (see movie\_impact15).

The range of column densities, a factor four, is much smaller than the range of 171~\AA\ intensities. It is probably not the main parameter in determining the 171~\AA\ intensity but as discussed below it may play a factor in determining I$_{UV}$/I$_{EUV}$.

\begin{figure}
	\centering
	\includegraphics[width = 0.7\linewidth]{column_mod-eps-converted-to.pdf}
	\caption{Column density during the downflow of impact 14. The yellow box outlines the region used to compute the column density in Table 1.}
	\label{columndensity}
\end{figure}

\subsection{Magnetic field}
We have looked at the relationship of the impacts' intensities to the local field at the impact site. The majority of impacts occurred in quiet Sun and we include one impact on the edge of an active region  (9) and one low column-density impact  in a coronal hole region (15).

The images of impacts towards the coronal hole show bright EUV threads that were probably illuminating the structure of the magnetic field on the edge of the coronal hole.  They are best seen in the movie, impact15\_movie. Here one sees one dark fragment to the south and a broad, very faint fragment heading towards the coronal hole. Fig.~\ref{gbm} gives an overview of the downflows and impacts. The downflow track and 1600~\AA\ emission in Fig.~\ref{gbm}(a) and the southern 171~\AA\ emission in (c) is impact 14.  The display of brightenings on the edge of the coronal hole seen in (d) are grouped together as impact 15. They were produced by a faint sheet-like fragment that is not visible in individual images but can be seen in impact15\_movie.
Although there were no high column-density fragments associated with the EUV threads, they were   as bright in the  EUV  as the neighbouring  impact caused by a high column-density impact in the chromosphere.

\begin{figure}
	\centering
	\includegraphics[width = 0.9\linewidth]{spl_gbm_fig-eps-converted-to.pdf}
	\caption{Overview of impacts 14 and 15, as seen in UV  (1600~\AA) and EUV  (171~\AA) channels. The red/blue contours outline magnetic fields of +/-50~G. (a) composite of 1600~\AA\ and downflow tracks. (b) 171~\AA\ base-difference image showing a faint downflow. (c - d)  171~\AA\ base-difference images of the impact, scaled between $\pm300$~DN~s$^{-1}$.  The white arrow points to the position of a very faint falling fragment. It is visible in the movie, impact15\_movie. Impact 14 is the compact site indicated by an arrow in (c). Impact 15 is the large region of diffuse 171 in (d).
}
	\label{gbm}
\end{figure}

The impact in the active region, impact 9, was similar to the low column-density impact in that it produced lower I$_{UV}$/I$_{EUV}$ for its velocity than impacts in the quiet-Sun chromosphere. The accompanying movie of the impact, impact9\_movie, shows that it started as a high column-density fragment that, about 3~min before impact, dispersed in the corona as it crossed a region affected by an earlier small  splash. An overview of the impact is provided by  Fig.~\ref{piglet}.
The images show a bright fan-shaped region of EUV emission at the impact site and very little UV, implying that the downfalling fragment dissipated its kinetic energy in the corona  and did not  reach the chromosphere.

\begin{figure}
	\centering
	\includegraphics[width = 0.9\columnwidth]{spl_piglet_fig-eps-converted-to.pdf}
	\caption{Overview of an impact, impact 9, in an active region as seen in UV (1600~\AA) and EUV (171~\AA) channels. The red/blue contours outline magnetic fields of +/-100, 300, 500~G. (a) composite of 1600~\AA\ and downflow tracks. (b) 171~\AA\ base difference image showing the downflow filaments. (c - d)  171~\AA\ base-difference images of the impact, scaled between $\pm300$~DN~s$^{-1}$. The base image was at 07:10:00~UT. The red arrow points to the impacting filament and the white arrow points to an earlier impact that seems to have affected the structure of this impact.
	The evolution of the impact is shown in impact9\_movie.
	}
	\label{piglet}
\end{figure}

\subsection{Splashes}
EUV splashes were seen after several of the impacts in the  quiet Sun. They mostly appeared where the impacting fragment was unusually long or wide. Several of the  splashes were as bright as the impact itself in the 171~\AA\ channel.
The  main features of quiet-Sun splashes are illustrated by the biggest sequence of impacts that started  around 08:00~UT and had an impact speed about 400~km~s$^{-1}$. As shown in impact12\_movie, there were two successive streams of impacts following roughly the same path. The first stream, shown in the right and middle panels of Fig.~\ref{imp_3}, produced the `region 1' investigated by \citet{Gilbert13}. Where the impact was in the chromosphere, the landing site brightened in all channels (UV and EUV). The splash, represented in the lower images of the middle panel, moved back towards the inflowing material with a plane-of-sky velocity of about 100~km~s$^{-1}$. The effect of the splash on the second stream was to prevent the impacting fragment from reaching the chromosphere. As shown in the top-left image of the the right-hand panel of Fig.~\ref{imp_3},  little UV emission was produced by the second stream.
 For comparison, the splash structure simulated by \citet{Reale14} had a speed of 200~km~s$^{-1}$\ and was more extended in the direction away from the infalling plasma.

Fig.~\ref{imp_211} shows the impact and splash sequence in the 1600, 171, 193 and 211~\AA\ channels.
The lower density, filamentary parts of the downflows on the edge of the main downflow, dissipated in the lower corona producing bright EUV emission (e.g. the region in (f) marked with a white arrow). The splash appeared later and was visible in the EUV channels. Compared with the neighbouring low column-density impacts dissipating in the corona, the splash had a higher I$_{171}$/I$_{193 }$  (or I$_{171}$/I$_{211}$). This implies that the splash was cooler than the impacts dissipating in the corona.

The light curves of the 1600,  and 193~\AA\ emission, obtained by integrating  base-difference intensities over the whole region, are  shown in Fig.~\ref{lc_imp}.
The head of the fragment produced the brightening around 12~min seen in the 1600~\AA\ image (Fig.~\ref{imp_211}(e)). The impacts 12 and 13 caused the series of spiky brightenings between 17 and 22~min. A broader, more gradual  increase of 1600~\AA\ emission occurred around 37~min.
The images show that this is because there were many small, barely perceptible  brightenings at 1600~\AA\  (Fig.~\ref{imp_211}(m)) coinciding with the EUV impact emission from the second stream.
We note that there was no bright 1600~\AA\ emission in the corona contrary to the predictions by \citet{Reale14}. This may be because the density of the observed fragments was  lower than those simulated by \citet{Reale14}. They assumed spherical fragments with density $5\times10^{10}$~{cm$^{-3}$}\ and radii $1.4-2.6\times 10^8$~cm , whereas the dark fragments in Fig~\ref{imp_211} have widths of typically $8\times10^8$~cm and column densities $3.7\times10^{18}$~{cm$^{-2}$}\ which corresponds to a density of about $5\times10^9$~{cm$^{-3}$}. The observed fragments  may therefore have had a factor ten or so lower density than the simulated ones, and this could explain why the fragments rapidly ionise in the corona and produce very little 1600~\AA\ emission.

There are subtle differences in the 171 and 193~\AA\ intensities that can be attributed to the effects of the low column-density impacts and the splashes. The low  column-density impacts caused the early increase in the 193~\AA\ intensity and the bump in the 193~\AA\ intensity around 25~min.
The splash was responsible for the increased 171~\AA\ intensity immediately after the impacts at 20 and 40~min.
The decrease in 171~\AA\ intensity at the beginning is because the falling fragment obscured bright 171~\AA\ emission during its descent.

\begin{figure*}
	\centering
	\includegraphics[width = 1.0\linewidth]{spl_imp_3-eps-converted-to.pdf}
	\caption{Overview of the two consecutive streams in the large impact, impact 12, as seen in UV (1600~\AA) and EUV  (171~\AA) channels.
	Each panel of four shows in (a) a composite of 1600~\AA\ emission and tracks of downflows from 193~\AA\ images and (b-d) base-difference images chosen to show the movement of the bright splash emission. The red/blue contours outline magnetic fields of +/-50~G. The base image was taken at 7:50:00~UT. The 171~\AA\ images are scaled between $\pm300$~DN~s$^{-1}$. Left: The first EUV brightenings were from low density impacts that dissipated in the corona.  Middle: A strong splash was created by the first series of  chromospheric  impacts. Right: The splash from the first series prevented the second series of fragments from reaching the chromosphere so there was very little 1600~\AA\ emission. Movies impact12\_171movie and impact12\_193movie, available online, show the full sequence of base-difference images at 171 and 193~\AA.}
	\label{imp_3}
\end{figure*}

\begin{figure*}
	\centering
	\includegraphics[width = 0.7\linewidth]{spl_imp_211_fig-eps-converted-to.pdf}
	\caption{Overview of the large impact, impact 12, as seen in UV (1600~\AA) and EUV  (171,  193 and 211~\AA) channels. The EUV images are base differences with a base image at 07:50:00~UT. The red/blue contours outline magnetic fields of +/-50~G. The white arrow in (f) points to bright EUV emission produced by a low density impact. The black arrow in (m) points to diffuse 1600~\AA\ emission during the second stream of impacts.
	}
	\label{imp_211}
\end{figure*}

\begin{figure}
	\centering
	\includegraphics[width = 0.7\linewidth]{lc_imp-eps-converted-to.pdf}
	\caption{Light curves during the big impact, 12,  taken over the region shown in Fig.~\ref{imp_211}: red -  1600~\AA, black -171~\AA, and blue - 193~\AA. }
	\label{lc_imp}
\end{figure}

A second large impact with interesting splash effects in the 1600~\AA\ and various EUV channels is shown in Fig.~\ref{aty_211}. This  was the only impact in our study
with structured 1600~\AA\ suggesting bright \ion{C}{iv}\ emission from falling material above the chromosphere especially as it was seen along
the front of a second stream of infalling plasma   \citep{Reale14}. The brightest sites, indicated with white arrows in Fig.~\ref{aty_211}, were probably chromospheric impacts because they produced circular brightenings at 1700~\AA\ as well. The threads in between have much lower I$_{1700}$/I$_{1600}$, indicating  transition region emission.
These threads are unlikely to have been caused by scattered impacts in the chromosphere.
It was also the lowest velocity impact. Thus consistent with  high I$_{UV}$/I$_{EUV}$ observed in low-velocity impacts in the quiet chromosphere we see that low-velocity impacts dissipating higher in the atmosphere also have higher I$_{UV}$/I$_{EUV}$. The high I$_{211}$/I$_{171}$ at the same site suggests that the 1600~\AA\ was from the impacting plasma not from the splash.
The splash can be seen in impact1\_movie propagating back from the impact site in the direction of the falling material.

\begin{figure*}
	\centering
	\includegraphics[width = 0.8\linewidth]{spl_aty_211_fig-eps-converted-to.pdf}
	\caption{Overview of a large early impact just after '1' as seen in UV (1600~\AA) and EUV  (171,  193 and 211~\AA) channels. The EUV images are base differences with a base image at 07:50:00~UT. The movie impact1\_movie shows the evolution at 171~\AA. The arrows point to 1600~\AA\ sites that were also bright at 1700~\AA.
	}
	\label{aty_211}
\end{figure*}

\section{Discussion}
We have studied the UV and EUV emission of 16 filament fragments that fell onto the Sun after the 2011 June 7 eruption with impact velocities ranging from 230 - 450~km~s$^{-1}$\ assuming  ballistic propagation: an assumption that has been justified by stereoscopically tracking the path of one fragment in EUVI-A and AIA images. The impacts range  over two orders of magnitude in 171~\AA\ intensity,  and had column densities ranging from less than 1.5 to 7.7$\times10^{18}$~cm$^{-2}$. All impacts produced bright EUV emission and the strength of the I$_{UV}$/I$_{EUV}$ emission depended on impact velocity, column density, magnetic environment, and presence of splashes caused by earlier impacts.

For impacts in the quiet Sun not affected by splashes, the I$_{UV}$/I$_{EUV}$  decreased with velocity.
This suggests that at low velocities there was not as much impact energy to heat and ionise all plasma at the impact site to coronal temperatures so the I$_{UV}$/I$_{EUV}$ was relatively large \citep{Sacco10}. The lowest column-density impact had the highest I$_{193}$/I$_{171}$ suggesting that low density impacts result in higher temperature emission. Image sequences of the  plasma during the largest quiet-Sun impact (Fig.~\ref{imp_211}) showed the same tendency. We suspect that the reduced I$_{UV}$/I$_{EUV}$ was because low density filaments become essentially fully ionised before reaching the chromosphere. Therefore the impacting plasma was trapped in the corona and did not reach the chromosphere. Since the coronal density was low, the impact rapidly heated the  coronal plasma to temperatures above 2~MK, and conduction transported the heat down to the chromosphere. In contrast, the kinetic energy of  impacts  into the high-density, low-temperature  chromosphere  was concentrated at the impact site creating a large increase in local pressure \citep{Reale13}. The dissipated energy heated the local chromosphere from less than $0.01$ to more than 3~MK, resulting in a broad range of emission temperatures, with a bias towards the cooler coronal values.

Dissipation in the corona also explains the emission observed from impact 9 in the active region. Compared to impacts with the same velocity in the quiet Sun, the I$_{193}$/I$_{171}$ was higher and the I$_{UV}$/I$_{EUV}$ lower.
Also one sees in
  impact9\_movie that the EUV emission spreads forward whereas  in the higher density quiet-Sun cases the EUV emission brightened at the impact site  and/or  splashed backwards. Probably near the active region the falling material was ionised when still high in the corona. Thus it was trapped and dissipated in the magnetic structures in the corona. The column density of the active-region fragment was relatively high and if falling directly onto the quiet Sun, it should have created a UV kernel.  It may have been ionised either by the increased EUV flux from the active region or shocked, heated and ionised by the splash from an  earlier small impact (see Section~4.3 and Fig~\ref{piglet}).

Splashes were observed in several of the impacts (1, 4, 11, 12, 13). They are most visible at 171~\AA, implying a temperature slightly under 1~MK. Their plane-of-sky propagation velocity was about 100~km~s$^{-1}$\ which is roughly the sound speed in the corona. When impacts fell into a previously created splash, they  dissipated in the corona creating hotter impact emission than in the splash because the falling material was shock heated by the back-flowing high-pressure splash. \citet{Reale14} suggested that the shocked falling plasma may explain the redshifted \ion{C}{iv}\ emission seen in accretion flows. Only the slow, large impact stream produced 1600~\AA\ emission in the falling plasma.  Most the observed fragments have lower column density and larger size than the spherical fragments in the \citet{Reale14} simulations which suggests that they were significantly less dense. This may be the reason that no 1600~\AA\ emission was observed at the head of the falling fragments when they hit a rising splash.

For impacts with velocity greater than about 300~km~s$^{-1}$\,
the biggest influence on the impacts' I$_{UV}$/I$_{EUV}$  was the level of ionisation in the falling fragment since this determined whether the fragment could penetrate to the chromosphere or was trapped and dissipated in the coronal magnetic field. If the fragments reach the chromosphere as predicted by HD simulations \citep{Sacco10, Reale13} then  UV emission which is only weakly absorbed by overlying cool material, should have been bright at all impacts sites. But in situations where the ionisation of the infalling fragment was high, we found only weak UV emission.
The I$_{UV}$/I$_{EUV}$ decreased with impact velocity which suggests that the ratio is more influenced by impact kinetic energy rather than absorbing column.
Therefore, we think that if the solar impacts are a good proxy for stellar accretion flows, absorption of EUV and X-ray emission at the impact site is not a likely explanation for the lower accretion rates obtained by analysis of X-ray compared with UV, visible and infrared observations.
 Rather, we agree with \citet{Bonito14} that a more plausible explanation for the absorption of X-rays is cool  material in the unperturbed accretion stream above the impact site.
Modelling of both the  UV and EUV emission would give greater understanding of the fragment and impact dynamics.

\begin{acknowledgements}
We are indebted to the SDO and STEREO teams for providing the high
resolution data.
\end{acknowledgements}

\bibliographystyle{aa}