Spatiotemporal Fusion in 3D CNNs: A Probabilistic View

Despite the success in still image recognition, deep neural networks for spatiotemporal signal tasks (such as human action recognition in videos) still suffers from low efficacy and inefficiency over the past years. Recently, human experts have put more efforts into analyzing the importance of different components in 3D convolutional neural networks (3D CNNs) to design more powerful spatiotemporal learning backbones. Among many others, spatiotemporal fusion is one of the essentials. It controls how spatial and temporal signals are extracted at each layer during inference. Previous attempts usually start by ad-hoc designs that empirically combine certain convolutions and then draw conclusions based on the performance obtained by training the corresponding networks. These methods only support network-level analysis on limited number of fusion strategies. In this paper, we propose to convert the spatiotemporal fusion strategies into a probability space, which allows us to perform network-level evaluations of various fusion strategies without having to train them separately. Besides, we can also obtain fine-grained numerical information such as layer-level preference on spatiotemporal fusion within the probability space. Our approach greatly boosts the efficiency of analyzing spatiotemporal fusion. Based on the probability space, we further generate new fusion strategies which achieve the state-of-the-art performance on four well-known action recognition datasets.


Introduction
For numerous video applications, such as action recognition [31,43,33], video annotation [41] and person reidentification [37], spatiotemporal fusion is an integral component. Taking action recognition as an example, the spatiotemporal fusion in deep networks can be roughly classified into two main categories: fusion/ensemble of two modalities (i.e, spatial semantics in RGB and temporal dynamics in optical flow) in a two-stream architecture [31,23] and fusion of spatial and temporal clues in single-stream 3D  [29] & MiCT [43] Top Heavy [36] Bottom Heavy [36] R(2+1)D [27] LGD [21] Network-level Observations The proposed probabilistic approach. We propose to analyze the spatiotemporal fusion by finding a probability space where each individual fusion strategy is considered as a random event with a meaningful probability. We first introduce a template network based on basic fusion units to support a variety of fusion strategies. We then embed all possible fusion strategies into the probability space defined by the posteriori distribution over fusion strategy. As a result, various fusion strategies can be evaluated/analyzed without separate network training to obtain network-level observations and layer-level preference. Here S, ST and S + ST are basic fusion units instantiated by 2D, 3D, and a mix of 2D/3D convolutions, respectively.

Basic units (a) Existing Approaches (ad-hoc designs) (b) Our Probabilistic Approach
CNNs [29,43]. In this paper, we focus on the latter.
Conceptually, 3D CNNs are capable of learning spatiotemporal features responding to both appearance and movement in videos. Recent research also shows that pure 3D CNNs can outperform 2D ones on large scale benchmarks [7]. However, we still observe noticeable variations in accuracy by employing additional spatial or temporal feature learning explicitly in 3D CNNs. As shown at the top of Fig. 1, different spatiotemporal fusion strategies [29,21,36,27,43] have been studied and recommended for action recognition. They explore spatial semantics and temporal dynamics in videos through the combinations of different types of basic convolution unit at each layer in 3D CNNs. Though with different conclusions, these works have one thing in common -they draw conclusions based on the performance of networks employing one or several fusion strategies designed empirically [27,36,26]. Each fusion strategy is predefined, fixed, and evaluated in each individual network, leading to a network-level analysis of fusion strategies. Due to the proliferation of combinations and prohibitive computational costs, it is difficult for existing solutions to simulate a great number of fusion strategies for evaluation, nor can they support fine-grained and layerlevel analysis.
In this paper, we propose to analyze the spatiotemporal fusion in 3D CNNs from a different point of view, i.e., a probabilistic one. To be specific, we make the spatiotemporal fusion analysis an optimization problem, aiming to find a probability space where each individual fusion strategy is treated as a random event and assigned with a meaningful probability. The probability space will be constructed to meet the following requirements. First, the effectiveness of each spatiotemporal fusion strategy (event) can be easily derived from the probability space, so that we can analyze all the fusion strategies based on the derived effectiveness rather than training each network defined by each fusion strategy. Second, from the probability which is closely correlated with the performance of each fusion strategy, it should be able to deduce the layer-level metrics of the fusion efficiencies, making it possible to perform layer-level, fine grained analysis of fusion strategies. Now, the question becomes how we build this probability space.
Recent research shows that optimizing a neural network with dropout (applied on every channel of kernel weights) is mathematically equivalent to the approximation to the posteriori distribution over the network weights [5] and architectures [42]. It inspires us to construct the probability space via dropout in 3D CNNs. In our approach, we propose to first design a template network based on basic fusion units. We define the basic unit as different forms of spatiotemporal convolutions in 3D CNNs, e.g., spatial, spatiotemporal, and spatial+spatiotemporal convolutions, as illustrated in Fig. 1. The probability space can then be defined by the posteriori distribution on different sub-networks (fusion strategies) along with their associated kernel weights in the template network. Note that in our fusion analysis, we need to approximate posteriori distribution on basic fusion units rather than on kernels as in [5]. Therefore, based on the variational Dropout [15] and Drop-Path [16], we present a Variational DropPath (v-DropPath) by using a variational distribution which factorizes over the probability of the dropout operations that are applied on every basic fusion unit. Then the posterior distribution can be inferred by minimizing the Kullback-Leibler (KL) divergence between the variational distribution and the posteriori distribution, which proves to be equivalent to optimizing the template network with the v-DropPath. We will show that such a probability space fully satisfies the two requirements mentioned above in Section 3.1 and 3.3.
Once we obtain such distribution, we acquire a variety of fusion strategies from the template network by executing v-DropPath w.r.t. its optimized drop probability. Those fusion strategies can be directly evaluated without training. In addition, we also utilize the derived probability space to provide numerical measurements for layer-level spatiotemporal fusion preference.
Experimental results show that our proposed probabilistic approach can produce very competitive fusion strategies to obtain state-of-the-art results on four widely used databases on action recognition. It also provides general and practical hints on the spatiotemporal fusion that can be applied to 3D networks with different backbones, such as ResNet [9], MobileNet [22], ResNeXt [35] and DenseNet [10], and achieve good performance.
In summary, our work has four main contributions: 1. We are the first to investigate the spatiotemporal fusion in 3D CNNs from a probabilistic view. Our proposed probabilistic approach enables a highly efficient and effective analysis on varieties of spatiotemporal fusion strategies. The layer-level fine-grained numerical analysis on spatiotemporal fusion also becomes possible.
2. We propose the Variational DropPath to construct the desired probability space in an end-to-end fashion.
3. New spatiotemporal fusion strategies are constructed based on the probability space and achieve the state-ofthe-art performance on four well-known action recognition datasets.
4. We also show that the hints on spatiotemporal fusion obtained from the probability space are generic and suitable for benefiting different backbone networks.

Related Work
Spatiotemporal fusion has been widely investigated in various tasks and frameworks [21,18,44]. In this paper, we choose one of its typical scenarios, i.e., action recognition, to discuss the related work. We further roughly group the spatiotemporal fusion methods for action recognition into two categories: fusion in two-stream (RGB and optical flow) CNNs and fusion in single 3D CNNs. Due to space limitations, here we review only the most related workspatiotemporal fusion in single 3D CNNs.
There exists a considerable body of literature on spatiotemporal fusion in 3D CNNs. Some of these works show that the efficiency of 3D CNNs can be improved by empirically decoupling the spatiotemporal feature learning in a specific way [29,3,21,43,4,45,2,13]. For example, Wang et al. [29] present the fusion method that utilizes 3D convolution with square-pooling to capture the appearance-independent relation and 2D convolution to capture the static appearance information. These two features are then concatenated and fed into a 1x1 convolution to form new spatiotemporal features. Results show that this fusion method can significantly improve the performance with model size and FLOPs similar to the original 3D architecture. Feichtenhofer et al. [3] also propose a fusion approach which combines the 3D and 2D CNNs. They use 2D convolution (with more channels) to capture rich spatial semantics from individual frames at lower frame rate, and factorized 3D convolution to extract motion information from frames at high temporal resolution which is fused by lateral connection to the 2D semantics. Zhou et al. [43] present a mixed 3D/2D convolutional tube, MiCT-block, which integrates 2D CNNs with 3D convolution via both concatenated and residual connections in 3D CNNs. It encourages each 3D convolution in 3D network to extract temporal residual information by adding its outputs to the spatial semantic features captured by 2D convolutions.
Instead of presenting one specific fusion strategy, some other work investigates the spatiotemporal fusion in 3D CNNs by evaluating a group of pre-defined fusion methods [27,36,26]. For instance, four fusion methods are constructed, trained and evaluated individually in [36] including bottom-heavy-I3D, top-heavy-I3D as shown in Fig.1. More fusions such as mixed convolutions and reversed mixed convolutions are investigated in a similar way in [27,26]. Although with meaningful observations, these methods can only analyze a limited number of fusion strategies, provide network-level hints, and suffer from huge computational costs.
In contrast to all the above presented methods, in this paper, we propose to construct a probabilistic space that encodes all possible spatiotemporal fusion strategies under a predefined network topology. It not only provides a much more efficient way to analyze a variety of fusion strategies without training them individually, but also facilitates the fine-grained numerical analysis on the spatiotemporal fusion in 3D CNNs.

Spatiotemporal Fusion in Probability Space
We observe that a fusion strategy in an L-layer 3D CNN can be expressed with a set of triplets {(l, v, u)} L , where l (1 ≤ l ≤ L) is the layer index, v is a binary vector of length l − 1 denoting the features from which layer/layers will be used, and u (u ∈ U ) denotes the basic fusion units  As concrete examples, existing fusion strategies can be well represented by the triplets, e.g., top-heavy structure [36], SMART-block [29]/MiCT-block [43] and global diffusion structure [21], as shown in Fig. 2, respectively.

The Probability Space
As discussed in the introduction, we construct the probability space with the posteriori distribution over different fusion strategies along with their associated kernel weights. In the probability space, M = {(l, v, u)} L should be a random event. We also define W M to be the kernel weight of the corresponding strategy M, which is also a random event in such space. Therefore, we give the full definition of the probability space denoted with (Ω, B, F), where • Probability measure function F. We use the posteriori distribution to assign probabilities to the events as where D = {X, Y } indicates the data samples X and ground-truth label Y used for training.
In this probability space, various fusion strategies and their associated kernel weights are sampled as pairs and we can make direct evaluation without training. The overall performance of one strategy can be obtained only at the cost of network testing. Therefore, the first requirement for the probability space is satisfied. Now, The core of embedding spatiotemporal fusion strategies into such probability space is to derive the measure function defined in Eq. 1.

Embedding via Variational DropPath
It is hard to obtain the posteriori distribution in Eq. (1), as usual. In our approach, we present a variational Bayesian method to approximate it. We first build a template network based on the basic fusion units that will be studied in the spatiotemporal fusion. For instance, we can design a densely connected 3D CNN with U = {S, ST, S + ST }, as shown in Fig. 1. We then incorporate a variational distribution that factorizes over every basic unit in the template network, which are re-parameterized with kernel weight multiplying a dropout rate. We further propose the v-DropPath inspired by [15,5,42] that enables us to minimize the KL distance between the variational distribution and the posteriori distribution via training the template network. More details will be presented below.
By incorporating the template network, the posterior distribution in Eq. (1) can be converted to where • is the Hadamard product (with broadcasting), M ∈ (0, 1) L×L×3 is a binary random matrix and M(l, i, u) = 1/0 denotes that the feature from the layer i and the fusion unit u is enabled/disabled at layer l in the template network, respectively. W T ∈ R L×L×3×V denotes the random weight matrix of the template network, where we use V to denote kernel shape for simplicity. This conversion actually integrates the kernel weights into fusion strategies. Since we can fully recover the M from the embedded version M • W T (it is because the kernel is defined in real number field, the probability of being zero for every element can be ignored), the first requirement is still satisfied. We then approximate the posteriori distribution by minimizing the KL divergence where Q(·) denotes a variational distribution. Instead of factorizing the variational distribution over convolution channels as in [5], we factorize Q( M • W T ) over fusion units in each layer as By re-parameterising the q( M(l, i, u) · W T (l, i, s, :)) with l,i,u · w l,i,u , where l,i,u ∼ Bernoulli(p l,i,u ) and w l,i,u is the deterministic weight matrix associated with the random weight matrix W T (l, i, u, :), minimizing Eq. 3 is approximately equivalent to minimizing where k l,i,u is a pre-defined length-scale prior and N is the number of training samples. The gradients w.r.t. the Bernoulli parameters p are computed through Gumbel-Softmax [12]. For step-by-step proofs of Eq. 5, please refer to our supplementary material. Eq. 5 reveals that approximating the posteriori distribution can be achieved by training the template 3D network where each spatial or temporal convolutions is masked by a logit subject to Bernoulli distribution with probability p. It is exactly the drop-path proposed in [16]. But here both the network weight and the drop rate need to be optimized. We adopt Gumbel-Softmax for the indifferentiable Bernoulli distribution to enable a gradient-based solution. Please find more details in supplementary material.

Spatiotemporal Fusion
Once the probability space defined by the posteriori distribution is obtained, we can investigate the spatiotemporal fusion very efficiently at both the network and layer levels.
Network-level. Conventionally, the network-level fusion strategies are explored by training and evaluating each individual network defined by one fusion strategy. In our scheme, we successfully eliminate the individual training and evaluation by using the embedded probability space. We study the fusion strategies by directly sampling a group of strategy and kernel weight pairs It is doable since each (M, W M ) t can be fully recovered from the embedded version M • W T . The above sample process is equivalent to randomly choosing l,i,u based on the Bernoulli distribution with the optimized p l,i,u as defined in Eq. 5, which is further equivalent to randomly dropping some paths in the template network. The effectiveness of each fusion strategy can then be easily derived from the test performance on a validation dataset. Because the sampling and evaluation are light-weight, our approach can greatly expand both the number and form of fusion strategies for analysis. Layer-level. The network-level analysis shows the overall effectiveness of different spatiotemporal fusion strategies, but rarely reveals the importance of the fusion strategies at each layer. Interestingly, numerical metrics for such fine-grained, layer-level information are also achievable in our approach. Recall that we factorize the variational distribution in Eq. 4 over different fusion strategies using the reparametrisation trick [15]. We thus can deduce the marginal probability of fusion unit at each layer as Please refer to supplementary material for detailed derivation. Eq. 7 suggests that the marginal distribution of a spa- tiotemporal fusion strategy can be retrieved from the optimized dropout probability. It indicates the probability of using a fusion unit among all the possible networks that can interpret the given dataset well and satisfy prior constrains (sparsity in our case). We propose using this number as the indicator of the layer-level spatiotemporal preference. Therefore, the second requirement on the probability space is met, too.

Experiments
In this section, we will verify the effectiveness of our probabilistic approach from three aspects. Four action recognition databases are used in the experiments. After the description of experimental setups, we will first show the performance of the fusion strategies obtained by our approach in comparison with those of state-of-the-arts. Then several main observations are provided based on the analysis of different fusion strategies generated from our probability space. At last, we verify the robustness of the obtained spatiotemporal fusion strategies on different backbone networks.

Experimental Setups
Template Network. Fig. 3 sketches the basic structure of the template network designed for our approach. The template network is a densely connect one that comprises of mixed 2D and 3D convolutions. Here we choose U = {S, ST, S + ST } so that the fusion units explored in our approach are conceptually included in most of other fusion methods for fair comparison. We also factorize each 3D convolution with a 1D convolution and a 2D convolution, and use element-wise summation to fuse the 2D and 3D convolutions for simplicity. Besides, we add several transition blocks to reduce the dimension of features and the total number of layers is set to be 121 as in [10]. We put more details of the template network in the supplementary material. In practice, we share the variational probability on the variables i defined in Section. 3 for computational efficiency.
Datasets. We apply the proposed scheme on four well-known action recognition datasets, i.e., Something-Something(V1&V2) [6], Kinetics400 [14] and UCF101 [24]. Something V1/V2 consist of around 86k/169k videos for training and 12k/25k videos for validation, respectively. Video clips in these two datasets are first-person videos with 174 categories that focus more on temporal modelling. Ki-netics400 is a large-scale action recognition database which provides around 240k training samples and 20k validation samples from 400 classes. UCF101 has around 9k and 3.7k videos for training and validation. They are categorized into 101 classes. Both the Kinetics400 and the UCF101 contain complex scene and object content in video clips and have large temporal redundancy.
Training. As mentioned before, we approximate the posteriori distribution of different fusion strategies by training the template network with v-DropPath. We initialize the drop rate of each convolution operation as 0.1. We train the template network with 90 epochs for Something-Something(V1&V2)/UCF101 and 110 epochs for Kinet-ics400, respectively. The batch size is 64 for Kinetics and 32 for the others. The initial learning rates are 0.005 (Something&UCF) and 0.01 (Kinetics) and we decay them by multiplying 0.1 at 40th, 60th, 80th epochs for Something/UCF and 40th, 80th epochs for Kinetics. The video frames are all resized to 256 (short edge) and randomly cropped to 224x224. The length-scale prior k in Eq. 5 is determined by grid search, where k = 250 for SomethingV1, k = 10 for Kinetics400 and k = 50 for the rest. In practice, warmup is used before training the template network with v-DropPath, i.e., removing all the v-DropPath operations and training the template network from scratch for 50 epochs. All experiments are conducted with distributed settings and synchronized Batch Normalization [11] on multiple  v100 GPUs with 32G memory.
Sampling and Inference. We derive various spatiotemporal fusion strategies from the probability space through sampling different combinations of spatiotemporal convolutions w.r.t. the drop probability of v-DropPath. The sampled strategies are directly evaluated on validation dataset.
During the inference of each spatiotemporal fusion strategy, we resize the short edge of the input video frames to 256 and make center crop to get a 256x256 region. We uniformly sample multiple clips in a video and average the prediction scores to obtain video level predictions. The number of clips varies from dataset to dataset and will be discussed along with the results.

Ablation Study
In order to demonstrate the effectiveness of our probability space, for each dataset, we sample 100 fusion strategies from the constructed space and choose the best one according to the performance on the held-out validation dataset. We denote the best strategy as 'Optimized'(Opt). We then compare it with its counter-part strategies 'S','ST', and 'S+ST' in Fig. 2, which are designed with one fixed corresponding basic fusion unit, S, ST , or S + ST , at all layers, respectively. It can be observed that our probability space can generate better strategies on all the dataset. Our 'Opt' method even outperforms its counter-part 'ST+S' which has more parameters and higher FLOPs.

Comparisons with the State-of-the-arts
Our proposed method analyzes the spatiotemporal fusion strategies from the perspective of the probability. It not only enables an advance analysis approach, but also achieves high-performance spatiotemporal fusion strategies. In this section, we compare the strategies drawn from the probability space with state-of-the-art fusion methods on four action recognition datasets. Our approach has very compet-itive performance, i.e., performing the best among all the schemes on three of these datasets and obtaining the second best on UCF101, even though some of the compared results are achieved with better backbones and/or with extra modules such as non-local, motion encoder, or gated functions.
Something-Something V1&V2. Table. 1 exhibits the performance of different spatiotemporal fusion methods on Something V1 dataset. It shows that our approach leads to the fusion strategy that outperforms all the other schemes including so far the most advanced 3D network S3D by a large margin with 50% fewer FLOPs and frames. Surprisingly, it performs even better than those methods with carefully designed functional modules, e.g., STM employs a channel-wise motion module to explicitly encode motion information, and Non-local I3D + GCN explicitly incorporates the object semantics with graphs. Similar results can be observed on the recently released dataset Something V2. As shown in Table. 3, our fusion strategies significantly outperform the conventional I3D solutions and its bottomheavy and top-heavy counterparts which incorporates 3D convolutions in bottom layers and top layers, respectively. We employ ImageNet pre-training for both datasets and our fusion strategy can achieve higher accuracy than those pretrained on the large-scale dataset Kinetics such as ECO.
Kinetics400. Accuracy achieved by different fusion methods on Kinetics400 are reported in Table 4. In order to make apple-to-apple comparisons, all methods are trained from scratch. It can be observed that our configuration of spatiotemporal fusion outperforms the second best R(2+1)D on Top1 accuracy with 97% fewer FLOPs , where R(2+1)D is a 3D network that uses ResNet34 as backbone. Compared with R(2+1)D, we actually utilize more spatial convolutions in the shallow layers as can be viewed in Fig. 4.  [40] 48.8% 77.6% Two-stream TRN [40] 55.5% 83.1% TSM(ImageNet+ Kinetics400) [17] 59.1% 85.6% TSM dual attention [34] 55.5% 82.0% I3D-ResNet50 [34] 43.8% 73.2% 2D-3D-CNN w/ LSTM [19] 51.6% -Ours (ImageNet) 62.9% 88.0% UCF101. Since UCF101 has only 9k training videos, we make evaluations with the ImageNet pre-training and Kinetics400 pre-training, respectively. When incorporating ImageNet pre-training only, our fusion strategy produces the most advanced results, which has 1.5% higher accuracy than the I3D that performs pure spatiotemporal fusions. When using Kinetics400 as pre-training dataset, the overall performance is still state-of-the-art. Please note that we do not employ any extra functional module here, so the performance is slightly worse (0.3%) than the most advanced 3D networks S3D-G that incorporates attention mechanism.

Observations
We visualize the strategies derived from the probability space that have the highest accuracy on the test datasets in Fig. 4. We also illustrate the marginal probability of using different basic units in each layer based on Eq. 7. The amplitude of bars in blue, green and yellow indicates the marginal probability of using the units S, ST and S + ST in each layer, respectively. The dotted-line in orange shows the selected layer-level basic fusion units that produce the best accuracy. From these figures, we observe that Observation I. As indicated by the colored bars, the unit S + ST has higher marginal probability in the lower-level feature learning compared with the other two units. The dotted line in orange also shows a similar trend. The S + ST unit has the highest percentage of total usage in all the fusion units, especially in the lower layers. It suggests that a proper spatiotemporal fusion strategy can be designed based on S + ST units, particularly in lower layers. Observation II. More ST units are preferred in higher layers as there is a higher marginal probability on the ST unit in the higher-level feature learning (except on UCF101 which will be discussed below).
Observation III. Additional S units could be beneficial when scene semantics are complex. For instance, Kinet-ics400/UCF101 contain videos in the wild with 400/101 different categories, respectively. The scene content is more complex than that in the first-person videos in Something-Something. By comparing Fig. 4 (c) and (d) with the others, it shows that more S or S + ST units are selected.

Generalization
We further discuss the generalization of our observations as well as the selected fusion strategies. We extend our fusion strategies to three backbone networks including ResNet50 [9], and ResNeXt50/ResNeXt101 [35]. They differ from each other in terms of topology, parameter size and FLOPs. We report clip-level accuracy on Something V1 for quick comparison. Please find more results and discussions on other backbone networks in the supplementary material.    We employ four different fusion strategies 'Opt', 'S+ST', 'S' and 'ST' as defined in Section 4.2 for comparison. Note that here the fusion strategy denoted by Opt' is not optimized using our probabilities approach but straightforwardly designed based on our observations. Specifically, we construct the fusion strategy 'Opt' according to Fig. 4  (a) and (b), which uses S + ST unit in both the first half and the last three layers, and ST unit in the remaining layers. As shown in Table. 6, the fusion method 'Opt' performs the best among all the evaluated fusion strategies.

Conclusion and Discussion
In this paper, we convert the problem of analyzing spatiotemporal fusion in 3D CNNs into an optimization prob-lem which aims to embed all possible fusion strategies into the probability space defined by the posteriori distribution on each fusion strategy along with its associated kernel weights. Such probability space enables us to investigate spatiotemporal fusion from a probabilistic view, where various fusion strategies are evaluated and analyzed without the needs of individual network training. The numerical measurements on layer-level fusion preference are available. By further proposing the Variational DropPath, the optimization problem can be efficiently solved via training a template network. Experimental results on four action recognition databases demonstrate the effectiveness of our approach. We also observe several useful hints with our probabilistic approach which can be extended to design high performance fusion strategies on different backbones.
Since log(p(D)) is the constant evidence , minimizing Eq. S2 is equivalent to minimizing the negative evidence lower )]. This lower bound can be further rewritten as We assume the Q( M • W T ) to have a factorized form and we factorize it over fusion units at each layer as We further re-parameterize the random kernel weight W T with a deterministic kernel weight multiplying a random variable that subjects to some distribution. Take a univariate Gaussian distribution x ∼ q θ (x) = N (µ, σ) as an example, its re-parametrization can be x = g(θ, ) = µ+σ with ∼ N (0, 1) a parameter-free random variable, where µ and σ are the variational parameters θ. Following [3], we choose the Bernoulli distribution for the re-parametrization, which leads to M(l, i, u) · W T (l, i, u, :) = M(l, i, u) · (w l,i,u · z l,i,u ), where z l,i,u ∼ Bernoulli( p l,i,u ).
Here w l,i,u is the deterministic weight matrix associated with the random weight matrix W T (l, i, u, :). Since each M(l, i, u) controls the utilization of the fusion units u with binary values 1 and 0, it also subjects to Bernoulli distribution, denoted as Bernoulli( p l,i,u ). Therefore, we use a new Bernoulli distribution to replace the original two as where l,i,u ∼ Bernoulli(p l,i,u ), Now, we replace the M(l, i, u) · W T (l, i, u, :) with its reparameterized form in the second term in Eq. S4, which leads to We use Monte Carlo estimation to approximate the integral term in the above equation, where t indicates t-th sampling. Combining Eq. S4 with Eq. S8, our objective function is converted to minimizing The second term in Eq. S9 is equivalent to the inference of a neural network with DropPath on dataset {x i , y i }. However, the derivatives w.r.t. the Eq. S9 is still difficult to compute because of the KL term.
Here we leverage the Proposition 4 in [2] which proves that given fixed M, C ∈ N, a probability vector p = (p 1 , p 2 , ..., p C ), and Σ h ∈ R M ×M diagonal positivedefinite for h = 1, 2, ..., C, with the elements of each Σ h not dependent on M, and let q(x) = C h=1 p h N (x; µ h , Σ h ) be a mixture of Gaussians with N components, where µ h ∈ R M , if assuming that µ h −µ j ∼ N (0, I), the KL divergence between q(x) and p(x) = N (0, I k ) can be approximated as (S10) Actually, Eq. S7 suggests that and we can approximate each q( M(l,i,u)·W T (l,i,u,:)| l,i,u ) as a narrow Gaussian with a small standard deviation Σ = σ 2 I. Therefore, q( M(l,i,u)·W T (l,i,u,:)) = q( M(l,i,u)·W T (l,i,u,:)| l,i,u )p( )d is also a mixture of two Gaussians with small standard deviations (similar with the one in the above proposition), where one component fixed at zero and another one fixed at w l,i,u . If we assume the prior of u to be 'S+ST' at all layers in the template network and the prior of kernel weight to be Guassian distribution N (w l,i,u ; 0, I/(k l,i,u ) 2 ), where k l,i,u is a prior length scale, the prior distribution of each M (l, i, u) · W T (l, i, u, :) is still Guassian. Given the Eq. S5 and the proposition Eq. S10, it can be easily derived that (S12) In addition to the variational parameters w l,i,u , the optimal distribution of random variable which encodes the network architecture information also needs to be found. In order to facilitate a gradient based solution, we employ Gumbel-softmax to relax the discrete Bernoulli distribution to continuous space. More specifically, instead of drawing l,i,u w.r.t. the Bernoulli(p l,i,u ), we deterministically draw the l,i,u with s.t. r 1 , r 2 ∼ U nif orm(0, 1). (S13) Under this re-parametrisation, the distribution of l,i,u is smooth for τ > 0 and p( l,i,u ) → Bernoulli(p l,i,u ) as τ approaches 0 and. Therefore, we have well-defined gradients w.r.t. the probability p l,i,u by using a small τ . Combining Eq. S12 and S13, we can obtain the gradients presented by the Eq. (5) in the paper.

Equation 7 in the paper
We use v 0 to denote index (l 0 , i 0 , u 0 ) for simplicity. It is straightforward to derive that  Figure 1: Architecture of the template network used in our method. The notion 'x6' indicates that the basic module (a single grey box) is repeated by six times. The spatial stride size used for all the convolution and pooling operations is one and two, respectively. The temporal stride size is always set to be one.
Once the variational distribution Q is available, it is easy to derive that According to Eq. S7, S14 and S15, we have (S16) In practice we optimize p l,i,u = p l,i,u · p l,i,u as a whole, and the p l,i,u and p l,i,u are initialized with the same value, therefore, we have Hereby, we have the posterior probability of fusion unit at each layer, which can be used as numerical measurements for the layer-level importance of the fusion units as described in the paper.

Template Network
We visualize the complete structure of the template network in Fig. 1. As can be viewed in the figure, each layer in the template network contains three convolution operations which is used to derive the three basic fusion units as illustrated in the Fig. 3 in the paper. The feature map output from each layer will be used as input for all the succeeding layers, which forms a densely-connected 3D network. Similar to [6], we also insert reduction block for memory efficiency.  Fig. 1 in this supplementary material). It illustrates how the basic units are derived from the template network. Each grey block refers to a module that contains three convolutions as well as three drop-path operations (D1, D2, and D3). By activating each drop-path operation w.r.t. its probability, we can derive three basic fusion units, i.e., S (activate D3 only), ST (activate D2 only) and S+ST (no activation), plus one residual operation (activate D1). This design helps us to construct the probability space via network training with DropPath, as described in the Section 3.2 in the paper.

More Results on Generalization
In order to justify that the observations obtained from the probability space can generalize, we construct new spatiotemporal strategies based on the observations discussed in Section. 4.4 in the paper, but with five very different backbone networks. They are DenseNet121 [6], ResNet50 [5], MobileNetV2 [10], ResNeXt50 [11], and ResNeXt101 [11], respectively. They differ from each other in terms of topology, parameter size and FLOPs. We inflate them into 3D CNNs with the same module visualized in our template network. We compare four different fusion strategies on each backbone, i.e., optimized(Opt), fused(S+ST), spatial(S) and spatiotemporal(ST). 'Opt' means we follow the observations to design the fusion strategies (except for Densenet we directly use the best one sampled from the probability space). Please note that we can only roughly follow the observations because the network topology varies from backbone to backbone. 'S+ST' indicates that we employ S+ST convolution all the way. 'Spatial' and 'spatiotemporal' indicate using spatial convolutions and spatiotemporal convolutions all the way, respectively. Please also note that there are 3D poolings existing in the 'spatial' mode, so it is not pure 2D network. We report clip-level performance in this section for quick comparison.
On Something-Something V1 and Something-Something V2, we follow the Observation I and II to construct the strategy 'Opt' where fused convolutions are used for the first half and the last three layers of network, and spatiotemporal convolutions are applied on the remaining layers. As can be viewed from the Fig.  2(b) and 3(b), although backbones are quite different, all the networks perform better with the optimized fusion strategy. One exception is MobileNetV2, where the 'Opt' strategy is slightly worse than the fused mode. We think the reason is that MobileNetV2 is too small to fit this large dataset and any kind of bonus on the parameter size would help improve the performance greatly. The 'S+ST' mode contains 7 more 2D convolutions than the 'Opt' mode in the MobileNetV2 backbone.
Similar results can be observed on UCF101, where the 'Opt' strategy makes all the four backbone networks outperform their counterparts, which is consistent with the Observation III. Please note that the optimized strategy is equivalent to the fused strategy on UCF101 according to the Fig.  4.
On Kinetics400, we roughly follow the Observation I II and III as well as the patterns in Fig. 5(a) to use the S+ST convolutions and S convolutions periodically in the first two third of network and ST convolutions in the remaining layers for the strategy 'Opt'. We can see from the Fig. 5  We also implement a small experiment to illustrate the learned probability space can help the spatiotemporal fusion strategy capture the character of the data. More specifically, we reduce the temporal resolution of input video clips by employing temporal pooling with a stride of 2 and window size of 3. We draw the corresponding marginal probability and best-sampled fusion strategy on Something V1 in Fig.  6. It can be easily observed that the layer-level preference changes accordingly. There are more spatial convolutions and less spatiotemporal convolutions utilized in the bestsampled strategy when compared with Fig. 2 in which no additional temporal pooling is used.

More Discussions on NAS
We believe that our proposed algorithm can be extended for NAS. But in this paper, it is specially designed for analyzing spatiotemporal fusion from a probabilistic view. To ensure the whole scheme is theoretically sound and correct, we have to construct a valid probability space to facilitate efficient and effective analysis with numerical measurements. To our best knowledge, no published NAS papers manage to formulate and construct such probability space.
Additionally, existing NAS methods have shortcoming in spatiotemporal analysis. For example, differentiable NAS methods such as DARTS [7] assign a scalar unit (regularized by softmax function) to each operation and jointly optimize weight and the unit in an alternative fashion. Only     one single architecture is derived by selecting the operation that has the largest activation in each layer and the weight needs to be trained from scratch to produce final performance. It can not properly sample a group of diverse ar-     Table: The performance of Densenet121 constructed with the spatiotemporal hints and their counter-parts Figure 6: Spatiotemporal fusion hints obtained from the probability space on Something-Something V1 with less temporal resolution.
chitectures to be directly evaluated for further fusion analysis. Sampling-based NAS methods such as [1,4] randomly sample different sub-networks and naively inherit the kernel weights from the template network. A group of candidates can be obtained in this way. However, the inherited weight may not match the sampled architectures and thus the per-formance of each sub-network does not match the ideal performance as well, which suggests it can not be used to facilitate an effective analysis on spatiotemporal fusion in 3D CNNs. As another example, Evolution-based NAS methods iteratively derive new child architectures from population. Due to the high complexity of evolution algorithm, it cannot facilitate fine-grained layer-level spatiotemporal analysis. For instance, evolved module is repeated several times to form a homogeneous network which has few variations in terms of network-level structure diversity [8], and only connectivity among several predefined modules can be evolved [9].