Coupled Recurrent Network (CRN)
Lin Sun Kui Jia Yuejia Shen Silvio Savarese Dit Yan Yeung Bertram E. Shi

0
=2
=3



1,2,3]Lin Sun
4]Kui Jia
4]Yuejia Shen
2]Silvio Savarese
1]Dit Yan Yeung
1]Bertram E. Shi
[1]The Hong Kong University of Science and Technology
[2]Stanford University 
[3]Samsung Strategy and Innovation Center, USA
[4]South China University of Technology
Coupled Recurrent Network (CRN)
    [
    
===============================





In vision problems, multiple input sources can be used in combination to provide complementary prediction that may be redundant but convey information more effectively. Similarly, many semantic video analysis tasks can benefit from multiple, heterogenous signals. For example, sequences of RGB images and optical flow are usually processed simultaneously to boost the performance of human action recognition in videos. To learn from these heterogenous inputs, existing methods reply two-stream architectures that contain independent, parallel streams of networks. However, two-stream networks do not fully exploit the reciprocal information contained in the multiple signals, let alone exploit it in a recurrent manner. Therefore, we propose, in this paper, a novel recurrent architecture, termed Coupled Recurrent Network (CRN), to deal with multiple input sources. In CRN, the parallel streams of Recurrent Neural Networks (RNNs) are intertwined with each other using Recurrent Interpretation Block (RIB) and Recurrent Adaptation Block (RAB). RIB supports learning of reciprocal representations from multiple signals. RAB makes the features adapted to the next recurrence. Different from the training of typical RNNs which stack the loss at each time step or the last time step,  we propose an effective and efficient training strategy for CRN. Experiments show the efficacy of the proposed CRN. In particular, we achieve the new state-of-the-art on the benchmark datasets of human action recognition and multi-person pose estimation.




§ INTRODUCTION
 


Many computer vision tasks rely on semantic analysis of data in sequential forms. Typical examples include video-based human action recognition <cit.>, image/video captioning <cit.>, speech recognition <cit.> etc. In other cases, tasks of interest might be recast as sequential learning problems, so that their learning objectives can be easily and iteratively achieved. For example, in human pose estimation, the joint locations can be predicted using multi-stage CNNs, the hidden features produced by one stage are used as input for the next stage. This multi-stage scheme for pose estimation can be cast using the recurrent scheme that the hidden output from one time step will be fed into in the next time step for refinement. 

RNNs in which each neuron or unit can use its internal memory to maintain information of the previous input have become the de facto learning models for many computer vision tasks whose observed input data are or can be recast in a sequential form. In these tasks, we might have access to multiple and heterogenous input sources. In general, having multiple input sources working together towards the same goal, can contribute to a more robust algorithm. The input sources each may be more effective at expressing one form or aspect of information than others. This is the reason why the two-stream architectures is widely accepted in human action recognition. RGB frames and corresponding motion inputs (e.g. optical flow or RGB difference) is used as multiple input sources which, separately, will pass through the independent network. The results of the individual models will be combined together to predict the actions happened in the video. 


The independent use of networks on different input sources and fusing the probabilities at the end does not fully exploit the reciprocal information between each other. Moreover, there are few researches on how to exploit reciprocal information between complementary inputs. Therefore, in this paper, we propose an end-to-end architecture, Coupled Recurrent Network (CRN) to investigate how to aggregate the features to better fuse spatial and temporal information and achieve more effective sequential learning from multiple inputs. CRN has two recurrent branches, i.e. branch A and branch B, each of which takes one input source. Different from two-stream architecture which is trained independently, during the learning of CRN, branch A and branch B can `communicate' with each other. In order to well interpret the reciprocal information between them, within CRN, we propose two modules, recurrent interpretation block (RIB) and recurrent adaptation block (RAB). The hidden output from branch B will pass through RIB to extract reciprocal representations for branch A. The extracted reciprocal information will be concatenated with current input sources of branch A and then they will pass through the RAB to obtain the re-mapped input for branch A. The distilling module RIB and remapping module RAB is shared inside the recurrent networks. An illustration of a CRN and its corresponding computational flow is shown in Fig. <ref>. 



The proposed CRN can be generlized to many computer vision tasks which can be understood by two or several input sources. In this paper, we apply the proposed CRN to two human centric problems, i.e. human action recognition and multi-person pose estimation. In human action recognition, a sequence of RGB frames and corresponding motion signals (e.g. optical flows or RGB difference) are the two heterogenous input sources for two branches in CRN. Two cross entropy losses for the same recognition target with identical form are applied at the end of each branch. While in multi-person pose estimation, besides the commmonly used individual body joints, a field of part/joint affinities that characterizes pair-wise relations between body joints <cit.> is used as the additional supervision information. Two different regression losses, one for joint estimation and the other one for vector prediction, are applied at the end of each network. 

The standard procedure for training a RNN is to apply appropriate loss for each input at each time step or only apply at the last time step. However, in the experiments, we find neither of two training strategies work well for CRN. Having supervision at each time step seems to make supervision signals assertive and arbitrary and the whole training becomes numerically unstable, leading to a poor performance; while having supervision only at the last time step makes supervision signal too weak to reach the end and leads to the performance drop on the considered tasks. Therefore, we propose a new training scheme which has a good balance of the supervision strength along time steps. Apart from having the loss at the last time step, we randomly select some previous time steps for supervision. Only the losses at selected time step will contribute to the back-propagation.

Comparative experiments show that our proposed CRN outperforms the baselines by a large margin. CRN sets a new state-of-the-art on benchmark datasets of human action recognition (e.g., HMDB-51 <cit.>, UCF-101 <cit.> and larger dataset, Moments in Time <cit.>) and multi-person pose estimation (e.g., MPII <cit.>). Moreover, since better reciprocal information can be exploited within CRN, using RGB and RGB differences as input sources, CRN can achieve more than 93% accuracy on the UCF-101. Compared to optical flow, RGB difference can be calculated online without burden. Therefore, more than 200 FPS can be achieved for real-time action recognition. We summarize the contributions as below:



  * We propose a novel architecture, Coupled Recurrent Network (CRN), to deal with multiple input sources in a reciprocal and recurrent manner. In order to interpret the representations of each input source for the other in a recurrent approach, recurrent interpretation block (RIB) and recurrent adaptation block (RAB) are proposed. RIB is used to distill the useful information from the output of one branch for the other branch. RAB provides the remapping of two concatenated representations.  


  * Two tasks, i.e. human action recognition and human pose estimation, are investigated and analyzed using proposed CRN. However, our proposed method can be generalized to many computer vision tasks which initially is or can be recast as sequential learning. 


  * A effective and efficient training strategy for CRN is proposed. 


  * The state-of-the-art performance for real-time action recognition can be achieved using CRN which will definitely push forward the application of action recognition in industry. 


  * Extensive quantitative and qualitative evaluations are presented to analyze and verify the effectiveness of our proposed method. We also conduct several ablation studies to validate our core contributions. 




§ RELATED WORKS


In this section, we first provide a brief review of using RNNs for different computer vision tasks, particularly focusing on those algorithms that deal with multiple sources of sequential inputs. Then, we review representative methods for action recognition and human pose estimation. 



 §.§ RNNs for Multiple Sources of Sequential Data


RNN is a class of artificial neural network where connections between nodes form a directed graph along a temporal sequence. Therefore, RNNs natually have advantages in processing sequential data. Among all of recurrent networks, Long Short-Term Memory network, a.k.a LSTM <cit.>, has been observed to be the most effective. On one hand, with sequential inputs, RNNs can produce outputs that are sequences.  Countless learning tasks require dealing with this situation, such as image captioning <cit.>, speech recognition <cit.>, and musicial composition <cit.>. On the other hand, RNNs can provide the prediction from a sequential input, such as handwriting recognition <cit.>, video analysis <cit.> etc.  RNNs also extends its success to sequential data with multiple modalities, e.g. RNNs integrating the video and audio information for video captioning <cit.>. 



 §.§ Human Action Recognition


Neural networks have achieved remarkable results for image-based vision tasks, so, not surprisingly, there have been many recent attempts to extend these successes to videos. However, human actions in videos are three-dimensional (3D) spatio-temporal signals, therefore, typical 2D networks can not handle it well. For that reason, <cit.> proposes to use a two-stream architecture. In this design, spatial network is processing RGB frames and temporal network is processing optical flows, the probability will be averaged or weightdely averaged at the end. Multiple input sources can be used in combination to provide complementary methods that may be redundant but convey information more effectively. Here, RGB images provide apperance representations, while optical flows explicitly capture motion information of videos. Many works follow the two-stream architecture. Tran et al. explore 3D ConvNets <cit.> on realistic and large-scale video datasets, where they try to learn both appearance and motion features with 3D convolution operations. Sun <cit.> propose a factorized spatiotemporal ConvNet using the difference between neighobouring RGB images as additional motion information. Wang et.al <cit.> propose a temporal segment network (TSN), which is based on the idea of long-range temporal structure modeling, for RGB images and optical flows. Besides the original two stream networks, <cit.> applies a gating CNN to combine the information from RGB images and optical flows. <cit.> presents new approaches to combine different sources of knowledge in spatial and temporal networks. They propose feature amplification, where they use an auxiliary, hand-crafted, feature (e.g. optical flow) to perform spatially varying soft-gating on intermediate CNN feature maps. They present a spatially varying multiplicative fusion method for combining multiple CNNs trained on different sources. Even the algorithm is sophisticated, the performance is far from satisfying. <cit.> proposes that spatio-temporal features can be fused at a convolution layer. In <cit.>, the authors extend the residual block to temporal domain within each stream. This modification makes their algorithm similar to RNNs, that is, the information from the current step will be fed into the next step. <cit.> inflates the 2D kernels pre-trained on ImageNet to 3D for videos. 

RNNs is another choice for this task. <cit.> proposed to train video recognition models using LSTMs that capture temporal state dependencies and explicitly model short snippets of ConvNet activations. Ng et al. <cit.> demonstrated that two-stream LSTMs outperform improved dense trajectories (iDT) <cit.> and two-stream CNNs <cit.>, although they need to pre-train their architecture on extra videos. VideoLSTM <cit.> applies convolutional operations within LSTM on sequences of images or feature maps. Additionally, an attention model is stacked on top of the ConvLSTM to further refine the temporal features. Sun et.al <cit.> also propose a lattice LSTM for the long and complex temporal modeling. These two-stream RNNs are all trained independently and combined on the probability level. Even lattice LSTM has joint training on the gates between the two streams, their representations are not completely coupled. 



 §.§ Human Pose Estimation


Human pose estimation considers to use multiple branches to improve the precision as well. Based on the multi-stage work       <cit.> , Cao et al. <cit.> present a real-time pose estimation method by adding a bottom-up representation of association scores via part affinity fields (PAFs). Joint associate network works as backup when joint detection network fails in some context, a robust multi-person pose estimation is provided. 



§ LSTM-BASED COUPLED RECURRENT NETWORK

LSTM is commonly used recurrent architecture, due to its superior performance in many tasks. In this paper, we adopt LSTM, particularly convolutional LSTM (ConvLSTM) <cit.> as the basic unit within CRN. To simplify presentation, in the rest of the paper, the abbreviation LSTM instead of ConvLSTM will be used.


Formally, let's denote {X^a_t,  X^b_t} as two input sources at the time step t, where a and b are the indices of input sources. {X^a_t,  X^b_t} are usually in the form of 2D images or feature representations. A CRN contains two branches, each of which handles one input source. Since two branches in CRN are symmetric, in the following paragraph, we will illustrate branch A step by step to present the whole flow of computation. Starting from the cell memory C^a_t at time t which maintains the information over time within recurrence, 


    C^a_t    = tanh(W_xc∗X̂^a_t + W_hc∗H^a_t-1),

where W_xc and W_hc are, respectively, the weights for the input and hidden states. The symbol ∗ denotes the convolution operation and H^a_t-1 is the hidden output at time step t-1. The concatenation of X_t^a and the interpreted reciprocal hidden representations Ĥ^b_t-1 (described in  <ref>) from branch B will pass through Recurrent Adapted Block (RAB) to obtain the remapped representations X̂^a_t :

    X̂^a_t = R^a_ab(X^a_t ⊕Ĥ^b_t-1) ,

where R^a_ab denotes the functions of RAB. R^a_ab can be one or several convolutional layers which are shared at different time steps. ⊕ is the concatenation operation. 

The input gate i^a_t and forget gate f^a_t at time step t are computed using remapped X̂^a_t as

    i^a_t    = σ(W_xi∗X̂^a_t + W_hi∗H^a_t-1), 
    
    f^a_t    = σ(W_xf∗X̂^a_t + W_hf∗H^a_t-1),

where W_x., W_h. are distinct weights for the input and hidden states in gates. At the end of time step t, we can obtain the updated memory cell C^a_t from the previous memory cell C^a_t-1 and C_t^a:

    C^a_t    = f^a_t∘ C^a_t-1 + i^a_t∘C^a_t,

where `∘' denotes the pixel-wise multiplication, a.k.a, the Hadamard product. The input and forget gate together determine the amount of dynamic information entering/leaving the memory cell. The final hidden output H^a_t  is controlled by output gate o^a_t, 

    o^a_t    = σ(W_xo∗X̂^a_t + W_ho∗H^a_t-1), 
    H^a_t    = o^a_t∘ tanh(C^a_t).

where W_xo, W_ho are distinct weights for the output gate. The branch B is processed in a similar way to obtain H^b_t. Although the features within CRN are coupled, the parameters are initialized independently for each branch. 



 §.§ Adapting to Different Tasks

After obtaining H^a_t, we stack additional transformation layer(s) (e.g., convolutional or linear) to extract final representations Q^a_t, adapting to different supervision tasks. 

    Q^a_t = G^a(H^a_t) ,

where G^a denotes the transformation function(s). 

In the training process, identical or different losses can be applied to each branch. In our case, L^a(·) and L^b(·) are two loss functions, 

    L^a_t    = L^a(Q^a_t, Q_t^*a),  
    
    L^b_t    = L^b(Q^b_t, Q_t^*b),


where Q_t^a* and Q_t^b* is the supervision target for branch A and branch B. Depending on the tasks, L^a(·) and L^b(·) can be the euclidean distance or cross-entropy loss etc. L^a_t and L^b_t are the losses for branch A and B at time step t, respectively. The overall loss for CRN is 

    L = ∑_t∈T (L_t^a + L_t^b) .

where T is a set of selected time steps for supervision. 



 §.§ Extracting Reciprocal Representations
 
The interpreted reciprocal representation Ĥ^b_t-1 can be obtained by passing the hidden output H^b_t-1 from branch B through the Recurrent Interpretation Block (RIB):


    Ĥ^b_t-1 = R^a_ib(H^b_t-1) .

where R^a_ib denotes the RIB. Like RAB, it consists of one or several convolutional layers which hold shared parameters at different time steps. Although the input sources or the supervision targets of two branches are related, directly concatenating the input sources (i.e. X^a_t or X^b_t) with the hidden output (i.e. H^b_t or H^a_t) leads to little improvement. Certain input source can provide a richer context information, the key is how to `borrow' the really useful and complementary information from the other. Therefore, RIB is designed to distill the reciprocal information from each other at every time step. As illustrated in Fig. <ref> (a), in order to effectively extract the reciprocal information, RIB has a similar design like an inception module which has three parallel convolutions with different dilation ratios. To investigate the importance of capacity of RIB, a simplized version, sRIB is provided in Fig. <ref> (b). Experiments present that neither direct concatenation nor sRIB perform as effectively as RIB. It indicates that RIB is a useful design for CRN and when learning transferable knowledge from different input sources, more complicated architecture can distill more reciprocal information. 







§ USE OF CRN FOR SPECIFIC TASKS

We present, in this section, how can our proposed CRNs be applied to two tasks, namely video based human action recognition and image based multi-person pose estimation. Fig. <ref> gives an illustration, where recurrent networks are unrolled to better present the processing flow. The architecture of non-coupled recurrent network (N-CRN) is provided on the left side of the figure as a baseline while the exemplar design of CRN is shown on the right side. N-CRN is just a modified two-stream architecture in which input source concatenate the previous hidden output of itself independently. [X_1^*,X_2^*, ⋯ X_T^*] represents the sequential data for input source a and b where `*' can be a or b. The inputs can be the raw images or feature maps extracted from intermediate layers. [Ĥ_1^*,Ĥ_2^*, ⋯Ĥ_T^*] are interpreted hidden outputs,  [Q_1^*,Q_2^*, ⋯ Q_T^*] represent the final representations from each branch. 

In human action recognition, the input is a sequence of RGB frames and a corresponding sequence of optical flows or RGB differences, and the objective is to classify the input video as one of the action categories. The output of branch A at time step t is Q_t^a∈Ω^1× N, N is the number of categories. Q_t^b has the same dimension and supervision target as Q_t^a. Two cross entropy losses with identical form, are added at the end of each input source. Two probabilities from two branches will be averaged for final prediction.  

In multi-person pose estimation, the input is a sequence of the repeated images, and the objective is to estimate 2D locations of body joints for each person in the image. CRN simultaneously outputs a set of hidden features Q_t^a for heat maps and Q_t^b of for 2D PAFs prediction, which encode the degrees of association between body joints. Q_t^a∈Ω^w× h × M has M feature maps with w × h resolution and each of them corresponds to one body joint at time step t. Q_t^b∈Ω^w× h × 2N has N vectors whose width is w and height is h. Each of vector corresponds to a limb of the human body at the time step t. l2 loss is applied at the end of each branch to minimize the heat maps and PAFs. We follow the greedy relaxation as <cit.> to find the optimal parsing. 



§ EXPERIMENTS


We apply our proposed CRNs to two computer vision tasks. For human action recognition, we use three large scale benchmark datasets:

UCF-101 <cit.> is composed of realistic web videos. It has 101 categories of human actions with more than 13K videos. It has three split settings to separate the dataset into training and testing videos. The mean classification accuracy over these three splits is used for evaluation. 

HMDB-51 <cit.> has a total of 6766 videos organized as 51 distinct action categories. Similar to UCF-101, HMDB-51 has three split settings, and the mean classification accuracy over these three splits is used for evaluation.

Moments in Time <cit.> consists of over 1,000,000 3-second videos corresponding to 339 different verbs depicting an action or activity. Each verb is associated with over 1,000 videos, resulting in a large balanced dataset for learning a basis of dynamical events from videos.  

For human pose estimation, we use a multi-person pose estimation : 

MPII dataset <cit.> consists of 3844 training and 1758 testing groups with crowded, occlusion, scale variation and overlapped people from the real world. We use 3544 images for training, leaving 300 images for validation.



 §.§ Implementation Details

For the action recognition, bninception <cit.> and inceptionv3 <cit.> are used as backbone networks,  features from the last convolutional layers will be fed into the CRN. RGB frames and corresponding optical flows, and RGB frames and RGB differences will be paired and pass through these backbone simultaneously. For each branch in CRN, it is a two-layer LSTM. Within convolutions, all the kernels are 3×3  and the number of hidden feature maps is 512. Batches of ten sequential frames will be fed into system for training. The detailed architecture of RIB is shown in Fig. <ref> (a) and the RAB is just one convolutional layer with kernel size 1 × 1. In the experiments, we find adding more layers for RAB or make it complicated does not help the final performance. The transformation layer G is a global pooling followed by a fully connection layer. The whole system can be trained end-to-end using SGD. The initial learning rate for CRN is 1e-2 and for backbone is 1e-3. The momentum is set to 0.9 and weight decay is 5e-4. Besides the last time step, we randomly select additional one time step from the previous time steps for back-propagation. When testing, we regularly sample four clips and average their probabilities. 

Like <cit.>, we pre-process pose estimation images using VGG-19 <cit.> which is pre-trained on ImageNet <cit.>. The whole system is trained using SGD where the initial learning rate for CRN and backbone is 2e-4 and 5e-5, respectively. For each branch in CRN, a two-layer LSTM with all 7× 7 convolutions is applied. Batches of ten repeated images will be fed into system. We use the same architecture of RIB and RAB as described in the action recognition task.

All the implementations are built using pytorch <cit.>. All the experiments are run on GTX 1080 and evaluated under the same settings as instructed. 





 §.§ Evaluation on Action Recognition

The effect of RIB: In order to verify that distilling reciprocal information from complementary input source is useful, we evaluate CRN using different backbones with/without different RIB architectures. The detailed results are presented in Table <ref>, where 'S-Nets` denotes the spatial networks and 'T-Nets` denotes the temporal networks. The paired input sources can be RGB frames and optical flows or RGB frames and corresponding RGB differences. From the table, 1 vs. 2, 3 and 4 vs. 5, 6 in UCF-101 and HMDB-51, respectively, compared to the results generated with  RIB, directly concatenating the hidden output from the other branch without distilling/RIB performs much worse. And under all the backbones, CRN with RIB can achieve better performance, with about 1% performance gain, over CRN with sRIB. Stronger RIB module makes our CRN distill more appropriate representations to the other branch. All the results indicate that how and how much information is distilled from complementary input source affects the final performance. What is more, the whole procedure is recurrent, iteratively refining the interpreted representations makes our coupled learning generate better representations for each input source. A CRN built on the inceptionv3 backbone and with RIB can achieve the best performance. Without specific notation, in the following paragraph, we adopt this architecture for comparison. Surprisingly, the performance of CRN achieved using one branch on two benchmark datasets is already better than fused performance of some sophisticated two-stream algorithms. 






The effect of training strategy: CRN can not be well trained by adding the loss at the end or at each time step. We evaluate different training strategies on split 1 of HMDB-51 in Table. <ref>, a indicates supervising at the end, b indicates supervising at each time step, c indicates supervising at the end and one previous selected time step and d indicates supervising at the end and two previous selected time steps. As stated in the introduction, c balances the supervision strength within CRN and therefore, better performance can be achieved. Thus, c is accepted for training CRN. 

The effect of coupled recurrence: From Table <ref>, we can see that leveraging the reciprocal information of each other, both spatial and temporal networks can achieve better performance. Since our CRN needs a paired input and generates a paired output, spatial networks listed here are the average of the two CRNs which are trained by RGB images, flows and RGB images, RGB differences. The accuracy on split 1 of UCF-101 with a bninception backbone is 90.4% for spatial networks, 91.8% for temporal networks trained using flows and,  89.5% for temporal networks trained using RGB differences. With an inceptionv3 backbone we can achieve 92.1% for spatial networks, 93.5% for temporal networks trained using flows and,  91.6% for temporal networks trained using RGB differences on split 1 of UCF-101. They surpass all the independently trained two-stream networks in both spatial and temporal networks. Although our N-CRN is not a strong model, the performance boosted by combining the models (CRN+N-CRN) is significant. Since statistics generated by the coupled training is different from independent training, we expect our CRN models to be a good compensation for any independently trained two-stream models. 




Comparison with alternative designs and other state-of-the-art methods: The mean accuracy on the three splits of UCF-101 and HMDB-51 compared with the state-of-the-art and alternative designs can be seen from Table <ref>. CRN can achieve comparable if not better performance. Together with a N-CRN model, ours outperforms the state-of-the-art on both datasets by a large margin. We also present the alternative designs for the fusion of multiple input sources as shown in Fig. <ref>. Most of previous state-of-the-art methods adopt (a), a two-stream architecture, we also experiment (b) and (c) for a fair comparison. (b) is a form of late fusion, two hidden outputs from two branches will be fed into a fusion module. (c) is a form of early fusion, the concatenated inputs will be fed into a fusion module and then pass through a recurrent network.  Even effective compared to some other methods, neither late fusion nor early fusion can provide better representations than CRN for action recognition task.






Besides these relatively large datasets, we also evaluate CRN on a larger dataset, Moments in Time <cit.>. The performance can be seen in Table <ref>. CRN sets a new benchmark on Moments in Time by a lager margin. Single spatial and temporal model can beat the sophisticated assembled ones. 





  §.§.§ Real Time Action Recognition

Real-time action recognition is important for practical applications. Inspired by <cit.>, the RGB difference between the neighboring frames can be a good substitute of optical flows. Compared with optical flows which require certain amount of calculations, RGB difference can be directly inferred from RGB frames online without burden. Balance of speed and accuracy, RGB frames and RGB difference are the good input sources for real-time action recognition. However, as indicated in <cit.>, compared to optical flows, RGB difference only provides weak motion information which will degrade the performance. Since CRN enable spatial and temporal networks interpret complementary information from each other, the learning of RGBDiff can be calibrated by RGB, thus, we can achieve much better performance compared to previous state-of-the-art (93% vs. 91%). As shown in Table <ref>, applying CRN boosts the real-time action recognition performance by 2%. 





 §.§ Evaluation on Multi-person Pose Estimation




Human pose estimation is another dimension to analyze human activity. CRNs with different numbers of hidden maps is evaluated on MPII as shown in Table <ref>. 





Here, CRN_F* indicates a CRN with the corresponding `*' hidden feature maps in the LSTM. Even with 64 feature maps, CRN can exceed the state-of-the-art method <cit.>. When the number of hidden feature maps increases, the performance becomes better. Note that even with 128 feature maps, the size of our proposed model is still smaller the model proposed in <cit.>. 






































The performance varies when different time steps are applied for the inference. As shown in Fig. <ref>, prediction of easy joint locations, such as, head or shoulders, even in early stage (t ≥ 3), perform well. However, for more difficult ones, such as, wrist or ankle, more time steps are required for better performance. This experiments present the effectiveness of the recursive refinement for image based computer vision task using CRN. 



 §.§ Quantitative Performance Evaluation

The joint location and PAFs prediction at different time steps are generated in Fig. <ref>. As the time step increases, the joint as well as PAFs prediction becomes more and more confident (the brightness reveals the confidence level). Pay attention to the joint prediction shown in the dashed yellow rectangles along the time axis, initially, the confidence of prediction is pretty weak, however, with the time step increasing, the confidence is highly augmented. It verifies our assumption that CRN does iterative refining from one side. 

The visualization of the pose estimation on sample images from MPII <cit.> are shown in Fig. <ref>. 




In this figure, (a), (c), (e) and (g) are the results generated using CRN and (b), (d), (f) and (h) are the results generated using the method proposed in <cit.>. From this figure, we can see that our proposed method, Coupled Recurrent Network (CRN), can deal well with rare poses or appearances with less/no false parts detection. Even for images with substantial overlap of the body parts of two people, our proposed method still works well, correctly associating parts for each person. (a) vs. (b), (e) vs. (f) and (g) vs. (h) presents that CRN can work well in different situations with variation of viewpoint and appearance. (c) vs. (d) shows that CRN can work better for occluded poses than other state-of-the-art method proposed in <cit.>. 



§ SUMMARY
 
In this paper, we propose a novel architecture, called a Coupled Recurrent Network (CRN), to learn better representations from the multiple input sources. With the RIB module, reciprocal information can be well distilled from the related input source. Iterative refinement using re-currency improves the performance step by step. Extensive experiments are conducted on two tasks, human action recognition and multi-person pose estimation. Due to the effective integration of features from different sources, our model can achieve the state-of-the-art performance on these human-centric computer vision tasks. Hope our work shed the light on other computer vision or machine learning tasks with multiple inputs. 



§ ACKNOWLEDGMENT


The author would like to thank Dr. Xingyu Zhang for constructive comments that greatly improved the manuscript. 


unsrt


