Videos have become ubiquitous on the Internet. And video analysis can provide lots of information for detecting and recognizing objects as well as help people understand human actions and interactions with the real world. However, facing data as huge as terabyte (TB) level, effective methods should be applied. Recurrent neural network (RNN) architecture has wildly been used on many sequential learning problems such as Language Model, Time-Series Analysis, etc. In this paper, we propose some variations of RNN such as stacked bidirectional Long Short Term Memory (LSTM) Gated Recurrent Unit (GRU) network with attention mechanism to categorize large-scale video data. We also explore different multimodal fusion methods. Our model combines both visual and audio information on both video and frame level and received great result. Ensemble methods are also applied. Because of its multimodal characteristics, we decide to call this method Deep Multimodal Learning (DML). Our DML-based model was trained on Google Cloud and our own server and was tested in a well-known video classification competition on Kaggle held by Google.
@cite_8: Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Multimodal is an emerging area which focuses on using various modalities to improve the performance of the model. More specifically, it tends to solve these five problems: Representation, Translation, Alignment, Fusion and Co-learning. Representation means learning how to represent and summarize multimodal data that exploits the complementarity and redundancy of multiple modalities. Translation studies how to translate data from one modality to another. Alignment tries to identify the direct relations between elements from over two modalities. Fusion is a challenge joining information from two or more modalities to perform a prediction. Co-learning transfers knowledge between modalities, their representation, and their predictive models @cite_8 . What we concern in our work is Fusion, cause the whole task is about prediction with two modalities: visual and audio. It includes two types of methods: Joint and Coordinated. We mainly cares about Joint method, which consists of three types of methods, including neural networks, graphical models and sequential. We care about the first one because our model adopts end-to-end neural networks.