To improve software quality, one needs to build test scenarios resembling the usage of a software product in the field. This task is rendered challenging when a product's customer base is large and diverse. In this scenario, existing profiling approaches, such as operational profiling, are difficult to apply. In this work, we consider publicly available video tutorials of a product to profile usage. Our goal is to construct an automatic approach to extract information about user actions from instructional videos. To achieve this goal, we use a Deep Convolutional Neural Network (DCNN) to recognize user actions. Our pilot study shows that a DCNN trained to recognize user actions in video can classify five different actions in a collection of 236 publicly available Microsoft Word tutorial videos (published on YouTube). In our empirical evaluation we report a mean average precision of 94.42 across all actions. This study demonstrates the efficacy of DCNN-based methods for extracting software usage information from videos. Moreover, this approach may aid in other software engineering activities that require information about customer usage of a product.
@cite_5: Deep learning subsumes algorithms that automatically learn compositional representations. The ability of these models to generalize well has ushered in tremendous advances in many fields. We propose that software engineering (SE) research is a unique opportunity to use these transformative approaches. Our research examines applications of deep architectures such as recurrent neural networks and stacked restricted Boltzmann machines to SE tasks.
It was suggested @cite_5 that DCNNs may be used in source code analysis for ... viz. code suggestion, code summarization, traceability link recovery, and feature location''. To the best of our knowledge, no-one in SE has considered extracting UI-based actions from videos using DCNN.