To improve software quality, one needs to build test scenarios resembling the usage of a software product in the field. This task is rendered challenging when a product's customer base is large and diverse. In this scenario, existing profiling approaches, such as operational profiling, are difficult to apply. In this work, we consider publicly available video tutorials of a product to profile usage. Our goal is to construct an automatic approach to extract information about user actions from instructional videos. To achieve this goal, we use a Deep Convolutional Neural Network (DCNN) to recognize user actions. Our pilot study shows that a DCNN trained to recognize user actions in video can classify five different actions in a collection of 236 publicly available Microsoft Word tutorial videos (published on YouTube). In our empirical evaluation we report a mean average precision of 94.42 across all actions. This study demonstrates the efficacy of DCNN-based methods for extracting software usage information from videos. Moreover, this approach may aid in other software engineering activities that require information about customer usage of a product.
@cite_4: This paper presents a vision of how the Internet of Things will impact the study of software engineering by 2025 and beyond. The following questions guide this inquiry. What will it mean to be able to deploy hundreds of sensors and data collectors running concurrently over months to gather very large and rich datasets of the physical, digital, and social aspects of software engineering organizations and the products and services those organizations create? How might such datasets change the types of research questions that can be addressed? What sort of tools will be needed to allow interdisciplinary communities of researchers to collaboratively analyse such datasets? How might such datasets help us understand the principles governing the interplay of physical, cyber, and social aspects of software engineering and its products, and automate aspects of such systems?
Video recordings are already used in Software Engineering (SE); e.g., in ethnographic studies of development organizations and in user experience research (see @cite_4 for review). To the best of our knowledge, automatic classification of sequences in videos is not utilized.