We consider the task of controlling a quadrotor to hover in front of a freely moving user, using input data from an onboard camera. On this specific task we compare two widespread learning paradigms: a mediated approach, which learns a high-level state from the input and then uses it for deriving control signals; and an end-to-end approach, which skips high-level state estimation altogether. We show that despite their fundamental difference, both approaches yield equivalent performance on this task. We finally qualitatively analyze the behavior of a quadrotor implementing such approaches.
@cite_18: Estimating the 6D pose of known objects is important for robots to interact with the real world. The problem is challenging due to the variety of objects as well as the complexity of a scene caused by clutter and occlusions between objects. In this work, we introduce PoseCNN, a new Convolutional Neural Network for 6D object pose estimation. PoseCNN estimates the 3D translation of an object by localizing its center in the image and predicting its distance from the camera. The 3D rotation of the object is estimated by regressing to a quaternion representation. We also introduce a novel loss function that enables PoseCNN to handle symmetric objects. In addition, we contribute a large scale video dataset for 6D object pose estimation named the YCB-Video dataset. Our dataset provides accurate 6D poses of 21 objects from the YCB dataset observed in 92 videos with 133,827 frames. We conduct extensive experiments on our YCB-Video dataset and the OccludedLINEMOD dataset to show that PoseCNN is highly robust to occlusions, can handle symmetric objects, and provide accurate pose estimation using only color images as input. When using depth data to further refine the poses, our approach achieves state-of-the-art results on the challenging OccludedLINEMOD dataset. Our code and dataset are available at this https URL.
@cite_26: We describe the first method to automatically estimate the 3D pose of the human body as well as its 3D shape from a single unconstrained image. We estimate a full 3D mesh and show that 2D joints alone carry a surprising amount of information about body shape. The problem is challenging because of the complexity of the human body, articulation, occlusion, clothing, lighting, and the inherent ambiguity in inferring 3D from 2D. To solve this, we first use a recently published CNN-based method, DeepCut, to predict (bottom-up) the 2D body joint locations. We then fit (top-down) a recently published statistical body shape model, called SMPL, to the 2D joints. We do so by minimizing an objective function that penalizes the error between the projected 3D model joints and detected 2D joints. Because SMPL captures correlations in human shape across the population, we are able to robustly fit it to very little data. We further leverage the 3D model to prevent solutions that cause interpenetration. We evaluate our method, SMPLify, on the Leeds Sports, HumanEva, and Human3.6M datasets, showing superior pose accuracy with respect to the state of the art.
@cite_8: We consider the design of vision-based control algorithms for unmanned aerial vehicles (UAVs), so as to enable a UAV to autonomously follow a person. A new vision-based control architecture is proposed with the goals of 1) robustly following the user and 2) implementing following behaviors programmed by manipulation of visual patterns. This is achieved within a detection tracking paradigm, where the target is a programmable badge worn by the user. This badge contains a visual pattern with two components. The first is fixed and used to locate the user. The second is variable and implements a code used to program the UAV behavior. A biologically inspired tracking recognition architecture, combining bottom-up and top-down saliency mechanisms, a novel image similarity measure, and an affine validation procedure, is proposed to detect the badge in the scene. The badge location is used by a control algorithm to adjust the UAV flight parameters so as to maintain the user in the center of the field of view. The detected badge is further analyzed to extract the visual code that commands the UAV behavior This is used to control the height and distance of the UAV relative to the user.
@cite_29: A wide range of human-robot collaborative applications in industry, search and rescue operations, healthcare, and social interactions require an autonomous robot to follow its human companion. Different operating mediums and applications pose diverse challenges by adding constraints on the choice of sensors, the degree of autonomy, and dynamics of the person following robot. Researchers have addressed these challenges in many ways and contributed to the development of a large body of literature. This paper provides a comprehensive overview of the literature by categorizing different aspects of person-following by autonomous robots. Also, the corresponding operational challenges are identified based on various design choices for ground, underwater, and aerial scenarios. In addition, state-of-the-art methods for perception, planning, control, and interaction are elaborately discussed, and their feasibilities are evaluated in terms of standard operational and performance metrics. Furthermore, several prospective application areas are identified, and open problems are highlighted for future research.
@cite_32: 
@cite_31: In this paper, we present an approach that allows a quadrocopter to follow a person and to recognize simple gestures using an onboard depth camera. This enables novel applications such as hands-free filming and picture taking. The problem of tracking a person with an onboard camera however is highly challenging due to the self-motion of the platform. To overcome this problem, we stabilize the depth image by warping it to a virtual-static camera, using the estimated pose of the quadrocopter obtained from vision and inertial sensors using an Extended Kalman filter. We show that such a stabilized depth video is well suited to use with existing person trackers such as the OpenNI tracker. Using this approach, the quadrocopter not only obtains the position and orientation of the tracked person, but also the full body pose - which can then for example be used to recognize hand gestures to control the quadrocopter's behaviour. We implemented a small set of example commands (“follow me”, “take picture”, “land”), and generate corresponding motion commands. We demonstrate the practical performance of our approach in an extensive set of experiments with a quadrocopter. Although our current system is limited to indoor environments and small motions due to the restrictions of the used depth sensor, it indicates that there is large potential for such applications in the near future.
@cite_13: AR.Drone is a quadcopter with onboard sensors including a frontal camera. The drone can be controlled from a computer via WiFi. This paper describes a method for autonomous tracking of a selected object by AR.Drone. We utilize a computer-vision approach called tracking-learning-detection (TLD) to track an arbitrary object selected by a user in the video-stream going from the front camera of the drone. Information about location of the tracked object is then used to guide the drone using the proportional-integral-derivative (PID) controller. The method was implemented in software FollowMe.
@cite_25: Abstract Head pose estimation is an old problem that is recently receiving new attention because of possible applications in human-robot interaction, augmented reality and driving assistance. However, most of the existing work has been tested in controlled environments and is not robust enough for real-world applications. In order to handle these limitations we propose an approach based on Convolutional Neural Networks (CNNs) supplemented with the most recent techniques adopted from the deep learning community. We evaluate the performance of four architectures on recently released in-the-wild datasets. Moreover, we investigate the use of dropout and adaptive gradient methods giving a contribution to their ongoing validation. The results show that joining CNNs and adaptive gradient methods leads to the state-of-the-art in unconstrained head pose estimation.
@cite_20: We introduce a socially motivated interaction technique with collocated flying robots (a quadrotor in our current prototype). Instead of the traditional remote interaction controllers often used when interacting with flying robots and UAVs, we explore the collocated interaction space and suggest a direct interaction technique motivated by social human-robot interaction themes. Our approach is inspired by the types of interaction humans have with birds, specifically falconeering, and is facilitated by gestures-based interaction, while the user is within the field of view of the flying robot. This paper outlines our research goals, task examples, and our overall design approach. The paper also discusses our current prototyping efforts, as well as a preliminary evaluation of our approach, performed through two design critiques, studying our collocated interaction technique concept, and its potential, drawbacks and benefits for users.
@cite_11: This article provides an overview of existing interaction techniques for controlling unmanned aerial vehicle (UAV) systems. This work focuses on user interfaces with nontraditional input modalities, such as gestures, speech, and gaze direction. Although the authors analyze interaction with UAV systems, most of the findings can be applied to human-robot interaction in general. The authors report on interaction techniques employed to control single as well as multiple UAV systems, define intuitiveness of input vocabularies in the considered context, and introduce a new classification scheme based on the mental models underlying the interaction vocabulary. This article is part of a special issue on drones.
The task we consider in this paper is aimed to proximity human-robot interaction @cite_20 @cite_31 : a drone should be able to fly at an appropriate distance in front of people, following them @cite_29 while waiting for possible command gestures @cite_11 . The use of visual markers @cite_8 simplifies this task but, in general, more sophisticated techniques, like Tracking-Learning-Detection @cite_32 , are needed @cite_13 . For instance, several deep learning approaches have been proposed to estimate the 6D pose of a person's head @cite_25 or body @cite_26 from monocular cameras (as well of general objects @cite_18 ). Adopting such perception modules would be a reasonable alternative to solve our task; however, in this paper the task acts as a model of a larger class of tasks.