On-orbit proximity operations in space rendezvous, docking and debris removal require precise and robust 6D pose estimation under a wide range of lighting conditions and against highly textured background, i.e., the Earth. This paper investigates leveraging deep learning and photorealistic rendering for monocular pose estimation of known uncooperative spacecrafts. We first present a simulator built on Unreal Engine 4, named URSO, to generate labeled images of spacecrafts orbiting the Earth, which can be used to train and evaluate neural networks. Secondly, we propose a deep learning framework for pose estimation based on orientation soft classification, which allows modelling orientation ambiguity as a mixture of Gaussians. This framework was evaluated both on URSO datasets and the ESA pose estimation challenge. In this competition, our best model achieved 3rd place on the synthetic test set and 2nd place on the real test set. Moreover, our results show the impact of several architectural and training aspects, and we demonstrate qualitatively how models learned on URSO datasets can perform on real images from space.
@cite_35: This paper describes a novel framework to determine the relative pose and range of a solid-of-revolution-shaped spacecraft from a single image without any artificial beacons. The translation and the symmetry axis of the spacecraft can be estimated from the imaged cross sections of the spacecraft body. Then the pose and range of the spacecraft are fully determined by means of the images of its solar panels and asymmetric feature. Our method has been validated by both synthetic and real images.
@cite_14: This paper focuses on the issue of estimating the complete 3D pose of the camera with respect to a potentially textureless object, through model-based tracking. We propose to robustly combine complementary geometrical and color edge-based features in the minimization process, and to integrate a multiple-hypotheses framework in the geometrical edge-based registration phase. In order to deal with complex 3D models, our method takes advantage of GPU acceleration. Promising results, outperforming classical state-of-art approaches, have been obtained for space robotics applications on various real and synthetic image sequences and using satellite mock-ups as targets.
@cite_4: Presents a framework for three-dimensional model-based tracking. Graphical rendering technology is combined with constrained active contour tracking to create a robust wire-frame tracking system. It operates in real time at video frame rate (25 Hz) on standard hardware. It is based on an internal CAD model of the object to be tracked which is rendered using a binary space partition tree to perform hidden line removal. A Lie group formalism is used to cast the motion computation problem into simple geometric terms so that tracking becomes a simple optimization problem solved by means of iterative reweighted least squares. A visual servoing system constructed using this framework is presented together with results showing the accuracy of the tracker. The paper then describes how this tracking system has been extended to provide a general framework for tracking in complex configurations. The adjoint representation of the group is used to transform measurements into common coordinate frames. The constraints are then imposed by means of Lagrange multipliers. Results from a number of experiments performed using this framework are presented and discussed.
@cite_7: This paper addresses the challenge of detecting and localizing a poorly textured known object, by initially estimating its complete 3D pose in a video sequence. Our solution relies on the 3D model of the object and synthetic views. The full pose estimation process is then based on foreground background segmentation and on an efficient probabilistic edge-based matching and alignment procedure with the set of synthetic views, classified through an unsupervised learning phase. Our study focuses on space robotics applications and the method has been tested on both synthetic and real images, showing its efficiency and convenience, with reasonable computational costs.
@cite_28: 
@cite_36: Autonomous rendezvous and docking is necessary for planned space programs such as DARPA ASTRO, NASA MSR, ISS assembly and servicing, and other rendezvous and proximity operations. Estimation of the relative pose between the host platform and a resident space object is a critical ability. We present a model-based pose refinement algorithm, part of a suite of algorithms for vision-based relative pose estimation and tracking. Algorithms were tested in high-fidelity simulation and stereo-vision hardware test bed environments. Testing indicated that in most cases, the model-based pose refinement algorithm can handle initial attitude errors up to about 20 degrees, range errors exceeding 10 of range, and transverse errors up to about 2 of range. Preliminary point tests with real camera sequences of a 1 24 scale Magellan satellite model using a simple fixed-gain tracking filter showed potential tracking performance with mean errors of < 3 degrees and < 2 of range.
Previous monocular solutions @cite_28 @cite_36 @cite_35 @cite_14 @cite_7 to spacecraft tracking and pose estimation rely on model-based approaches @cite_4 that align a wireframe model of the object to an edge image (typically given by a Canny detector) of the real object based on heuristics. However objects are more than just a collection of edges and geometric primitives. Convolutional Neural Networks (CNNs) can learn more complex and meaningful features to the task at hand while ignoring background features (e.g. clouds) based on context.