In this paper we derive and test a probability-based weighting that can balance residuals of different types in spline fitting. In contrast to previous formulations, the proposed spline error weighting scheme also incorporates a prediction of the approximation error of the spline fit. We demonstrate the effectiveness of the prediction in a synthetic experiment, and apply it to visual-inertial fusion on rolling shutter cameras. This results in a method that can estimate 3D structure with metric scale on generic first-person videos. We also propose a quality measure for spline fitting, that can be used to automatically select the knot spacing. Experiments verify that the obtained trajectory quality corresponds well with the requested quality. Finally, by linearly scaling the weights, we show that the proposed spline error weighting minimizes the estimation errors on real sequences, in terms of scale and end-point errors.
@cite_21: All existing methods for vision-aided inertial navigation assume a camera with a global shutter, in which all the pixels in an image are captured simultaneously. However, the vast majority of consumer-grade cameras use rolling-shutter sensors, which capture each row of pixels at a slightly different time instant. The effects of the rolling shutter distortion when a camera is in motion can be very significant, and are not modelled by existing visual-inertial motion-tracking methods. In this paper we describe the first, to the best of our knowledge, method for vision-aided inertial navigation using rolling-shutter cameras. Specifically, we present an extended Kalman filter (EKF)-based method for visual-inertial odometry, which fuses the IMU measurements with observations of visual feature tracks provided by the camera. The key contribution of this work is a computationally tractable approach for taking into account the rolling-shutter effect, incurring only minimal approximations. The experimental results from the application of the method show that it is able to track, in real time, the position of a mobile phone moving in an unknown environment with an error accumulation of approximately 0.8 of the distance travelled, over hundreds of meters.
@cite_22: Gyroscope is playing a key role in helping estimate camera rotation during mobile video capture. The fusion of gyroscope and visual measurements needs the knowledge of camera projection parameters, the gyroscope bias and the relative orientation between gyroscope and camera. Moreover, the timestamps of gyroscope and video frames are usually not well synchronized. In this paper, we propose an online method that estimates all the necessary parameters while capturing videos. Our contributions are (1) simultaneous online camera self-calibration and camera-gyroscope calibration based on an implicit extended Kalman filter, and (2) generalization of coplanarity constraint of camera rotation in a rolling shutter camera model for cellphones. The proposed method is able to accurately estimate the needed parameters online with all kinds of camera motion, and can be embedded in gyro-aided applications such as video stabilization and feature tracking.
@cite_17: We present a system that rectifies and stabilizes video sequences on mobile devices with rolling-shutter cameras. The system corrects for rolling-shutter distortions using measurements from accelerometer and gyroscope sensors, and a 3D rotational distortion model. In order to obtain a stabilized video, and at the same time keep most content in view, we propose an adaptive low-pass filter algorithm to obtain the output camera trajectory. The accuracy of the orientation estimates has been evaluated experimentally using ground truth data from a motion capture system. We have conducted a user study, where the output from our system, implemented in iOS, has been compared to that of three other applications, as well as to the uncorrected video. The study shows that users prefer our sensor-based system.
Visual-inertial fusion on rolling shutter cameras has classically been done using Extended Kalman-filters (EKF). @cite_17 use an EKF to track cell-phone orientation for the purpose of video stabilization. @cite_21 extend this to full device motion tracking, and @cite_22 add tracking of changes in relative pose between the sensors and changes in linear camera intrinsics.