Video stabilization algorithms are of greater importance nowadays with the prevalence of hand-held devices which unavoidably produce videos with undesirable shaky motions. In this paper we propose a data-driven online video stabilization method along with a paired dataset for deep learning. The network processes each unsteady frame progressively in a multi-scale manner, from low resolution to high resolution, and then outputs an affine transformation to stabilize the frame. Different from conventional methods which require explicit feature tracking or optical flow estimation, the underlying stabilization process is learned implicitly from the training data, and the stabilization process can be done online. Since there are limited public video stabilization datasets available, we synthesized unstable videos with different extent of shake that simulate real-life camera movement. Experiments show that our method is able to outperform other stabilization methods in several unstable samples while remaining comparable in general. Also, our method is tested on complex contents and found robust enough to dampen these samples to some extent even it was not explicitly trained in the contents.
@cite_18: Video stabilization technique is essential for most hand-held captured videos due to high-frequency shakes. Several 2D-, 2.5D- and 3D-based stabilization techniques are well studied, but to our knowledge, no solutions based on deep neural networks had been proposed. The reason for this is mostly the shortage of training data, as well as the challenge of modeling the problem using neural networks. In this paper, we solve the video stabilization problem using a convolutional neural network (ConvNet). Instead of dealing with offline holistic camera path smoothing based on feature matching, we focus on low-latency real-time camera path smoothing without explicitly representing the camera path. Our network, called StabNet, learns a transformation for each input unsteady frame progressively along the time-line, while creating a more stable latent camera path. To train the network, we create a dataset of synchronized steady unsteady video pairs via a well designed hand-held hardware. Experimental results shows that the proposed online method (without using future frames) performs comparatively to traditional offline video stabilization methods, while running about 30 times faster. Further, the proposed StabNet is able to handle night-time and blurry videos, where existing methods fail in robust feature matching.
@cite_9: Many existing video stabilization methods often stabilize videos off-line, i.e. as a postprocessing tool of pre-recorded videos. Some methods can stabilize videos online, but either require additional hardware sensors (e.g., gyroscope) or adopt a single parametric motion model (e.g., affine, homography) which is problematic to represent spatially-variant motions. In this paper, we propose a technique for online video stabilization with only one frame latency using a novel MeshFlow motion model. The MeshFlow is a spatial smooth sparse motion field with motion vectors only at the mesh vertexes. In particular, the motion vectors on the matched feature points are transferred to their corresponding nearby mesh vertexes. The MeshFlow is produced by assigning each vertex an unique motion vector via two median filters. The path smoothing is conducted on the vertex profiles, which are motion vectors collected at the same vertex location in the MeshFlow over time. The profiles are smoothed adaptively by a novel smoothing technique, namely the Predicted Adaptive Path Smoothing (PAPS), which only uses motions from the past. In this way, the proposed method not only handles spatially-variant motions but also works online in real time, offering potential for a variety of intelligent applications (e.g., security systems, robotics, UAVs). The quantitative and qualitative evaluations show that our method can produce comparable results with the state-of-the-art off-line methods.
@cite_3: The acquisition of digital video usually suffers from undesirable camera jitter due to unstable random camera motion, which is produced by a hand-held camera or a camera in a vehicle moving on a non-smooth road or terrain. We propose a real-time robust video stabilization algorithm to remove undesirable motion jitter and produce a stabilized video. We first compute the optical flow between successive frames, followed by estimating the camera motion by fitting the computed optical flow field to a simplified affine motion model with a trimmed least squares method. Then, the computed camera motions are smoothed temporally to reduce the motion vibrations by using a regularization method. Finally, we transform all frames of the video based on the original and smoothed motions to obtain a stabilized video. Experimental results are given to demonstrate the stabilization performance and the efficiency of the proposed algorithm.
@cite_24: Amateur digital video footage, shot with a hand-held camera, is characterized by visually annoying, jerky movement between successive frames. In this paper a fast and robust image stabilization technique is presented to remove the effects of unwanted camera motion while not effecting true pans or zooms. Unlike other image stabilization techniques, which are typically used as a pre-processing step to more complicated image processing operations, the proposed technique is specifically aimed at providing "visually jerk-free video" for consumer viewing. Empirical evidence suggested that a simple global translational model for each pair of successive images followed by local integration of cumulative motion is sufficient for this purpose. We use a novel technique to estimate the translational motion model which reduces the total computational burden to less than 1 computation pixel, while providing robust jerk elimination, thus making real-time operation possible on most PC UNIX platforms. Tests with a number of video sequences have shown the efficacy of the proposed method.
@cite_13: 
Besides the previous offline approaches, several online methods are also proposed, which are favored by real-time applications. Ratakonda @cite_24 proposed a primitive online stabilization algorithm of integrating cumulative motion curve in two dimensions separately. Optical flows were calculated between successive frames in @cite_3 , and the camera path was optimized with regularization. Liu al @cite_9 presented MeshFlow using only the past motions. Karpenko al @cite_13 demonstrated real-time stabilization with gyroscopes for automatic calibration on a mobile phone. Integrating ConvNet, Wang al @cite_18 recently proposed StabNet to perform online stabilization by loading historical frames into the network and directly producing the homography matrix entries. However, this method may lead to serious error propagation in that the former lapses cause latter abnormality in terms of the visual performance.