Video stabilization algorithms are of greater importance nowadays with the prevalence of hand-held devices which unavoidably produce videos with undesirable shaky motions. In this paper we propose a data-driven online video stabilization method along with a paired dataset for deep learning. The network processes each unsteady frame progressively in a multi-scale manner, from low resolution to high resolution, and then outputs an affine transformation to stabilize the frame. Different from conventional methods which require explicit feature tracking or optical flow estimation, the underlying stabilization process is learned implicitly from the training data, and the stabilization process can be done online. Since there are limited public video stabilization datasets available, we synthesized unstable videos with different extent of shake that simulate real-life camera movement. Experiments show that our method is able to outperform other stabilization methods in several unstable samples while remaining comparable in general. Also, our method is tested on complex contents and found robust enough to dampen these samples to some extent even it was not explicitly trained in the contents.
@cite_14: 1 Problem Statement Let I and J be two 2D grayscaled images. The two quantities I(x) = I(x, y) and J(x) = J(x, y) are then the grayscale value of the two images are the location x = [x y] , where x and y are the two pixel coordinates of a generic image point x. The image I will sometimes be referenced as the first image, and the image J as the second image. For practical issues, the images I and J are discret function (or arrays), and the upper left corner pixel coordinate vector is [0 0] . Let nx and ny be the width and height of the two images. Then the lower right pixel coordinate vector is [nx − 1 ny − 1] . Consider an image point u = [ux uy] on the first image I. The goal of feature tracking is to find the location v = u + d = [ux+dx uy +dy] on the second image J such as I(u) and J(v) are “similar”. The vector d = [dx dy] is the image velocity at x, also known as the optical flow at x. Because of the aperture problem, it is essential to define the notion of similarity in a 2D neighborhood sense. Let ωx and ωy two integers. We define the image velocity d as being the vector that minimizes the residual function defined as follows:
Our proposed system employs ConvNet to extract the interframe transformation, which does not explicitly estimate the camera motion path or feature trajectory but leave them for the network to learn implicitly. In order to achieve multiple levels of refinement on the stabilization process, we referred to the pyramidal implementation in @cite_14 , which improved the performance of optical flow detection, and constructed three levels of different resolutions to do iterative adjustment in stabilization. To our knowledge, our method StableNet is the first which uses deep ConvNet in a multiscale setting for video stablization and produces competitive results compared with state-of-the-art methods.