In this paper, we show how a 3D Morphable Model (i.e. a statistical model of the 3D shape of a class of objects such as faces) can be used to spatially transform input data as a module (a 3DMM-STN) within a convolutional neural network. This is an extension of the original spatial transformer network in that we are able to interpret and normalise 3D pose changes and self-occlusions. The trained localisation part of the network is independently useful since it learns to fit a 3D morphable model to a single image. We show that the localiser can be trained using only simple geometric loss functions on a relatively small dataset yet is able to perform robust normalisation on highly uncontrolled images including occlusion, self-occlusion and large pose changes.
@cite_14: Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.
The original STN @cite_14 aimed to combine these two processes into a single network that is trainable end to end. The localiser network estimated a 2D affine transformation that was applied to the regular output grid meaning the network could only learn a fairly restricted space of transformations. Jaderberg al @cite_14 also proposed the concept of a 3D transformer, which takes 3D voxel data as input, applies 3D rotation and translation, and outputs a 2D projection of the transformed data. Working with 3D (volumetric data) removes the need to model occlusion or camera projection parameters. In contrast, we work with regular 2D input and output images but transform them via a 3D model.