This paper proposes a novel rotation invariant feature for object recognition. Firstly, the local Fourier transform features of pixels in the described region are encoded by Fisher Vectors. Then, the encoded vectors are aggregated into a final representation by ordinal pyramid pooling, which hierarchically partitions the described region into sub-regions based on the orders of its pixels' rotation invariants. Since both the encoded Fisher Vectors and the ordinal pyramid pooling strategy are rotation invariant, the extracted feature is rotation invariant by nature. Two kinds of rotation invariants are investigated in this framework, one is the Radial Gradient Orientation and the other is the Radial Gradient Angle. Experiments on handwritten digit recognition and airplane/car detection in aerial images demonstrate the effectiveness of the proposed method, which outperforms the state of the art. (C) 2017 Elsevier B.V. All rights reserved.