As an important pre-filtering procedure for object detection, objectness estimation now draws lots of attentions from both the video analysis and computer vision communities. Among the existing approaches, BING [1] (binarized normed gradients) is the recently proposed one that possesses the advantages of high object detection rate, ultra fast computational speed (i.e. 300 fps), small number of proposals, and good generalization ability. However, within BING's framework, only the uniform hierarchical ranking structure and gradient information are employed for objectness characterization, which leads to high false positive rate. To address this issue, an effective and efficient cascade ranking method is proposed to refine BING, in this paper. It declares three main contributions. First, the concept of scale-sets histogram is novelly introduced. It helps to analyze the potential sizes of the objects. Secondly, more descriptive visual features (i.e. the color-texture consistence) are considered simultaneously for objectness characterization. Lastly, we propose hierarchical sorting to further leverage the ranking performance, according to the local contrast analysis between the inside and straddling superpixels of the object proposal windows. The experimental results on four challenging datasets PASCAL VOC2007, PASCAL VOC2010, PASCAL VOC2012 and ILSVRC2014 demonstrate that, the refined BING can achieve good balance between high detection rate (DR) (e.g. 96.7% DR with 1000 proposal windows on VOC2007), and low time consumption (e.g. 75 ms per image on VOC2007). It is also worth noting that, with a relatively small number of proposals (e.g. fewer than 200 proposals), our approach generally outperforms the state-of-the-art methods on the detection rate, on all the testing datasets. (C) 2016 Elsevier B.V. All rights reserved.