Texture feature extraction methods have been improved greatly in recent years. It is widely known that the local texture feature descriptor can achieve desired performance under the change of image geometric size, different poses and complex illumination conditions. In this paper, a novel local texture descriptor is proposed, named as the multi-degrees improved local difference binary (ILDB). Local difference binary is an promising feature description method, while it only computes the intensity and gradient difference on pairwise grid cells and ignores the image grid inherent texture gradient difference. ILDB can represent difference and texture information of the grid cells intensity and gradient simultaneously. In addition, the multiple-degree strategy is adopted to achieve richer texture description. At the same time, the optimized mutual information is proposed to capture more discriminant feature selection and reduce the dimensionality of the ILDB. Experimental results demonstrate that the proposed method is highly efficient and distinctive compared with several state-of-the-art approaches. Due to good performance of ILDB, it is expected that ILDB has a potential for widespread application in many computer vision fields.