Abstract Designing discriminative powerful texture features robust to realistic imaging conditions is a challenging computer vision problem with many applications, including material recognition and analysis of satellite or aerial imagery. In the past, most texture description approaches were based on dense orderless statistical distribution of local features. However, most recent approaches to texture recognition and remote sensing scene classification are based on Convolutional Neural Networks (CNNs). The de facto practice when learning these CNN models is to use RGB patches as input with training performed on large amounts of labeled data (ImageNet). In this paper, we show that Local Binary Patterns (LBP) encoded CNN models, codenamed TEX-Nets, trained using mapped coded images with explicit LBP based texture information provide complementary information to the standard RGB deep models. Additionally, two deep architectures, namely early and late fusion, are investigated to combine the texture and color information. To the best of our knowledge, we are the first to investigate Binary Patterns encoded CNNs and different deep network fusion architectures for texture recognition and remote sensing scene classification. We perform comprehensive experiments on four texture recognition datasets and four remote sensing scene classification benchmarks: UC-Merced with 21 scene categories, WHU-RS19 with 19 scene classes, RSSCN7 with 7 categories and the recently introduced large scale aerial image dataset (AID) with 30 aerial scene types. We demonstrate that TEX-Nets provide complementary information to standard RGB deep model of the same network architecture. Our late fusion TEX-Net architecture always improves the overall performance compared to the standard RGB network on both recognition problems. Furthermore, our final combination leads to consistent improvement over the state-of-the-art for remote sensing scene classification.
@cite_35: The local binary pattern (LBP) operator is a very effective multi-resolution texture descriptor that can be applied in many image processing applications. However, existing LBP operators can not use the information of non-uniform patterns efficiently and they are also sensitive to noise. This paper proposes a noise tolerant extension of LBP operators to extract statistical and structural image features for efficient texture analysis. The proposed LBP operator uses a circular majority voting filter and suitable rotation-invariant labeling scheme to obtain more regular uniform and non-uniform patterns that have better discrimination ability and more robustness against noise. Experimental results on the Brodatz, CUReT and MeasTex databases show the improvement of the proposed LBP operator performance, especially when a large number of neighbors are used for extracting texture patterns.
@cite_67: Presents a theoretically very simple, yet efficient, multiresolution approach to gray-scale and rotation invariant texture classification based on local binary patterns and nonparametric discrimination of sample and prototype distributions. The method is based on recognizing that certain local binary patterns, termed "uniform," are fundamental properties of local image texture and their occurrence histogram is proven to be a very powerful texture feature. We derive a generalized gray-scale and rotation invariant operator presentation that allows for detecting the "uniform" patterns for any quantization of the angular space and for any spatial resolution and presents a method for combining multiple operators for multiresolution analysis. The proposed approach is very robust in terms of gray-scale variations since the operator is, by definition, invariant against any monotonic transformation of the gray scale. Another advantage is computational simplicity as the operator can be realized with a few operations in a small neighborhood and a lookup table. Experimental results demonstrate that good discrimination can be achieved with the occurrence statistics of simple rotation invariant local binary patterns.
@cite_33: Person description is a challenging problem in computer vision. We investigated two major aspects of person description: 1) gender and 2) action recognition in still images. Most state-of-the-art approaches for gender and action recognition rely on the description of a single body part, such as face or full-body. However, relying on a single body part is suboptimal due to significant variations in scale, viewpoint, and pose in real-world images. This paper proposes a semantic pyramid approach for pose normalization. Our approach is fully automatic and based on combining information from full-body, upper-body, and face regions for gender and action recognition in still images. The proposed approach does not require any annotations for upper-body and face of a person. Instead, we rely on pretrained state-of-the-art upper-body and face detectors to automatically extract semantic information of a person. Given multiple bounding boxes from each body part detector, we then propose a simple method to select the best candidate bounding box, which is used for feature extraction. Finally, the extracted features from the full-body, upper-body, and face regions are combined into a single representation for classification. To validate the proposed approach for gender recognition, experiments are performed on three large data sets namely: 1) human attribute; 2) head-shoulder; and 3) proxemics. For action recognition, we perform experiments on four data sets most used for benchmarking action recognition in still images: 1) Sports; 2) Willow; 3) PASCAL VOC 2010; and 4) Stanford-40. Our experiments clearly demonstrate that the proposed approach, despite its simplicity, outperforms state-of-the-art methods for gender and action recognition.
@cite_7: 
@cite_60: By combining Histograms of Oriented Gradients (HOG) and Local Binary Pattern (LBP) as the feature set, we propose a novel human detection approach capable of handling partial occlusion. Two kinds of detectors, i.e., global detector for whole scanning windows and part detectors for local regions, are learned from the training data using linear SVM. For each ambiguous scanning window, we construct an occlusion likelihood map by using the response of each block of the HOG feature to the global detector. The occlusion likelihood map is then segmented by Mean-shift approach. The segmented portion of the window with a majority of negative response is inferred as an occluded region. If partial occlusion is indicated with high likelihood in a certain scanning window, part detectors are applied on the unoccluded regions to achieve the final classification on the current scanning window. With the help of the augmented HOG-LBP feature and the global-part occlusion handling method, we achieve a detection rate of 91.3 with FPPW= 10−6, 94.7 with FPPW= 10−5, and 97.9 with FPPW= 10−4 on the INRIA dataset, which, to our best knowledge, is the best human detection performance on the INRIA dataset. The global-part occlusion handling method is further validated using synthesized occlusion data constructed from the INRIA and Pascal dataset.
@cite_28: This paper presents a novel approach for texture classification, generalizing the well-known local binary pattern (LBP) approach. In the proposed approach, two different and complementary types of features (pixel intensities and differences) are extracted from local patches. The intensity-based features consider the intensity of the central pixel (CI) and those of its neighbors (NI); while for the difference-based feature, two components are computed: the radial-difference (RD) and the angular-difference (AD). Inspired by the LBP approach, two intensity-based descriptors CI-LBP and NI-LBP, and two difference-based descriptors RD-LBP and AD-LBP are developed. All four descriptors are in the same form as conventional LBP codes, so they can be readily combined to form joint histograms to represent textured images. The proposed approach is computationally very simple: it is totally training-free, there is no need to learn a texton dictionary, and no tuning of parameters. We have conducted extensive experiments on three challenging texture databases (Outex, CUReT and KTHTIPS2b). Outex results show significant improvements over the classical LBP approach, which clearly demonstrates the great power of the joint distributions of these proposed descriptors for gray-scale and rotation invariant texture classification. The proposed method produces the best classification results on KTHTIPS2b, and results comparable to the state-of-the-art on CUReT.
@cite_85: Extending recognition to uncontrolled situations is a key challenge for practical face recognition systems. Finding efficient and discriminative facial appearance descriptors is crucial for this. Most existing approaches use features of just one type. Here we argue that robust recognition requires several different kinds of appearance information to be taken into account, suggesting the use of heterogeneous feature sets. We show that combining two of the most successful local face representations, Gabor wavelets and Local Binary Patterns (LBP), gives considerably better performance than either alone: they are complimentary in the sense that LBP captures small appearance details while Gabor features encode facial shape over a broader range of scales. Both feature sets are high dimensional so it is beneficial to use PCA to reduce the dimensionality prior to normalization and integration. The Kernel Discriminative Common Vector method is then applied to the combined feature vector to extract discriminant nonlinear features for recognition. The method is evaluated on several challenging face datasets including FRGC 1.0.4, FRGC 2.0.4 and FERET, with promising results.
@cite_1: Abstract Current approaches to color texture analysis can be roughly divided into two categories: methods that process color and texture information separately, and those that consider color and texture a joint phenomenon. In this paper, both approaches are empirically evaluated with a large set of natural color textures. The classification performance of color indexing methods is compared to gray-scale and color texture methods, and to combined color and texture methods, in static and varying illumination conditions. Based on the results, we argue that color and texture are separate phenomena that can, or even should, be treated individually.
@cite_57: Making recognition more reliable under uncontrolled lighting conditions is one of the most important challenges for practical face recognition systems. We tackle this by combining the strengths of robust illumination normalization, local texture-based face representations, distance transform based matching, kernel-based feature extraction and multiple feature fusion. Specifically, we make three main contributions: 1) we present a simple and efficient preprocessing chain that eliminates most of the effects of changing illumination while still preserving the essential appearance details that are needed for recognition; 2) we introduce local ternary patterns (LTP), a generalization of the local binary pattern (LBP) local texture descriptor that is more discriminant and less sensitive to noise in uniform regions, and we show that replacing comparisons based on local spatial histograms with a distance transform based similarity metric further improves the performance of LBP LTP based face recognition; and 3) we further improve robustness by adding Kernel principal component analysis (PCA) feature extraction and incorporating rich local appearance cues from two complementary sources-Gabor wavelets and LBP-showing that the combination is considerably more accurate than either feature set alone. The resulting method provides state-of-the-art performance on three data sets that are widely used for testing recognition under difficult illumination conditions: Extended Yale-B, CAS-PEAL-R1, and Face Recognition Grand Challenge version 2 experiment 4 (FRGC-204). For example, on the challenging FRGC-204 data set it halves the error rate relative to previously published methods, achieving a face verification rate of 88.1 at 0.1 false accept rate. Further experiments show that our preprocessing method outperforms several existing preprocessors for a range of feature sets, data sets and lighting conditions.
@cite_74: We show that combining multiple texture description methods significantly improves the performance compared to using the single best texture method alone.We further propose to use information theoretic compression approach to compress high-dimensional multi-texture features into a compact heterogeneous texture representation.We perform a comprehensive evaluation of color features, popular in object recognition, for the task of texture classification.We show that late fusion of our compact texture descriptor with discriminative color feature outperforms state-of-the-art results on challenging texture recognition datasets. Describing textures is a challenging problem in computer vision and pattern recognition. The classification problem involves assigning a category label to the texture class it belongs to. Several factors such as variations in scale, illumination and viewpoint make the problem of texture description extremely challenging. A variety of histogram based texture representations exists in literature. However, combining multiple texture descriptors and assessing their complementarity is still an open research problem. In this paper, we first show that combining multiple local texture descriptors significantly improves the recognition performance compared to using a single best method alone. This gain in performance is achieved at the cost of high-dimensional final image representation. To counter this problem, we propose to use an information-theoretic compression technique to obtain a compact texture description without any significant loss in accuracy. In addition, we perform a comprehensive evaluation of pure color descriptors, popular in object recognition, for the problem of texture classification. Experiments are performed on four challenging texture datasets namely, KTH-TIPS-2a, KTH-TIPS-2b, FMD and Texture-10. The experiments clearly demonstrate that our proposed compact multi-texture approach outperforms the single best texture method alone. In all cases, discriminative color names outperforms other color features for texture classification. Finally, we show that combining discriminative color names with compact texture representation outperforms state-of-the-art methods by 7.8 , 4.3 and 5.0 on KTH-TIPS-2a, KTH-TIPS-2b and Texture-10 datasets respectively.
@cite_47: Local or global rotation invariant feature extraction has been widely used in texture classification. Local invariant features, e.g. local binary pattern (LBP), have the drawback of losing global spatial information, while global features preserve little local texture information. This paper proposes an alternative hybrid scheme, globally rotation invariant matching with locally variant LBP texture features. Using LBP distribution, we first estimate the principal orientations of the texture image and then use them to align LBP histograms. The aligned histograms are then in turn used to measure the dissimilarity between images. A new texture descriptor, LBP variance (LBPV), is proposed to characterize the local contrast information into the one-dimensional LBP histogram. LBPV does not need any quantization and it is totally training-free. To further speed up the proposed matching scheme, we propose a method to reduce feature dimensions using distance measurement. The experimental results on representative databases show that the proposed LBPV operator and global matching scheme can achieve significant improvement, sometimes more than 10 in terms of classification accuracy, over traditional locally rotation invariant LBP method.
@cite_34: This paper introduces a rotation invariant extension to the blur insensitive local phase quantization texture descriptor. The new method consists of two stages, the first of which estimates the local characteristic orientation, and the second one extracts a binary descriptor vector. Both steps of the algorithm apply the phase of the locally computed Fourier transform coefficients, which can be shown to be insensitive to centrally symmetric image blurring. The new descriptors are assessed in comparison with the well known texture descriptors, local binary patterns (LBP) and Gabor filtering. The results illustrate that the proposed method has superior performance in those cases where the image contains blur and is slightly better even with sharp images.
In the field of texture recognition, local binary patterns (LBP) @cite_67 is one of the most commonly used texture description approaches. Besides texture recognition, LBP based texture description has been applied to other vision tasks, including face recognition @cite_85 , gender recognition @cite_33 and person detection @cite_60 . The LBP descriptor works by thresholding intensity values of a pixel around its neighborhood. The threshold is computed from the intensity of each neighborhood's center pixel. A circular symmetric neighborhood is employed by interpolating the locations not exactly at the center of a pixel. A variety of LBP variants have been proposed in literature, including Local Ternary Patterns @cite_57 , Local Binary Pattern Variance @cite_47 , Noise Tolerant Local Binary Patterns @cite_35 , Completed Local Binary Patterns @cite_7 , Extended Local Binary Patterns @cite_28 and Rotation Invariant Local Phase Quantization @cite_34 . In addition to the introduction of different LBP variants, the fusion of LBP descriptor with color features have also been investigated in previous studies @cite_1 @cite_74 .