Texture Object Segmentation Based on Affine Invariant Texture Detection
Jianwei Zhanga, * and Xu Chenb, Xuezhong Xiaob
b

a
Beijing University of Posts and Telecommunications, Beijing, 100876, China
Nanjing University of Posts and Telecommunications, Nanjing, 210028, China

ABSTRACT
To solve the issue of segmenting rich texture images, a novel detection methods based on the affine invariable principle is proposed.
Considering the similarity between the texture areas, we first take the affine transform to get numerous shapes, and then utilize the
KLT algorithm to verify the similarity. The transforms include rotation, proportional transformation and perspective deformation to
cope with a variety of situations. Then we propose an improved LBP method combining canny edge detection to handle the boundary
in the segmentation process. Moreover, human-computer interaction of this method which helps splitting the matched texture area
from the original images is user-friendly.
Keywords: texture detection, affine invariant, image segmentation

(a)

(b)

Figure 1 Selecting the tiny area and deriving the global similar texture content.

1. Introduction
For most applications in the area of image processing and
computer vision, image segmentation is an indispensable part. It
usually appears in the preprocessing step, such as image
recognition and comprehension. And the performance of
subsequent steps will be affected by its output enormously. Thus,
image segmentation technology has attracted a lot of attention. In
the last several decades, there are numerous remarkable
achievements, which could be roughly separated into two
categories including the image segmentation techniques in the
spatial [13,14] and frequency domains [10,20,21]. However, due
to the complexity of their backgrounds, none of these technologies,
particularly the segmentation techniques aiming at image objects
with rich textures, are practical.
In this work, we focus on the segmentation of images with the
rich texture regions. To achieve a fast method, we first select a
small area as the prior knowledge. Considering the specific region
possesses the self-similarity among the texture elements, then we
propose a method which can detect the similar texture area based
on affine invariant principle. We use the method of the translation,
rotation, scale and perspective transformation to detect the texture
target area. Meantime, for those images with strong three-

dimensional information, the perspective transformation could
handle the images well comparing with the affine transformation.
We have got a specific texture region after all the
transformations. To make the edge smoother, we first utilize the
canny edge detection to get the peripheral line of the texture area.
Then enhance the local binary pattern method to make the
boundary more accurate. Segmenting automatically is the mainly
advantage of our algorithms. We don’t need redundant user
interactions while processing, while the grab cut algorithms [16]
may need more interactions. Extensive experiments on different
images including clothing, construction, Knitwear and other rich
texture objects show that the proposed algorithms significantly
outperforms the state-of-the-art. As we can see, with the trend of
online shopping for goods like clothes, the algorithms could be
useful to change the texture of the clothes. And the experiment
result for the long-time consumption of segmentation is even better.
2. Related Work
For those color texture images, Kan-Min Chen [4] has proposed
the method of utilizing the feature distribution. It uses color and
local edge pattern as the extract texture descriptors to measure the
homogeneity for texture region. By quantizing the color image,
they then fed it into a color texture segmentation algorithm to
segment the image into diﬀ erently textured regions. Qing Zhang

* Corresponding author. Tel.: +86-18146539952; e-mail: zhangjianweibupt@bupt.edu.cn
1

[5] also used the color and texture cues to compute saliency by
integrating the background-based and foreground-based patches to
present a new bottom-up salient object detection method. Q Zhao
[23] has employed the EM variants into a color-texture image
segmentation algorithm. like X. Liu [17] and J. Yuan [18]
proposed, Local Spectral Histogram is also useful in the Gabor
filter. The Gabor filter in the bank is also tuned to detect patterns
of a specific frequency and orientation [24]. J Shi [19] has utilized
the oriented Gaussian filter in the segmentation. Combining the
filters like Cimpoi M has raised the proposal [22] could apply to
the texture recognition, description, and segmentation. There may
be several distinct areas in a picture, and the clustering algorithms
could be a way to solve the segment problem [11, 12, 25].

Pij (s×s)

mi (s×s pixels)

M
Figure 2. Explanation of some blocks

To express the self-similarity between the two blocks, we define
the f(Ii, Ij) as the value of self-similarity. Mathematically, it can be

Though the grab cut algorithms [16] could produce a better
segment outcome, the multiple interactions makes operations
cumbersome. The automatic segmentation is practicable in some
fields like the medical and so on [28-30]. Meanwhile some
scholars have put forward applying the machine learning
algorithm and deep learning methods into the segmentation [3137]. We could train the database to segment the interest region by
the learning algorithms. But the algorithm is an input-output
process which has not considered the content of regions. V. B.
Surya Prasath [6] proposed a flexible segmentation approach using
multi-channel texture and intensity in a globally convex
continuous optimization framework. And the local feature density
function is applying to the luminance-chromaticity decomposition.

expressed as Eq. 1:
i+s j +s

f (Ii , I j ) = åå|| mi (i, j) - Pij (i, j) ||2
i

Eq. 1

j

Finally, we can segment the texture region based on the
detection work. In the detection work we introduced an improved
local edge pattern method. And the smooth edge is achieved by
combing the canny operator, which is showed in detail in section
3.2. All algorithm flow chart is performed as Figure 3 follows:

As we can see, the rich texture area has a high degree of local
similarity. The features are invariant to image scaling, translation,
and rotation, and partially invariant to affine or perspective
transformation. David G. Lowe [2] has previously proposed the
basic algorithms SIFT which used the local image features. Wang
[3] has redefined the transform map to encode local affine
deformations. They chose each block a 2×3 matrix D that maps
any image point p=(x, y, 1)( to position Dp in the epitome. They
match the similar patches which were computed previously
through affine transformation with the chose block. To measure
the degree of self-similarities, they defined and calculated the error
e based on the sum of squared differences.
3. Algorithm
In this paper, we introduced a new method to search similar
contents based on the affine invariant principle. We select a block
‘M’ as the prior knowledge to derive the similar regions from the
original image. To improve the efficiency and practicality of the
algorithm, we split up the block into some small blocks named ‘mi’
(s×s pixels). These small blocks will be as references for the
match searching. to find the similar areas (Figure 2), we search
each block ‘Pij’ (s×s pixels) which traversed in the input image
(original I) spaced every 3 pixels. In the beginning of the all work,
we utilize the color histogram and feature corner point to delete
the apparent wrong blocks to exceed the operate rate.

Figure 3. the algorithm flow chart

Then, we mainly divide it into two parts. The first step is to
detect the similar texture region within the affine invariable
principle. We transform the block ‘mi’ into numerous shapes. And
perform match searching by using the Kanade-Lucas-Tomasi
(KLT) feature tracker [8], which is widely used to track the which
is widely used to track the movement between sequences of frames
[Lucas and Kanade 1981; Shi and Tomasi 1994]. Due to the KLT
algorithm is based on the basis of small movements, we should
initialize the KLT feature tracker with a good state first.

3.1. Texture detection within affine invariable principle
a.

KLT feature match
To measure whether the two blocks ‘mi’ and ‘Pij’ is matched,
we use KLT algorithm for match searching. And we detect the
feature points in the ‘mi’ by using the corner detector, initialize
separate KLT feature tracker to track these feature points, then
judge whether they still exist in frame. We use the corner detector
to detect the feature points in the ‘mi’, and initialize separate KLT
feature tracker to track these feature points, then judge whether
they still exist in the ‘Pij’ and record the information of the area
around the correspondence points. We stipulate that if the
2

proportion of unmatched feature point is less than 0.15, it is
regarded as the matched grid.

block ‘m1’

b.

Screening based on color histogram and feature corner points
The first approach is to filter out the areas differentiate from
‘mi’ apparently, which can be achieved by the HSV color
histogram. Compared with the RGB color space, the former can
retain more information in the post-processing. So, firstly we
compute the HSV color histograms of ‘mi’ and ‘Pij’, remove those
areas whose adjacent color histograms are too different from ‘mi’.
Then we have gained the rough area ∅(′P23 ′) by traversing the all
sets of grids. That means that all ‘mi’ are corresponds to the area
∅(′P23 ′) . Meanwhile we skip some blocks according to the
divergence in the number of feature points between ‘mi’
and ∅(′P23 ′). And we find that the first screening is necessary and
useful for improving the speed. For ‘Pij’ in the remaining areas,
we label it which means it is similar to ‘mi’.

Original I

Result I

Figure 5. The green rectangles in original I represent block ‘M’. The red
marks in the Result I’ mean that it matches block ‘m1’

Rotation and Scale We have gained the second round of
screening by translation and mirror reflection. But there still be
some areas which have not been found. Now we keep on judging
‘mi’ whether match the leftover grids through rotation and scaling
transformation. We not only rotate the ‘mi’ spaced 5 degrees, but
also consider the situation of minification and magnification. As
the image pyramid may lose the information at high level, we
direct use the scaling within the affine transformation. We only
think over the scale ranging between 0.8 and 1.3. And the result is
as Figure 6 shows.

c.

Transformation
We have got the rough region which exists some wrong labels
after the first round of screening. To make the results more
accurate, we must check every block in ∅(′P23 ′) to judge whether
it is similar with ‘mi’. and we use KLT method to detect the
similarity between them. At the same time, many areas in ∅(′P23 ′)
have been transformed and lost some feature points. We need to
make some transformations to ‘mi’, and then use it as the input of
the KLT feature tracker. Not only have we considered some simple
transformations, but also complicated transforms like affine
transformation and perspective transformation are included. The
area obtained by affine transformation and perspective
transformation combining proportional transformation as Figure 4
shows.

Figure 6. The blue marks mean that it matches block ‘mi’ which has been
transformed by rotation and scale.

block ‘mi’

Perspective Transformation For some images which possess the
3-dimensional information, only considering the affine
transformation is not enough. So we introduce the perspective
transformation to dispose the problem that affine transformation
can’t solve. And because the affine transformation is subset of the
perspective transformation, we only need to think over the
perspective transformation to make it comprehensive. We have
considered the main four aspects: bend inward from above, below,
left, right direction. Meanwhile, the scaling and rotation is
included in every direction. Then we can get the transformed
block δ(‘mi’) and use it to initialize the KLT, which is still as a
measure to detect δ(‘mi’) whether match the transformed region
(Figure 7).

Figure 4. Transformation area

Transform steps as follows:
Translation and Mirror Reflection We make the specific
transformation for each ‘mi’ block first, than we use the
transformation blocks as the input of KLT to match the remaining
area in the original I and mark the similar region (Figure 5). For
the mirror reflection, the KLT match result is the same as the
translation because it don’t change the feature points and fearture
points of surrounding area.
Figure 7 The green mark means that it matches block δ(‘mi’).

In fact, we perform the above mentioned searching operation in
turn. While performing in the next round, we jump over the grids
labeled in previous processes. If an unlabeled block’s neighbors
are labeled, it has the similar content in most case and whether it
is marked will not play a significant impact on the segmentation.
To speed the next round detection, we can label it as right. As the
Figure 8 shows, we will label the block in the default way.

3

3.2. Segmentation based on the detection

Edge Processing We have got the rough edge of the region, and
then we need to make the edge smoothly. We simply use canny
edge operator [15] and low-pass filter to achieve it (Figure 11).

For those labeled grids which have been verified to possess the
similar content with ‘mi’, we devise a form to record the positions
of them and set white to them on a binary image initialized with
black, obtaining an obvious dividing line. Then we commence to
handle the error areas and the grids near the line.
Delete Apparent Error Areas For some areas which existed
detection error like yellow block in Figure 9, we adopt a method
in probability to remove them by analyzing the surrounding grid.
It is obvious that there are 81 square grids in a result of traversing
spaced 3 pixels. We define the rate "α" for the ratio of the grid
labeled in all 81 grids to all grids. And once the value of α is less
than 0.05. We can assume that the yellow block is a wrong labeled
area that we will delete it.

Figure 11 The filter outcome

4. Experimental Results
We select the block size of ‘mi’ as 12×12 pixels, the same for
‘Pij’. We have tested our algorithms upon other images with rich
texture (see Figure 12). The wall in the first row is from a realistic
picture, and is made of countless small bricks. It’s easy to draw the
whole similar content from the tiny selected region. The textile in
the second row is the same way to segment the area of bottom right
corner. The third shows two stage zipper which exist some wrong
blocks labeled by red crosses in the middle image. It should be that
the background is not too complicate in the black edge. But it will
disappear a part of wrong blocks in the segment process. We are
basically able to separate the coat out at the model who wears a
black and white coat picture in the fourth row. The starfish is
almost segment well in the fifth row. The cobra in the last row has
strong space sense that is possible to detect the texture by
perspective transformation.

(a)

(b)

(c)

Block mi

Figure 8 Speeding the detection

Figure 9 Detect error areas

Edge Feature We have got a new idea from the Local binary
pattern (LBP) [9] operate on the 3 ×3 matrixs . The operator
calculated the mathematical relationship between the eight
neighbors and the central pixel. As we traversal the image by the
block ‘Pij’, it’s suitable that Pij could be divided into some
3×3 smaller block. We can assumed that the central pixel as X,
the eight neighbors as xi and take an example that the Pij as
12×12 block. And the formula is as Eq. 2:

The detection process is vital to determine the integrity of the
texture region, and segmenting the label region is not difficult to
work out. Speeding up the process is feasible in the future research,
especially for those composite textures images.

8

Eij =å (x i - X)2

Eq. 2

the value of i and j is 0,1,2,3, where Eij means the values of
3 ×3 smaller block . It’s obvious that the value of Eij will get
higher than neighbors [Figure 10.a.b] if the texture edge go
through the block. Now it’s easy to get the two-dimensional array
E [Figure 10.c], we define the

ti

as the mutation rate which

means the amount of change between Eij adjacently in the i-th
row.(Eq. 3)

ti =

| E ij +1 - E ij |

´

| E ij +1 - E ij |

max(E i ) - min(E i ) min(E ij +1 , E ij )

Eq. 3

39

42

24 28 187

41 38

41

41 156 174

1
8
179 212 2031 56

37 40

39

37 165 168

197 3218 23 68

45

（a）

（b）

230 203 1

185 5432 83

7
3

（c）
Figure 10 the description of local edge feature

4

Block selected

4500

Translation
And Mirror
Reflection

4000
3500
3000
2500

Rotation

2000
1500
1000
500

Perspective

0

12

13

14

Block size

15

16

Figure 15 The relationship between the block selected and block size

5. Conclusion
We have proposed a novel affine invariant method for detection
and segmentation in rich texture images. Our algorithms have
following advantages. First, for those rich texture images, we have
proposed a new solution which will not be affected by the complex
edges and contents compared with previous methods. Second, our
algorithms will not be affected by the existence of shadow or light
in the background based on the affine invariant. Third, we don’t
need to do many interactions since it is calculated automatically.
However, these algorithms also have limitations, due to traversing
and calculating the all sets of grid, the execution time is relative
long. The process can be accelerated by parallel-computing with
GPU in the future.

Figure 12 additional segmented images

Meanwhile, we have tested some pictures above by the Grab
cut algorithms, the outcome images are performed as follows
(Figure 13):

Acknowledgments
This work is supported by the Jiangsu province college student
innovation and entrepreneurship plan (program number
SYB2016007). And thanks for Shuo Chen`s valuable comments
which improves the quality of this manuscript.

Figure 13 images segmented by grab cut algorithms

Here is the comparison of final outcomes for two algorithms
(Figure 14):

References
[1] Jack Mezirow. 1978. Perspective transformation. Adult Education, vol. 28,
2: pp. 100-110.
[2] Lowe, David G. “Object Recognition from Local Scale-Invariant Features”.
Proceedings of the International Conference on Computer Vision. 2(1999),
pp. 1150–1157.
[3] Huamin Wang. 2008. Factoring Repeated Content Within and Among
Images. ACM Transactions on Graphics, Vol. 27, No. 3
[4] Kan-Min Chen, Shu-Yuan Chen. Color texture segmentation using feature
distributions. Pattern Recognition Letters 23 (2002) 755–771.
[5] QingZhang, JiaJun Lin,et. Salient object detection via color and texture
cues. Neurocomputing 243(2017)35-48.
[6] V. B. Surya Prasath, K.Palaniappan, Guna Seetharaman. Multichannel
Texture Image Segmentation using Local Feature Fitting based Variational
Active Contours. (2012) ACM 978-1-4503-1660-6/12/12.
[8] Jae Kyu Suhr. 2009. Kanade-Lucas-Tomasi(KLT) Feature Tracker.
Compute Vision, Fall 2009, Yonsel Univ.
[9] K. P. Kaur, C. Singh, E. W. Bhullar. Color Image Retrieval Using Color
Histogram and Orthogonal Combination of Linear Binary Pattern. Proceedings
of the 2014 Indian Conference on Computer Vision Graphics and Image
Processing Article No. 48.
[10] J. Chi, M. Eramian. Enhancing textural differences using wavelet-based
texture characteristic morphological component analysis: A preprocessing
method for improving segmentation. Computer Vision and Image
Understanding (2017), http://dx.doi.org/10.1016/j.cviu.2017.01.006.
[11] Sujaritha, M., and S. Annadurai. "A three-level clustering algorithm for
color texture segmentation." Proceedings of the International Conference and
Workshop on Emerging Trends in Technology. ACM, 2010.
[12] Clausi D A. K-means Iterative Fisher (KIF) unsupervised clustering
algorithm applied to image texture segmentation[J]. Pattern Recognition, 2002,
35(9): 1959-1972.
[13] Zhang J, Tan T, Ma L. Invariant texture segmentation via circular Gabor
filters[C]//Pattern Recognition, 2002. Proceedings. 16th International
Conference on. IEEE, 2002, 2: 901-904.

Figure 14 the comparison between the two algorithms

And it’s easily find that the traditional segment interaction
algorithms, such as the grab cut algorithm, may not get a good
effective graph after only one interaction, even the final effect after
several interactions is not ideal. Like Figure 13 images segmented
by grab cut algorithms, if the zipper and the cobra are completely
separated, it will take multiple interactions. As Figure 14 the
comparison between the two algorithms shows, the final segment
is uncompleted by grab cut. However, our algorithm could achieve
better experimental effect as well as higher degree of automation
than the grab cut algorithms.
Block Size The graph in Figure 15 explores how the number of
blocks selected varies as a function of the block size. Owing to the
selection of the size of the block has an effect on the detection. So
the comparison of the size of the different blocks makes sense. As
the block size increases, the selected blocks get fewer because the
content of the block will get more complexity. On the other hand,
the number of blocks selected by the perspective transformation is
more than other transformation. It’s the result of the image
containing the 3-dimensional information. We choose the Figure
4(200×300 pixels), set the quantity of the “mi” as 6 as an example
and the analysis as follows:

5

[14] Olowoyeye A, Tuceryan M, Fang S. Medical volume segmentation using
bank of Gabor filters[C]//Proceedings of the 2009 ACM symposium on
Applied Computing. ACM, 2009: 826-829.
[15] Cho H, Sung M, Jun B. Canny Text Detector: Fast and Robust Scene Text
Localization Algorithm[C]//Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition. 2016: 3566-3573.
[16] Rother C, Kolmogorov V, Blake A. "GrabCut": interactive foreground
extraction using iterated graph cuts[C]// ACM SIGGRAPH. ACM, 2004:309314.
[17] X. Liu and D. Wang, “Image and texture segmentation using local spectral
histograms,” IEEE Transactions on Image Processing, vol. 15, no. 10, pp.
3066–3077, 2006.
[18] J. Yuan, D. Wang, and A. M. Cheriyadat, “Factorization-based texture
segmentation,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp.
3488–3497, 2015.
[19] J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE
Transactions on pattern analysis and machine intelligence, vol. 22, no. 8, pp.
888–905, 2000.
[20] S. Arivazhagan and L. Ganesan, “Texture segmentation using wavelet
transform,” Pattern Recognition Letters, vol. 24, no. 16, pp. 3197–3203, 2003.
[21] Ji H, Yang X, Ling H, et al. Wavelet Domain Multifractal Analysis for
Static and Dynamic Texture Classification[J]. IEEE Transactions on Image
Processing A Publication of the IEEE Signal Processing Society, 2013,
22(1):286-99.
[22] Cimpoi M, Maji S, Kokkinos I, et al. Deep Filter Banks for Texture
Recognition, Description, and Segmentation[J]. International Journal of
Computer Vision, 2016, 118(1):65-94.
[23] Zhao Q, Liao Z, Li J, et al. A Comparative Study on EM Algorithms for
Color-Texture Image Segmentation[C]// International Conference on
Computing Communication and NETWORKING Technologies. ACM,
2016:18.
[24] Olowoyeye A, Tuceryan M, Fang S. Medical volume segmentation using
bank of Gabor filters[C]// ACM Symposium on Applied Computing. DBLP,
2009:826-829.
[25] Ma X, Wan W, Jiao L. Spectral clustering ensemble for image
segmentation[C]// Genetic and Evolutionary Computation Conference, Gec
Summit 2009, Proceedings, Shanghai, China, June. DBLP, 2009:415-420.

[26] Nabizadeh N, Kubat M. Automatic tumor segmentation in single-spectral
MRI using a texture-based and contour-based algorithm[J]. Expert Systems
with Applications, 2017, 77:1-10.
[28] Nabizadeh N, Kubat M. Automatic tumor segmentation in single-spectral
MRI using a texture-based and contour-based algorithm[J]. Expert Systems
with Applications, 2017, 77:1-10.
[29] Thierry Bouwmans, “Traditional and recent approaches in background
modeling for foreground detection: An overview,” Computer Science Review,
vol. 1112, pp. 31 – 66, 2014.
[30] Tabb A, Medeiros H. Automatic segmentation in dynamic outdoor
environments[J]. 2017.
[31] V. Andrearczyk and P. F. Whelan, “Using Filter Banks in Convolutional
Neural Networks for Texture Classification,” Pattern Recognition Letters, vol.
84, pp. 63–69, 2016.
[32] Andrearczyk V, Whelan P F. Deep Learning for Biomedical Texture
Image Analysis[C]// Irish Machine Vision and Image Processing Conference.
2016.
[33] M. Cimpoi, S. Maji, I. Kokkinos, and A. Vedaldi, “Deep filter banks for
texture recognition, description, and segmentation,” International Journal of
Computer Vision, vol. 118, no. 1, pp. 65–94, 2016.
[34] P. Luc, C. Couprie, S. Chintala, and J. Verbeek, “Semantic Segmentation
using Adversarial Networks,” arXiv preprint arXiv:1611.08408, 2016.
[35] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
for biomedical image segmentation,” in International Conference on Medical
Image Computing and Computer-Assisted Intervention, pp. 234–241, Springer,
2015.
[36] Long J, Shelhamer E, Darrell T. Fully convolutional networks for
semantic segmentation[J]. IEEE Transactions on Pattern Analysis & Machine
Intelligence, 2015, 39(4):640.
[37] Chen L C, Papandreou G, Kokkinos I, et al. Semantic Image Segmentation
with Deep Convolutional Nets and Fully Connected CRFs[J]. Computer
Science, 2014(4):357-361.

6

