Fashion Landmark Detection and Category Classification for Robotics
Thomas Ziegler Judith Butepage Michael C. Welle Anastasiia Varava Tonci Novkovic Danica Kragic
cs.LG stat.ML









        

        






































Fashion Landmark Detection and Category Classification for Robotics
     Thomas Ziegler^1,2, Judith Butepage^2, Michael C. Welle^2, Anastasiia Varava^2, Tonci Novkovic^1 and Danica Kragic^2  ^1Thomas Ziegler and Tonci Novkovic,  ETH Eidgenössische Technische Hochschule Zürich zieglert@ethz.ch, tonci.novkovic@mavt.ethz.ch  ^2 Thomas Ziegler, Judith Butepage, Michael Welle, Anastasiia Varava, and Danica Kragic, KTH Royal Institute of technology         butepage,mwelle,varava,dani@kth.se  
        ^1 Department of Statistical Sciences, University College London, London WC1E 6BT, UK

    ^2 School of Mathematical Sciences, Queen Mary University of London, London E1 4NS, UK

^3 The Alan Turing Institute, The British Library, London NW1 2DB, UK 


=======================================================================================================================================================================================================================================================================================================================================================================================================================================




    
    
    Research on automated, image based identification of clothing categories and fashion landmarks has recently gained significant interest due to its potential impact on areas such as robotic clothing manipulation, automated clothes sorting and recycling, and online shopping. Several public and annotated fashion datasets have been created to facilitate research advances in this direction. In this work, we make the first step towards leveraging the data and techniques developed for fashion image analysis in vision-based robotic clothing manipulation tasks.
    We focus on techniques that can generalize from large-scale fashion datasets to less structured, small datasets collected in a robotic lab. Specifically, we propose training data augmentation methods such as elastic warping, and model adjustments such as rotation invariant convolutions to make the model generalize better. Our experiments demonstrate that our approach outperforms state-of-the art models with respect to clothing category classification and fashion landmark detection when tested on previously unseen datasets. Furthermore, we present experimental results on a new dataset composed of images where a robot holds different garments, collected in our lab. 



Vision for Robotics, Cloth/Garment Classification, Data augmentation, Generalizations with Convolutional Neural Networks






§ INTRODUCTION





As the interest for fashion items increases in online shopping and e-commerce, the need for automated image analysis in the fashion industry is growing. This application area requires many tasks to be automatized, such as clothing category classification, fashion landmark detection, image retrieval and similarity based recommendations. Following the creation of large-scale fashion datasets <cit.>, significant progress has been made in fashion image analysis. Deep learning based models have achieved significant performance gain in clothing category classification <cit.>, item recommendation <cit.>, and retrieval <cit.>.

However, for robotic clothing manipulation,  the collection of large-scale datasets proves to be more difficult. Robotic clothing manipulation includes tasks such as clothing category classification <cit.> and tasks that require fashion landmark detection, such as grasp point detection <cit.>, folding <cit.>, sorting <cit.>, unfolding <cit.>, and dressing <cit.>. Compared to retail applications, the vast majority of existing work on clothing category classification in robotics uses custom datasets for evaluation. These datasets are often limited in the number of images and contain only a small number of different categories. 





In this work, we identify two tasks that are common to both retail and robotic applications, namely clothing category classification and fashion landmark detection. While the fashion industry often considers structured data, such as a human wearing clothes facing the camera, the data in robotic applications is less structured and can contain images of upside-down, crumpled clothing items. We build upon the progress made in fashion image analysis and propose a network architecture and training procedure on a large-scale fashion dataset DeepFashion <cit.>. Our model is capable to generalize well to the noisy, poorly controlled conditions encountered in robotic clothing manipulation. 

DeepFashion dataset contains over 280000 images of clothing separated into 46 categories and annotated with 4∼8 landmarks per item. In order to resemble the more challenging clothing configurations encountered in robotic manipulation, we introduce elastic warping, a novel image augmentation method. It uses random displacement fields to create authentic looking clothing configurations. Our proposed model incorporates rotation invariance and attention mechanisms in order to handle difficult configurations faced in robotic manipulation, such as random orientation.  

The performance of our model is evaluated extensively on different publicly available datasets. One of these is a small-scale dataset created by us, which contains real world images typically encountered in a robotic manipulation task, where robot arms are visible in the image as shown in Figure <ref>. In contrast to other, state-of-the-art methods, our approach is able to generalize to new, much smaller datasets without additional fine-tuning. 

We illustrate a potential application scenario of our model by performing landmark detection on a garment that is being folded by a robot. We demonstrate that the landmarks are successfully detected even when the garment is  partially occluded by the robot (see supplementary videofn:code). 


The landmark detection is very stable despite the robot occluding parts of the garment during the manipulation.
Therefore we believe the proposed method is a first step for robotic clothing manipulation tasks that require basic visual information such as category classification and landmark detection.   


The contributions of our work is fourfold:

(i) We propose a novel deep learning based network for clothing classification and landmark prediction specific for robot manipulation. To the best of our knowledge, this is the first deep learning based computer vision model for clothing manipulation trained solely on RGB images. (ii) We introduce elastic warping for landmark detection, a new data augmentation method that is capable of resembling more challenging clothing configurations which are not encountered in standard datasets. (iii) We provide extensive experimental results on different datasets which identify a lack of generalization to novel datasets of other state-of-the-art fashion networks. (iv) We created a small dataset, containing real world images of clothing in a realistic robotic manipulation environment. Additionally, we annotated landmark position in the CTU dataset <cit.>.  The annotations, the in-lab dataset and the implementation are publicly available [  <https://github.com/ThomasZiegler/Fashion_Landmark_Detection_and_Category_Classification>].




§ RELATED WORK



We survey the work from the computer vision and robotic communities related to our work.




 §.§ Fashion Networks


Fashion image analysis has drawn increased attention in the field of computer vision due to its impact on e-commerce and online shopping. Several deep learning networks evolved from this trend for the different analysis tasks of clothing recognition <cit.>,  recommendation <cit.>,  retrieval <cit.>, and fashion landmark localization <cit.>. In our work, the focus is on category classification and landmark localization.

  <cit.> propose a multi-branch network for simultaneous classification, retrieval and landmark localization. In <cit.>, the same authors propose a combination of three cascaded networks for a gradual refinement of landmark localization.  <cit.> use recurrent spatial transformers in combination with selected dilated convolutions to predict landmarks in unconstrained scenes. More recently,  <cit.> proposed a deep fashion grammar network for combined clothing category classification and landmark localization. The network encodes two attention mechanisms: landmark-aware and category-driven attention. A similar network was proposed by  <cit.> which has an increased resolution in the predicted heatmaps for the landmarks and uses a unified attention mechanism instead of two separate streams.





 §.§ Computer vision for robotic clothing manipulation


Robotic clothing manipulation is a well established research area with pioneer work going back more than twenty years <cit.>. It can be seen as a pipeline of several consecutive steps to bring an item of clothing from an unknown state into a desired one (folded or sorted) <cit.>. A broad overview over methods used for visual grasp point localization, classification and state recognition is given in <cit.>. 

Compared to the structured data used for retail applications, the image data used in the robotics community is of a different nature. The items are either lying in a spread or crumpled state on a flat surface <cit.> or they are in a hanging state when grasped by a robotic gripper <cit.>. 

A major difference to vision applications in the fashion industry is that the robotics community has mostly focused on task specific, handcrafted feature extraction, such as edges and corners <cit.> and wrinkles <cit.>. Additionally, due to the 3D nature of the manipulation task, the use of physics and volumental simulators is more common in robotics <cit.>. 

Some recent methods <cit.> use CNN instead of handcrafted features to classify hanging items of clothing. All models are shallow, containing a few convolutional layers followed by a few fully connected layers.  <cit.> use a CNN to create a global feature vector from depth maps. The CNN is trained on a large dataset with common 3D objects. For classification on a smaller clothing dataset, the CNN is used to extract features which are passed to a Support Vector Machine.


Our proposed network has a similar architecture as the networks proposed in <cit.>, but it has been extended to a more challenging clothing configurations encountered in robotic applications. To the best of our knowledge, this is the first deep learning based network designed as part of a robotic clothing manipulation pipeline that only uses RGB images.  












 


§ METHOD


In this section we formulate the problem and introduce two augmentation methods to resemble clothing configurations encountered in robotic clothing classification and landmark localization tasks. Finally we give a detailed description of the proposed network.  






 §.§ Problem Formulation


Given an image I the goal is to simultaneously predict the landmark locations L and category classification C. The landmarks are defined as L= {(x_k, y_k)}_k=1^n_L, where (x_k, y_k) is the kth pixel coordinate position in I and n_L the total number of landmarks per image. 
 
The category classification C ∈ [0,1]^n_C satisfies  ∑_i=1^n_C C_i = 1, where n_C is the number of categories depending on the used dataset.  






 §.§ Image augmentation


 

In order to make the available fashion datasets more representative for robotic applications we propose two types of data augmentation: image rotation and elastic warping. One challenge is to augment an image together with its landmarks. We define the image before transformation as input image I and the image after the transformation as transformed image I. In both cases w, h stand for the width and height of the image respectively. 

 The transformation can be represented as a mapping of the pixels, ∀ (x̃, ỹ) ∈ [1,w]× [1,h]:
 
    I(x̃,ỹ)=  I(x(x̃,ỹ), y(x̃,ỹ)).

Where x,y are the pixel location in the input image I and x̃,ỹ the pixel location in the transformed image I. The clothing landmark locations L= {(x_k, y_k)}_k=1^n_L are a set of n_L specific pixel coordinates in the input image I. 

When x(x̃,ỹ) and/or y(x̃,ỹ) are non-integer, interpolation is needed. We apply the commonly used bilinear interpolation <cit.> in such a case.






  §.§.§ Rotation


A rather simple but powerful augmentation is image rotation. Rotating images with a small angle is often used to increase the performance in classification and/or detection tasks <cit.>. When items of clothing lie on the ground or on a table, they can be in any orientation. We hence randomly sample an angle θ in the range [0,2π] for each rotation.
 
 
 


 
 

  §.§.§ Elastic Warping


To resemble the distortion of loose items of clothing, we propose an elastic warping method. The method is similar to the elastic deformation proposed in <cit.> but is further extended to produce realistic, task-specific images and to allow for landmark detection. 

The deformation is created by generating two random displacement fields Δ x(x̃, ỹ) and Δ y(x̃, ỹ). The whole augmentation is performed in four steps:

First: Sample n_S pixel positions uniformly in the transformed image:  S = {(x̃_i, ỹ_i) }_i=1^n_S.


Second: For each pixel location in ∀ (x̃_i, ỹ_i) ∈S sample a random displacement from a uniform distribution 𝒰(-α, α):

    Δ x(x̃_i, ỹ_i) ∼𝒰(-α, α),  Δ y(x̃_i, ỹ_i) ∼𝒰(-α, α).

All other entries in the displacement fields are set to 0.

Third: Convolve the two displacement fields with a Gaussian filter G,  ∀ (x̃, ỹ) ∈ [1,w] × [1,h] :

    Δx̅(x̃, ỹ) =  Δ x(x̃, ỹ) *  G(x̃, ỹ) 
      
    Δy̅(x̃, ỹ) =  Δ y(x̃, ỹ) *  G(x̃, ỹ)

where * denotes the convolution operator.   and G(x̃, ỹ) is a Gaussian filter with variance parameter σ.


Fourth: Use the smoothed displacement field to create the transformed image, ∀ (x̃, ỹ) ∈ [1,w] × [1,h] :

    I(x̃,ỹ) =  I(x̃ + Δx̅(x̃,ỹ)_x(x̃,ỹ), ỹ + Δy̅(x̃,ỹ_y(x̃,ỹ)))


The strength of the distortion can be adjusted by the number of initially displaced pixels n_S, the scaling of the uniform distribution α and the smoothness of the Gaussian filter σ. We use n_S=3, α=500 and σ=40 in our experiments. Figure <ref> shows some examples when using this configuration. 









Landmark warping
The displacement fields indicate where a pixel in the transformed image was located in the input image. Due to the random nature of these fields no inverse exists. That means that it is not trivial to know if/where the pixels of the input image are found in the transformed image. This is problematic for landmark warping, since their location is only defined for the input image. In the following we describe an efficient method for retrieving the landmark position in the transformed image.

For every landmark position L_k = (x_k, y_k) we find n possible pixels in the transformed image I which originated at or near the position of the landmark in the input image I: 

    X = *argmin-n_∀ (x̃, ỹ) ∈ [1,w] × [1,h]sort |x̃ + Δx̅(x̃,ỹ) -x_k | 
    Y = *argmin-n_∀ (x̃, ỹ) ∈ [1,w] × [1,h]sort |ỹ + Δy̅(x̃,ỹ) -y_k |,

where *argmin-n returns the n smallest values from a sorted set. Note that both X and Y contain coordinate pairs (x̃,ỹ). The value of n depends on the image size and the chosen parameters n_S, α, and σ in the elastic warping. We use n=200 in our experiments.
To get the transformed landmark L_k  we need to find the coordinate pair (x̃^*,ỹ^*) that is either present in both X and Y or the coordinate pair in X with closest neighbor in Y.



We use the fact that the pixel coordinates are unique integer values and create a hash table for all coordinate pairs in one set. In the following, one can search for each pair in the other set if a key exist in the hash table, which reduces time complexity for existing coordinated pairs to 𝒪(n).

If the hash table does not return a valid value, no exact match exists in X and Y. In this case, one can create a kd-tree for all coordinate pairs in Y and use kd-tree search <cit.> to find the nearest neighbor for the coordinate pairs in X. The average search time for kd-tree search is 𝒪(n log n).







 §.§ Network Architecture


The main network architecture is loosely based on the VGG-16 <cit.> network structure similar to the networks proposed in <cit.>. The structure can be seen in Figure <ref>. Compared to the base VGG-16 network, several structural changes are included: rotation invariance layers (Section <ref>), a landmark localization branch (Section <ref>) and attention branches for classification (Section <ref>). As many components are inspired by prior work, we focus the discussion on the main components and direct interested readers to the Appendix for detailed network structure descriptions. 







  §.§.§ Rotation invariance


Orientation variation occurs more often in robotic clothing classification images than they do in fashion images. 
In order to account for this, we replace the 2D convolution in the conv1 to conv4 layers with A-ORConv. They produce enriched feature maps with the orientation information explicitly encoded <cit.>. 


In our network (Figure <ref>), we use the A-ORConv with four orientation channels (N=4). We use the same filter size and the same number of total channels when replacing the standard 2D convolution in the conv1 to conv4 layers. This means that the effective number of parameters of the A-ORConv is only a quarter of the normal convolution blocks. 
In order to create rotation invariant features, a Squeeze-ORAlign (S-ORAlign) layer <cit.> is used to find the main response channel. 

 






  §.§.§ Landmark Localization Branch


The landmark localization branch is the same as proposed in <cit.>. The branch structure is depicted in Figure <ref>.


The landmark localization branch can be trained separately from the classification. Given that the extracted feature map F is of dimension w_f× h_f× n_L, where w_f and h_f are width and height of the feature map and n_L is the number of landmarks, the ground-truth heatmap and the predicted heatmap for the kth landmark can be denoted by M_k ∈ [0,1]^w_f× h_f and M_k ∈ [0,1]^ w_f × h_f respectively. The landmark localization branch is trained using pixel-wise mean square differences,

    ℒ_LM =  ∑_i=1^n_B∑_k=1^n_L∑_x=1^w_f∑_y=1^h_f M_k^i(x,y) -  M_k^i(x,y) _2^2,

where n_B is the total number of training samples. The ground-truth heatmap M_k^i is generated by adding a 2D Gaussian filter at the corresponding location L^i_k. Given a sample i the predicted coordinates for the kth landmark L^i_k corresponds to the maximal value in the predicted heatmap,


    L^i_k ∈(x,y) ∈ [1,w_f] × [1,h_f]argmax M^i_k(x,y).

If there is more than one maximum per landmark one of them is chosen at random.






  §.§.§ Attention Branch


The attention branch can be seen as a union of spatial attention <cit.> and channel attention <cit.>. The attention learns a saliency weight map  A.
Inspired by the proposed attention modules in <cit.> the spatial attention itself contains two types of attention, a landmark attention A^L_spatial and a category attention A^C_spatial. 
Thus, the attention branch is designed as a three branch unit; two branches for the spatial attention A_spatial^L,  A_spatial^C (Figure <ref>) and one for the channel attention A_channel (Figure <ref>). These are combined in a factorized manner as explained below,

    A = ( A^L_spatial+ A^C_spatial) × A_channel.







Spatial Attention - Landmark
Clothing landmarks represent functional regions of clothing and provide useful information about an item. The predicted heatmaps { M_k}_k=1^n_L are used to guide attention to the functional clothing regions. The weight map is created by downsampling the predicted heatmaps which is followed by a max-pooling operation. This attention is learned in a supervised manner since it is directly derived from the predicted heatmaps. 




Spatial Attention - Category
Since the landmark attention only covers corner points of a clothing item, an additional spatial attention is used that focuses more on the clothing center. The category attention (Figure <ref>) is modeled using an U-Net structure <cit.>. 
The model learns by itself which regions of an image are important. This is in contrast to our landmark attention, where the groundtruth heatmaps M, which resemble the landmark attention, are provided during training.  




Channel Attention
The channel attention (Figure <ref>) is implemented via a Squeeze-and-Excitation block <cit.>. First a squeeze operation creates  S, an embedding of the global distribution of the channel-wise feature responses. Then an excitation operation is performed on the channel wise aggregated feature map to create the channel attention. Following the proposal in <cit.> a bottleneck is created using two fully-connected layers, with a reduction rate r.




Factorization
The factorization (Figure <ref>) is performed by multiplying the channel-wise feature responses in the spatial attention with the corresponding channel weights,
To refine the attention, an additional 1×1 convolution layer is added afterwards. This is motivated by the fact that the spatial and channel attention are not mutually exclusive but with co-occurring complementary relationship <cit.>.





  §.§.§ Output architecture


Given A, we weight the S-ORAlign features F,  U = ( 1+ A) ∘ F, where ∘ denotes the Hadamard product and 1 is a tensor. Hence, features where A(·, ·, ·) ∈ [-1, 0) are reduced and features where A(·, ·, ·) ∈ (0, 1] are increased. Our attention incorporates semantic information and global information into the network helping to focus on important regions in the images. The features U are then fed in to the conv5-1 layer. The rest of the network follows the VGG-16 structure. 












 


§ EXPERIMENTS


This section describes several different experiments to evaluate the performance of the proposed network and learning procedure. The section starts with a description of the different datasets we use, followed by the descriptions of the individual experiments and results.






 §.§ Datasets



In the following, we introduce all datasets used for training and/or evaluation.



  §.§.§ DeepFashion dataset


The DeepFashion: Category and Attribute Prediction Benchmark (DeepFashion dataset[<http://mmlab.ie.cuhk.edu.hk/projects/DeepFashion/AttributePrediction.html>]) <cit.> is a large collection of fashion images. It offers 289222 images collected from the Google image search engine and from shopping websites. The dataset contains 8 different landmarks (left/right collar, left/right sleeve, left/right waistline, and left/right hem), 46 clothing categories and 1000 clothing attributes. For each image a bounding box is provided. We use this dataset to train our network with our proposed augmentation methods and then perform inference on small-scale datasets.



  §.§.§ CTU Color and Depth Image Dataset of Spread Garments


The CTU Color and Depth Image Dataset of Spread Garments (CTU dataset[<https://github.com/CloPeMa/garment_dataset)>]) <cit.> is designed for testing and benchmarking garment segmentation and recognition. The dataset contains 1372 images of size 1280×1024 taken from a bird's eye perspective. There are 17 different items divided into 9 categories. Compared to the DeepFashion dataset the clothing items can be in any orientation and they contain not only flat spread but also wrinkled items. We manually labeled the landmark positions in each image. We use this dataset to train our network and evaluate its performance on more challenging clothing configurations typical in robotics. We also use it to evaluate the effect of our proposed augmentation methods when purely trained on the DeepFashion dataset.



  §.§.§ In-Lab Dataset


As a first step towards generalizing classification and landmark detection results to images that are typical for robotic tasks, we created a small dataset. It contains 117 images from 6 different clothing categories (Tank, Tee, Sweater, Hoody, Jacket, Jeans). Each item is hold by two robotic arms at predefined grasping points (shoulders and waist). This state can be reached with an autonomous unfolding process as proposed in <cit.>. The images are of size 960×720. Each item of clothing is captured in 9 different configurations of the robotic arms, such that they can overlap with the bounding box around the item. Furthermore, the background is not uniform and is partially cluttered. We annotated the images with the same landmarks as in the DeepFashion dataset and extracted a similar bounding box around each item. We use this dataset to evaluate the performance of our network on previously unseen items in a realistic lab environment.













       

      









      














      

      

      

     
      

     

      


      

       

      
      


      

      

       

         

       
      
      

     

     

     
    
      

      
     
      

       














 


 






 §.§ Pretraining on the DeepFashion Dataset



In this section we describe the pre-training details for the DeepFashion Dataset. Experimental results on this dataset can be found in the Appendix.



  §.§.§ Experimental Setup

We use the same settings as <cit.> for training and evaluation. In total 209222 images are used for training and 40000 images for validation. The final evaluation is performed on the remaining 40000 images. We use the normalized error (NE) <cit.> as the landmark localization error measure. This is the l_2 distance between the predicted and groundtruth landmark in normalized coordinates. For the category and attribute classification top-k classification accuracy is used.

The images are cropped using the provided bounding boxes. We train our model with and without our proposed data augmentation steps whereas the evaluation is always performed without augmentation. For implementation details, please see the Appendix.
 





 §.§ Experiments on CTU Dataset



In this section we present experiments on the  CTU Dataset. We perform two different types of experiments on the CTU dataset.
In the first experiment, we analyze the inference performance of our network, solely trained on the entire DeepFashion dataset. Special interest lies in the proposed data augmentation methods, since the clothing configurations differ from the DeepFashion images. In the second experiment we evaluate the performance of our network when trained and evaluated on the CTU dataset.



  §.§.§ Experimental Setup


It is important to note that the DeepFashion dataset has more than five times the number of categories than the CTU Dataset. Moreover the categories do not overlap exactly; if an item has a collar it is categorized as polo in the CTU dataset even though it might look more like a jacket than a polo shirt to a human. Furthermore, the CTU dataset distinguishes between long and short sleeve items, whereas DeepFashion does not (. tshirt and tshirt-long can both be in the Tee category). We combine the categories as follows: bluse=(Blouse), hoody=(Hoodie, Sweater), pants=(Jeans, Jeggins, Joggers, Leggins), polo=(Tee, Button-Down), polo-long=(Button-Down, Henley, Jacket), skirt=(Skirt), tshirt=(Tee), tshirt-long=(Cardigan, Sweater, Tee). Since the DeepFashion dataset does not contain any towels, we ignore them in these experiments. 

For the second experiment, we split the images randomly into a train, validate, and test set. (787, 240, 270 images).  

Both experiments are compared to the publicly available implementation of  <cit.>. For a fair comparison we train both models with the same augmentation methods (no augmentation, elastic warping (EW), rotation (R), and rotation  & elastic warping (R & EW)). For implementation details, please see the Appendix.










  §.§.§ Performance Evaluation


The results of landmark prediction and category classification on the CTU dataset with pre-trained models are shown in Table <ref> (top) and <ref> respectively. The benefit of training with rotated images becomes apparent. This is not surprising since the pictures of garments in the CTU dataset are taken in any possible orientation, whereas in the DeepFashion dataset all items of clothing are upright. Adding elastic warping increases the performance further for the landmark prediction for all cases except the one where training was performed on DeepFashion with no rotation. Additional results, presented in the appendix, show that adjusting the parameters of elastic warping can improve the performance further in some cases.  Since it does achieve the best performance in the top-3 accuracy, we believe that a extended tuning of the elastic warping parameters could  therefore increase the landmark prediction as well as classification performance. The overall classification accuracy of 85% shows that our model is able to generalize well even when trained on a dataset with significantly different configurations (. items of clothing worn by persons) compared to 56% reached by   <cit.>.



The results of the second experiment, trained and evaluated on CTU dataset, are shown in <ref> (bottom). Note that landmark predictions are significantly better when learned on the original dataset. 

The elastic warping seems to especially boost the performance in the case of no rotations. This is probably connected to the dataset composition and size as the EW augmented images boost the performance.

We omit the classification results since all the tested models achieve 100% accuracy. 

Adding elastic warping as a data augmentation method can improve the performance in most of the evaluated cases. Our network outperforms the one proposed by  <cit.> when trained with the same augmentation methods in both experiments. This indicates that state-of-the-art methods are likely to not generalize well to more challenging datasets. 













 §.§ Experiments on In-Lab Dataset


In this experiment we analyze the inference performance of our network, solely trained on the DeepFashion dataset, on the images taken in a lab environment.  For implementation details, please see the Appendix.








The results for landmark prediction and category classification are shown in Table <ref> and <ref> respectively. Some landmark predictions are exemplified in Figure <ref>. There is one item (a hoody) that is in almost always misclassified, except when using elastic warping with our model. Furthermore, the long sleeve t-shirt (Figure <ref> top row in the middle) is often classified as a sweater. With these two challenging items the best accuracy we achieve is 78.63%. Without these two items the accuracy increases to 93.33%. Due to the limited size of our dataset these two items have a significant impact. As the dataset is very limited in size, elastic warping can have a negative effect as well, as can be seen, for instance, in the drop in classification accuracy for the class Jacket. Further adjustments of the elastic warping parameters to achieve a higher similarity from the base images towards the task image might improve the results. Nevertheless, the combination of rotation and elastic warping leads to the best overall performance. The results of the landmark localization also show that our network is able to perform well even when an image contains parts of the robot.
This can be seen in the video provided in the supplementary materialfn:code, where a garment is being folded and the robotic arms occlude large parts of it. 

Elastic warping improves the performance for  and leads to the best  performance of our network in both the landmark localization and the classification. This indicates that the augmentation helps the models to generalize between the datasets. Furthermore, the model adjustments, such as rotation invariant convolutions, improve the state of the art methods <cit.> and make an important step towards there usage in robotic clothing manipulation. 




§ CONCLUSION AND FUTURE WORK


In this work we use a large publicly available fashion image dataset with data augmentation to train a network for garment classification and landmark detection for robotic manipulation application. This is the first work where a deep learning model trained on RGB images is used for clothing category classification and landmark detection for robotic applications. We show that our model is able to generalize to robotic specific item configurations which differ significantly from the training dataset. We achieve this by utilizing rotation and our newly proposed elastic warping augmentation method during training. 

After training, we use different datasets to evaluate the performance of our network.  We observe that other state-of-the-art methods, while producing excellent results on the training set, are not able to generalize as well to novel datasets. 







§ ACKNOWLEDGEMENTS

This work has been supported by the Swedish Foundation for Strategic Research, Swedish Research Council and Knut and Alice Wallenberg Foundation.



§ APPENDIX


In this appendix we give detailed account of the network structure and the experimental implementation details.



 §.§ Network structure


We here describe the individual parts of the network in more detail.



  §.§.§ Rotation invariance


In order to account for rotation, we replace the 2D convolution in the layers conv1 to conv4 of the VGG16 network with A-ORConv layers. These produce enriched feature maps with the orientation information explicitly encoded <cit.>. A-ORConv are an improvement of the ORConv initially proposed in <cit.>. These convolution blocks use A-ARF and ARF, respectively. Both are a 5D tensors of size n_O × n_I × w_f × h_f × N, where n_O is the number of output channels, n_I the number of input channels, w_f and h_f are the width and height of the filter and N is the number of filter orientations. This means that in ARF for each materialized filter, N-1 immaterialized rotated copies of the same filter are present. Therefore, during forward propagation one ARF produces a feature map of N channels with orientation information encoded. Depending on the orientation of the input image a different copy of the filter has the highest response. The improvement of A-ORConv over ORConv comes from reducing the risk of gradient explosion during training by updating the feature map with the mean value of the gradients from all its rotated copies instead of the sum of all gradients.

In our network, we use the A-ORConv with four orientation channels (i.e. N=4). We use the same filter size and the same number of total channels when replacing the standard 2D convolution in the conv1 to conv4 layers. This means that the effective number of parameters of the A-ORConv is only a quarter of the normal convolution blocks. 

In order to get rotation invariant features S-ORAlign, proposed in <cit.>, is used to find the main response channel. The S-ORAlign is inspired by the Squeeze-and-Excitation (SE) block <cit.>, first a squeeze operation is performed by global average pooling. Then the main orientation channel is found via a maximum function and finally all channels are spun such that the main response channel is in the first position.  The whole structure is depicted in Figure <ref>.

 



  §.§.§ Landmark Localization Branch


The landmark localization branch is the same as proposed in <cit.>. The branch structure is depicted in Figure <ref>. It uses transposed convolutions <cit.> to produce heatmaps for all landmarks. The transposed convolutions allow for an upsampling of the S-ORAlign features F ∈ℝ^28×28×512 back to the original input image size. Given the features F a 1×1 convolution is applied to reduce the number of channels in the feature map to F_L^(1)∈ℝ^28×28×64. Then three blocks of two 3×3 convolutions followed by a 4×4 transposed convolution are utilized. The padding and stride of the transposed convolution are 1 and 2, respectively. Hence, such a block upsamples the feature maps by a factor of two, at the same time the number of channels is reduced by a factor of two. Finally a 1×1 convolution with a sigmoid activation is used to convert the F_L^(4)∈ℝ^224×224×16 feature map into the predicted heatmaps M ∈ [0,1]^224×224×8.

The landmark localization branch can be trained separately from the classification. Let  M_k ∈ [0,1]^224×224 and M_k ∈ [0,1]^224×224 denote the groundtruth heatmap and the predicted heatmap for the kth landmark, respectively. The landmark localization branch is trained using pixel-wise mean square differences,

    ℒ_LM =  ∑_i=1^n_B∑_k=1^8∑_x=1^224∑_y=1^224 M_k^i(x,y) -  M_k^i(x,y) _2^2,

where n_B is the total number of training samples. The groundtruth heatmap M_k^i is generated by adding a 2D Gaussian filter at the corresponding location L^i_k. Given a sample i the predicted coordinates for the kth landmark L^i_k corresponds to the maximal value in the predicted heatmap,


    L^i_k ∈(x,y) ∈{1,…,224}×{1,…,224}argmax M^i_k(x,y).

If there is more than one maximum per landmark one of them is chosen at random.

 


 

 
 






 


  


 


  
  

 
 








 
 

  §.§.§ Attention Branch


The attention branch can be seen as a union of spatial attention <cit.> and channel attention <cit.>. The attention learns a saliency weight map A ∈ [-1, 1]^28×28×512 of the same size as the S-ORAlign features F ∈ℝ^28×28×512. Inspired by the proposed attention modules in <cit.> the spatial attention itself contains two types of attention, a landmark attention A^L_spatial and a category attention A^C_spatial. 

We learn the spatial and channel attention in a factorized manner,

    A = ( A^L_spatial+ A^C_spatial) × A_channel.


The attention branch is designed in a three branch unit; two branches for the spatial attention A_spatial^L,  A_spatial^C (Figure <ref>, Figure <ref>) and one for the channel attention A_channel (Figure <ref>). With the factorization (equation <ref>) combining them at the end, Figure <ref>. 




  
Spatial Attention - Landmark
Clothing landmarks represents functional regions of clothing and providing useful information about the item. The predicted heatmaps { M_k}_k=1^8 are used to get attention on the functional clothing regions. The weight map is created by downsampling the predicted heatmaps to the same size as the feature map in F, followed by a max-pooling operation.


    M' = {downsample_ 8  M_k}_k=1^8


    A^L_spatial(x,y) = max_k M'_k(x,y) 
      ∀ (x,y) ∈{1,…,28}×{1,…,28}

This attention is learned in a supervised manner since it is directly derived from the predicted heatmaps.



  
Spatial Attention - Category

Since the landmark attention only covers corner points of a clothing item, an additional spatial attention is used that focuses more on the clothing center. The category attention is modeled using an U-Net structure <cit.>. 
Given the S-ORAlign features F ∈ℝ^28×28×512 a 1×1 convolution is applied to convert the features into F_A^(1)∈ℝ^28×28×32. The U-Net consists of a contracting path that consists of two 4×4 convolutions with stride 2, which squeeze the features down to F_A^(3)∈ℝ^7×7×128. The number of feature channels doubles at every contracting step. Then a 1×1 convolution and 4×4 transposed convolution are applied generating the features F_A^(4)∈ℝ^14×14×128. Followed by the U-Net expanding path, which consists of two 4×4 transposed convolution. The input of the transposed convolution is a concatenation of the output from the previous transposed convolution and the corresponding feature map from the contraction path. The number of feature channels halves at every expanding step. At the end a 1×1 convolution is used to convert the channels to the same number as in the S-ORAlign features.

The downpooling to a low resolution of 7×7 gives the spatial attention a large receptive field in the feature map of F. The up-sampling is then used to have a weight map of the same size as F.
The model learns by itself which regions of an image are important. This is in contrast to our landmark attention, where the groundtruth heatmaps M, which resemble the landmark attention, are provided during training. 

 
 


  
Channel Attention
The channel attention is implemented via a Squeeze-and-Excitation block <cit.>. First a squeeze operation creates S ∈ℝ^512, an embedding of the global distribution of the channel-wise feature responses in F. This channel descriptor is created using average pooling

    S(c) = 1/28×28∑_u=1^28∑_v=1^28 F(u,v,c)   ∀ c ∈{1,…,512}

where F(·,·,c) is the feature map of the cth channel.

Then an excitation operation is performed on the channel wise aggregated feature map to create the channel attention. 

    A_channel = σ(  W_2 ReLU( W_1  S) ),

where W_1 ∈ℝ^512/r× 512, W_2 ∈ℝ^ 512×512/r, and σ represents the sigmoid activation function. Following the proposal in <cit.> a bottleneck is created using two fully-connected layers, with a reduction rate r. We choose r=16 in all our experiments.



  
Factorization

The factorization is then performed by multiplying the channel-wise feature responses in the spatial attention with the corresponding channel weights,

    A(x,y,c) = ( A^L_spatial(x,y,c)+ A^C_spatial(x,y,c))  A_channel(c) 
    ∀ (x,y) ∈{1,…,28}×{1,…,28}  ∀ c ∈{1,…,512}.

To refine the attention, an additional 1×1 convolution layer is added afterwards. This is motivated by the fact that the spatial and channel attention are not mutually exclusive but with co-occurring complementary relationship <cit.>.

Afterwards, a tanh function is used to shrink the attention values into a range of A ∈ [-1,1]^28×28×512.






  §.§.§ Rest of the network


Given A ∈ [-1,1]^28×28×512 we weight the S-ORAlign features F ∈ℝ^28×28×512, 

    U = ( 1+ A) ∘ F,

where ∘ denotes the Hadamard product and 1 is a tensor of ones with size 28×28×512. Hence, features where A(·, ·, ·) ∈ [-1, 0) are reduced and features where A(·, ·, ·) ∈ (0, 1] are increased. Our attention incorporates semantic information and global information into the network helping to focus on important regions in the images. The features U are then fed in to the conv5-1 layer. The rest of the network follows the VGG-16 structure but with a reduced number of weights in the two fully connected layers (i.e. 1024 and 1000 instead of 4096 and 1000).




 








 §.§ Experiments


In this section we describe the implementation details for each of the experiments and present results for the DeepFasion dataset. 



  §.§.§ DeepFashion Experiments




  
Implementation Details
We build our network using the publicly available implementation[<https://github.com/fdjingyuan/Deep-Fashion-Analysis-ECCV2018>] of <cit.> as a starting point. The cropped images are resized to the input size of the VGG-16 network (224×224). The A-ORConv and normal convolution layers are pretrained on ImageNet <cit.>. The fully connected layers of the VGG-16 network are replaced with two separate fully connected layer branches, one for the category classification and the other for the attribute prediction. We use cross entropy loss for the category classification. Due to the inbalance between positive and negative samples asymmetric weighted cross entropy loss is used for the attribute prediction. The batch size is 32 and 64 during training and validation, respectively. The model is trained using the Adam optimizer <cit.> with an initial learning rate of 0.0002, which is multiplied by a factor of 0.8 every fifth epoch. The landmark detection branch is initially trained separately for 20 epochs. The landmark prediction is then locked and the learning rate is reset. Without locking the landmark prediction accuracy would decrease significantly during the classification training. The category classification and attribute prediction are trained for up to 50 epochs. We perform early stopping on the validation set. Meaning we track the best result on the validation set and stop the training if the result does not improve over 5 consecutive epochs. The model state that achieved the best result is then used in the evaluation on the test set. We do not perform specific parameter tuning depending on the dataset and/or the augmentation method. Furthermore, Table <ref> shows the actually best epoch tracked with our early stopping. 






  
Experimental results on DeepFashion

We compare our landmark prediction results to the following five models <cit.> and the clothing category classification to these models <cit.>. We copy the results in Table <ref> and <ref> as they were presented in <cit.> and add our own results. We also show the results when using our proposed data augmentation methods during training.  




One can see that we outperform all other system in the landmark localization task when no augmentation or rotation is used. This indicates that our rotation invariance network structure is generally beneficial. That is specially noticeable for left/right sleeves. These are the parts of clothing that generally have the most variation between images. On the other hand the category classification is not as good compared to the other systems. We assume that increasing the number of channels in the A-ORConv layers could increase the accuracy. This is because the actual number of feature channels is only a fourth due to the rotated copies. As we show in the experiments in the main paper, our pretrained network outperforms   <cit.> when tested on other datasets. This suggests that the state-of-the-art models are not able to generalize from the training dataset.  

One can also see that our introduced elastic warping performs worse on this dataset. When trained on augmented data, the network spreads its computational power over more possible clothing configurations which might decrease performance on a certain configuration (the untransformed testing data).
 



  §.§.§ Elastic Warping parameters Experiments

We run additional experiments for Lanmark detection trained on DeepFashion and CTU dataset evaluated on the CTU dataset. Table <ref> shows the result for α=150 and σ = 10, table <ref> shows the result for α=100 and σ = 10, and <ref> shows the result for α=200 and σ = 10. We can clearly see that the EW helps to boost the performance for the R & EW augmentation when trained on the DeepFashion net and for only EW and R & EW for the CTU case. The best performance for when trained on the DeepFashion or CTU is achived with α=100 and σ = 10.

















 
 
 
 





 §.§ Implementation details for the CTU experiments


 
In the first experiment, we use the network trained as described in Section <ref> and perform solely inference with it. We do only consider the 13 categories in Table <ref> as possible predictions and mask the others out.

The network setup for the second experiment is the same as for the DeepFashion dataset described in Section <ref> with the exception of the following changes. The last fully connected layer of the VGG-16 network is reduced from 1000 categories down to 9 categories. The number of epochs is increased to a maximum of 200 since the dataset is much smaller and the learning rate decreases every 25th epoch. The landmark prediction is initially trained for 50 epochs.   





  §.§.§ Implementation details for the In-Lab Dataset

In this experiment we use the network trained as described in Section <ref>. We perform solely inference with the network. For the evaluation we only consider the categories that are present and mask the rest out. 


IEEEtranN



 













 






