Object category learning and retrieval with weak supervision
Steven Hickson Anelia Angelova Irfan Essa Rahul Sukthankar
cs.CV cs.LG










Object category learning and retrieval with 
weak supervision
    Steven Hickson, Anelia Angelova, Irfan Essa, Rahul Sukthankar

Google Brain / Google Research

(shickson, anelia, irfanessa, sukthankar)@google.com
    December 30, 2023
=======================================================================================================================================================


We consider the problem of retrieving objects from image data and learning to classify them into meaningful semantic categories with minimal supervision. To that end, we propose a fully differentiable unsupervised deep clustering approach to learn semantic classes in an end-to-end fashion without individual class labeling using only unlabeled object proposals. The key contributions of our work are 1) a kmeans clustering objective where the clusters are learned as parameters of the network and are represented as memory units, and  2) simultaneously building a feature representation, or embedding, while learning to cluster it.
This approach shows promising results on two popular computer vision datasets: on CIFAR10 for clustering objects, and on the more complex and challenging Cityscapes dataset for semantically discovering classes which visually correspond to cars, people, and bicycles. 
Currently, the only supervision provided is segmentation objectness masks, but this method can be extended to use an unsupervised objectness-based object generation mechanism which will make the approach completely unsupervised.




§ INTRODUCTION


Unsupervised discovery of common patterns is a long standing task for artificial intelligence as shown in <cit.>.
Recent deep learning approaches have offered major breakthroughs in classification into multiple categories with millions of labeled examples (e.g. <cit.> and many others). These methods rely on a lot of annotated data for training in order to perform well. Unfortunately, labeling is an inefficient and expensive progress, so learning from unlabeled data is desirable for many complex tasks. At the same time, much of human knowledge and learning is obtained by unsupervised observations <cit.>. 

The goal of this work is to show that semantically meaningful classes can be learned with minimal supervision. Given a set of objectness proposals, we use the activations of foreground objects in order to learn deep features to cluster the available data while simultaneously learning the embedding in an end-to-end manner.
More specifically, we propose a differentiable clustering approach that learns better separability of classes and embedding. 
The main idea is to store the potential cluster means as weights in a neural network at the higher levels of feature representation. This allows them to be learned jointly with the potential feature representation.
This differentiable clustering approach is integrated with Deep Neural Networks (e.g. <cit.>) to learn semantic classes in an end-to-end fashion without manual class labeling. 

The idea of doing this `end-to-end' is that gradient descent can not only learn good weights for clustering, it can also change the embedding to allow for better clustering without the use of labels. We see that this leads to better feature representation.
Our results show also that different object categories emerge and can later be retrieved from test images never before seen by the network, resulting in clusters of meaningful categories, such as cars, persons, bicycles.

In this work we use given segmentation objectness masks, which are candidate objects without labels. This can be extended by using an independent objectness-based object generation mechanism <cit.> or by using unsupervised motion segmentation in videos or structure from motion <cit.>.



§ RELATED WORK


Unsupervised learning (<cit.>) and unsupervised deep learning (<cit.>, <cit.>, <cit.>) are central topics to Machine Learning.  
Unsupervised deep learning has been shown to improve results on classification tasks per <cit.>, especially given small datasets and complicated high dimensional data such as video. This has been explored by many representations including sequence to sequence learning and textual representations (<cit.>, <cit.>). 

Our work focuses on unsupervised deep learning for discovering visual object categories. This has also been shown to improve results such as in <cit.>. Unsupervised discovery of visual objects has been a large topic of interest in computer vision (<cit.>).

Building specialized, deep embeddings to help computer vision tasks is also a popular approach  such as in <cit.>. Transfer learning from supervised tasks has proven to be very successful. Further,  <cit.> propose learning the lower dimensional embedding through unsupervised learning and show improved performance when transfered to other supervised tasks.

Despite the popularity of building different embeddings, there is little work investigating the use of clustering to modify the embedding in an end-to-end deep learning framework. <cit.> investigate a differentiable version of the kmeans algorithm and examine its convergence properties. 
Our work focuses on learnable feature representations (instead of fixed ones as in <cit.>) and introduces memory units for the task.





§ UNSUPERVISED DEEP CLUSTERING




Our unsupervised deep clustering is inspired by <cit.>, who consider  differentiable clustering algorithms. 

We differ from this approach because the features we cluster also change with backpropogation.
In our work, we add a kmeans-like loss that is integrated end-to-end. Our idea is to store the potential cluster means as weights in the network and thus have them be learned.

The proposed clustering is done simultaneously while building an embedding. Given information of a potential object vs background (binary labels), clustering in a differentiable way provides a better embedding for the input data. We show that this method can be used for meaningful semantic retrieval of related objects. 





 §.§ Embedding with clustering


We train a convolutional neural network (CNN) to predict foreground and background using oracle labels of patches of  objects and background images. Concurrently, we learn the clustering of objects by imposing constraints that will force the embedding to be partitioned into multiple semantically coherent clusters of objects without explicit labels for different objects.  
 
For our experiments, we use random initialization on the fully-connected layers (the last two layers) and we add the differentiable clustering module after the second to last layer. Note that we only cluster the foreground labels as background activations are not of interest for clustering; the classifier can predict foreground vs background with high accuracy (above 90%). 


The objective function is shown in Equation <ref>.

    L_k = 1/2N∑_n=1^Nmin_k[(x_n - w_k)^2]


In this equation, N is the number of samples, k is the number of defined clusters, w is the “weight” (theoretically and typically the mean of the cluster) for each k, and x is the activations from the fully connected layer before the classification fully connected layer.
This is differentiable and the gradient descent algorithm is shown in Equation <ref>.


    δ w_k = w'_k - w_k = ∑_n=1^N
    
    l_r(x_n - w_k)    if  k = s(x_n,w)
    
    0    otherwise


where s(x_n,w) = argmin_k[x_n] and l_r is the learning rate.
We also add L_2 regularization over the weights to the loss L_2=∑_jw_j^2. Furthermore, we use a custom clustering metric M_C  that informs us if the clusters are evenly distributed as defined in Equation <ref> and Equation <ref>.  This is important since we have no labels to evaluate our data during training.


    M_C = 1/NK∑_k=0^K ∑_j=k^K | count_k - count_j |



    count_k = ∑_n=0^N
    
    0    if  argmin_k[x_n]=0
    
    1    if  argmin_k[x_n]=1



The final loss to be optimized is shown in (Equation <ref>)

    L= L_k + α_r L_2 + α_c L_C


where α_r and α_c are hyperparameters which are tuned during the training. L_C is the standard cross-entropy loss for our foreground vs background classification. For our method, we use α_r = 0.25 and α_c = 1. We apply this loss to every point that is labeled as potentially an object and ignore the background ones when clustering. This way we learn foreground vs background and then learn clustering of the foreground activations. 
Optimization was performed with a `RMSProp' optimizer, with a learning rate of 0.045, momentum 0.9, decay factor 0.9, and ϵ of 1.0.








§ EXPERIMENTAL EVALUATION


We experiment with a toy example using CIFAR10 and a more challenging example using Cityscapes.




 §.§ CIFAR10 dataset


We first test the proposed unsupervised clustering approach 
on the CIFAR10 <cit.> dataset. The goal of this experiment is to test if clustering can uncover separate categories in a simple toy problem with a two class setting. 



We selected as an example the dog and automobile classes to label as foreground. We then train a network from scratch based on the Network in Network architecture (NiN) of <cit.> from scratch for our experiments. All other classes of CIFAR are considered background for this experiment. By attaching our modified clustering objective function to the next to last layer, we attempt to cluster dog and automobile without labels.

We can see in our simple experimental results that classes are naturally clustered with the majority of examples correctly assigned. Table <ref> shows quantitative results on the test set. As seen 68.5% of the automobile classes and 82.1% of the dog examples are correctly assigned to separate clusters.
Note that in these cases, the concepts and classes of dog and automobile are unknown to the training algorithm and we are just looking at them after clustering for evaluation.




 §.§ Cityscapes dataset


The Cityscapes dataset (<cit.>) is a large-scale dataset that is used for evaluating various classification, detection, and segmentation algorithms related to autonomous driving.
It contains 2975 training, 500 validation, and 1525 test images, where the test set is provided for the purposes of the Cityscape competition only.
In this work, we used the training set for training and the validation set for testing and visualizing results (as the test set has no annotation results). Annotation is provided for classes which represent moving agents in the scene, such as pedestrian, car, motorcycle, bicycle, bus, truck, rider, train. In this work we only use foreground/background labels and intend to discover semantic groups from among the moving objects.



 §.§ Weakly supervised discovery of classes



In this experiment we considered the larger, real-life dataset, Cityscapes (<cit.>), described above to see if important class categories, e.g. the moving objects in the scene can be clustered into semantically meaningful classes. We extract the locations and extents of the moving objects and use that as weak supervision. Note the classes are uneven and car and person dominate.

We show results clustering 8 categories into 2 and 3 clusters despite the rarity of some of them (such as bicycle).
All results below are presented on the validation set. We report the results in terms of the number of object patches extracted from the available test images.
For this dataset, the CNN architecture is based on the Inception architecture proposed by <cit.>. Since there are a small number of examples, we pre-train only the convolutional layers of the network.


Results on clustering the 8 classes of moving objects into 2 and 3 clusters are presented in Table <ref> and Table <ref> respectively for the learned embedding by the proposed approach and the baseline embedding. The baseline embedding is calculated by fine-tuning the same architecture in the same manner, but without our loss (Equation <ref>); it uses the same amount of information as input as our embedding. For this experiment, we apply standard kmeans on both activations after training is completed. We see here that our method provides better clustering for the two dominant classes in the dataset (car and person). On the other hand, the baseline embedding clusters on one class only, similar to the two class case. We have consistently observed this behavior for different runs and hypothesize this is due to the sparse nature of the baseline embedding and it's activations.

Figure <ref> visualizes the three retrieved clusters (color-coded) when clustering into 3 clusters with our approach. We can see that people (in blue) and cars (in green) are often correctly retrieved.  Bikes are more rare and may be more often mistaken, for example in cases where a portion of the patch contains part of a car, or since the bicycle very often has a person riding it. Still this is exciting result, given that it is learned by not providing a single class label during training.















§ CONCLUSIONS


 

We propose a differentiable clustering objective which learns to separate classes during learning and build a better embedding. The key idea is to be able to learn the clusters which are stored as  weights, and simultaneously learn the feature representation and the clustering of the data. Our results show that the proposed approach is useful for extracting semantically related objects. 



aaai



