Deep Interactive Region Segmentation and Captioning
Ali Sharifi Boroujerdi Maryam Khanian Michael Breuss
cs.CV 68T45


Double-parton scattering effects in associated production
 of charm mesons and dijets
at the LHC

    Antoni Szczurek[also at University of Rzeszów, PL-35-959 Rzeszów, Poland]
    December 30, 2023
=================================================================================================




With recent innovations in dense image captioning, it is now possible to describe every object of the scene with a caption while objects are determined by bounding boxes. However, interpretation of such an output is not trivial due to the existence of many overlapping bounding boxes. Furthermore, in current captioning frameworks, the user is not able to involve personal preferences to exclude out of interest areas. In this paper, we propose a novel hybrid deep learning architecture for interactive region segmentation and captioning where the user is able to specify an arbitrary region of the image that should be processed. To this end, a dedicated Fully Convolutional Network (FCN) named Lyncean FCN (LFCN) is trained using our special training data to isolate the User Intention Region (UIR) as the output of an efficient segmentation. In parallel, a dense image captioning model is utilized to provide a wide variety of captions for that region. Then, the UIR will be explained with the caption of the best match bounding box. To the best of our knowledge, this is the first work that provides such a comprehensive output. Our experiments show the superiority of the proposed approach over state-of-the-art interactive segmentation methods on several well-known datasets. In addition, replacement of the bounding boxes with the result of the interactive segmentation leads to a better understanding of the dense image captioning output as well as accuracy enhancement for the object detection in terms of Intersection over Union (IoU).





§ INTRODUCTION


As one of the main sources of the human knowledge, our visual system including eyes, optic nerves and brain is able to easily detect, separate and describe each object of a scene. Inspired by this natural ability, interactive region segmentation and captioning is the task of parallel detection, separation and description of the visual user interests. This procedure can be exploited in several complex applications such as automatic image annotation and retrieval <cit.>. To approach the task, one needs to have a full understanding of the scene which is equivalent to recognize and also locate all the visible objects. To this end, several object recognition techniques <cit.> have been proposed to detect image objects in different scales. In most of the literature, detected objects are determined by drawing bounding boxes around them. Although this notation is able to facilitate the detection process by decreasing its computational complexity, such an output is less informative when dealing with geometrical properties of the objects. As a more illustrative visual recognition technique, semantic segmentation <cit.> aims to assign a label to each pixel of the image where the labels can be class-aware or instance-aware. While the multi-level nature of semantic segmentation increases the problem dimensionality, interactive image segmentation <cit.> tries to adjust the segmentation task with the user priorities in a simpler problem space. In reality, it sounds reasonable that human users may have a more restricted area of interest than the entire scope of the scene. Thus, the multi-dimensional semantic segmentation task can be shrunk to a binary segmentation problem aiming to separate the User Intention Region (UIR) as the foreground from other parts of the scene which requires less time and computations. 


Equipped by the rich semantic memory of the visual data, the human observer is easily able to provide detailed explanation about different parts of an image which is a hard task in artificial intelligence. Thanks to recent developments of language models <cit.>, image captioning <cit.> makes it possible to produce linguistic descriptions of an image through a multimodal embedding of the visual stimuli and the word representation <cit.> in a joint vector space <cit.>.


In this paper, we propose a novel hybrid deep architecture for integrated detection, segmentation and captioning of the user preferences where the amount of the user interactions is limited to one or a few clicks. To this end, we designed a heuristic technique for the efficient generation of the synthetic user interactions. In addition, the new architecture of the proposed Lyncean Fully Convolutional Network (LFCN) leads to a better sight of the deep component that is responsible for interactive segmentation. Last but not least, as depicted in Fig. <ref>, our combination of interactive segmentation and dense captioning tasks introduces a new class of outputs where the user intention recognition meets linguistic interpretations and vice versa. Let us stress at this point that our main contributions are (i) to provide the first deep framework for combined interactive segmentation and captioning, and (ii) to achieve segnificant improvements in the interactive segmentation over other methods.



 


§ MORE DETAILS ON RELATED WORKS

With the increasing popularity of deep learning architectures <cit.>, both detection and captioning procedures have attracted a new wave of considerations. Convolutional Neural Networks (CNNs) <cit.> have presented the ability to construct numerous visual features in different levels of abstraction through supervised learning. This property leads to feature generators that are able to reach near-human performance in various computer vision tasks <cit.>. In addition, the structure of Fully Convolutional Networks (FCNs) <cit.> made it feasible to apply inputs of any size to the network and generate associated output in the same spatial domain. In contrast to CNNs, FCNs are able to maintain spatial information which is crucial to perform a pixel-level prediction such as semantic segmentation, object localization <cit.>, depth estimation <cit.> and interactive segmentation. Furthermore, Recurrent Neural Networks (RNNs) <cit.> reveal potential for learning long term dependencies which is essential for simulating the continuous space of natural languages.


Recently, CNN-RNN models are proposed to wrap detection and captioning tasks in an end-to-end learnable platform <cit.>. However, up to now the results appear to be mostly an unorganized and overcrowded set of captions and bounding boxes. These results are not easily understandable especially in the presence of several overlapping region proposals cf. Fig. <ref> (c). In addition, they do not involve user intentions.


Before the success of CNNs in object detection, some classical techniques such as Histogram of Gradients (HoG) <cit.>, Deformable Part Models (DPM) <cit.> and selective search <cit.> (as an explicit region proposal method) were proposed. Later, in the Region-based CNN (R-CNN) model <cit.>, each proposed region has been forwarded through a separate CNN for the feature extraction. This model had some drawbacks such as a complex multi-stage training pipeline and expensive training process. To overcome those obstacles, the Fast R-CNN model <cit.> is proposed where a combination of a CNN and the Region of Interest (RoI) pooling mechanism is used to produce better information for region proposal. In the Faster R-CNN <cit.>, the CNN architecture is used not only for the feature extraction but also for region proposal itself. This leads to the invention of the Region Proposal Networks (RPNs) that are able to share full-image convolutional features with detection networks. The main achievement of this innovation is the parallel detection and localization of the objects in one forward pass of the network. In spite of these improvements, such models are not able to backpropagate through the bounding boxes information. Recently, Johnson et. al <cit.> proposed a localization layer based on Faster R-CNN architecture where the RoI pooling mechanism is replaced by bilinear interpolation <cit.> that makes it possible to propagate backward through all the information.

 
The primary purpose of the image captioning was image annotation <cit.> as the automatic assignment of some keywords to a digital image. By replacing keywords with some sentences that are able to describe not only the image objects but also the semantic relations in between, image captioning received more attention. The main problem in the automatic image description was the scarcity of the training data. Recent development of large datasets including images and their descriptions <cit.> makes it feasible to expand learning-based captioning techniques. Classical image captioning approaches produced image descriptions by generative grammars <cit.> or pre-defined templates working on some specific visual features <cit.>.  In contrast, recently developed deep learning solutions apply an RNN-based language model that is conditioned on the output vectors of a CNN to generate descriptive sentences <cit.>.


With the growing popularity of interactive devices such as smart phones and tablets, interactive image processing attracts more attention. Interactive segmentation offers a pixel-wise classification based on user priorities. Among all the traditional approaches of the interactive segmentation, stroke-based techniques <cit.> are often based on graph cut techniques. In these methods, an energy function based on region/boundary division is optimized to find the segmentation result. Alternative approaches include random walks <cit.> and geodesics <cit.>, mostly relying on low-level features such as color, texture and shape information. These types of attributes can be difficult to apply when the image appearance is complicated due to complex lighting conditions or existence of intricate textures. Recently, deep learning models have been used for interactive segmentation where the information of the image will be considered in higher semantic levels. To this end, FCNs as the standard frameworks for pixel-wise end-to-end learning tasks, have been applied <cit.>.




§ PROPOSED METHOD
  
Our model receives an input image as well as user interactions in the form of positive/negative clicks and provides a seamless framework to generate accurate segmentation as well as expressive description of the UIR. In the preprocessing step, an efficient morphological technique will be used to provide  a huge amount of training samples in the form of synthetic user interactions. Then, each set of positive/negative seeds will be transformed into separate Voronoi diagrams as shown in Fig. <ref>. Next, a sequence of dedicated LFCNs with different granularities are applied as the interactive segmentation modules. Afterwards, a dense captioning architecture inspired by <cit.> will be utilized to obtain a number of region proposals along with their captions. In the fusion step, a heuristic method will be provided to combine results of localization, segmentation and captioning procedures to acquire highlighted borders of the UIR along with its expositor caption. In the following, we will investigate all steps of our model in detail.





 §.§ User Action Imitation

During interactive segmentation, the user will be asked to provide some general information about the position of the intended region. The requested information consists of some positive and negative seeds as depicted in Fig. <ref> which are equivalent to internal and external points of the UIR, respectively. Next, each set of seeds will be used to shape a Voronoi diagram. We denote each seed by s_k, k={1,…,n}. The value of pixel v_i,j of the Voronoi diagram will be calculated by

    v_i,j:=min{D_1,D_2,…,D_n}

where D_k is the Euclidean distance of v_i,j to the seed s_k.
To summarize, the value of each pixel in the Voronoi diagram is the Euclidean distance of that pixel to the nearest seed. For the sake of clarity, there should be a minimum inter-cluster distance in each set of the seeds. In addition, a minimum intra-cluster distance is also required to retain boundary regions of the clusters as distinctive as possible. So:




  * Every pair of seeds in each set should preserve a pre-defined distance from each other:

    ∃ d_1∈ℝ^+ : ∀ (s_i,s_j)∈ S, (s_i,s_j)_2>d_1


  * All the seeds of each set should preserve a minimum distance from boundary pixels of the UIR (∂(UIR)):

    ∃ d_2 ∈ℝ^+ :∀ s_i∈ S, u∈∂(UIR), (s_i,u)_2>d_2


As expected, natural collection of such a data is unreasonably time consuming and expensive. Recently, Xu et al. <cit.> proposed some strategies for synthetic generation of user interactions. They ordained random generation for positive clicks inside the UIR while three distinct set of negative clicks are chosen as: 1) random background pixels with a certain distance to the UIR, 2) a point cloud inside the negative objects and 3) a uniform set of surrounding points of the UIR. Since their implementation is not publicly available, it seems their first and the second negative strategies do not obey natural interactions and the third one may be computationally expensive (see equation (2) in <cit.>). 


Morphological Cortex Detection (MCD). While the inside of the UIR can be quite small, the background region is usually large enough to provide useful geometric information about the UIR. Consequently, it is beneficial to generate negative seeds that surround the UIR uniformly. To provide an efficient implementation for such an interaction, we replace third negative strategy proposed in Xu et al. <cit.> with a Morphological Cortex Detection (MCD) technique that noticeably improves computational efficiency. Moreover, this method is able to simulate UIR cortex in different scales that enables convolutional filter of the LFCN to track UIR geometry in different layers. To implement this idea, a 1-pixel-wide boundary shape of the UIR will be extracted by performing a dilation on the binary mask of UIR in training dataset. Then, the original mask is subtracted from the dilation result. In the next step, this boundary path will be completely traversed using a 3×3 window to transfer all the boundary points' coordinates into a 1-D array in which the requested negative seeds can be selected uniformly. As the result of the MCD process, a uniform set of negative seeds will be obtained that represents the cortex of the UIR perfectly. The visual illustration of this technique is shown in Fig. <ref>. During our experiments, positive clicks are simulated randomly inside the UIR while negative seeds are generated by MCD mechanism in three different levels. 





 §.§ Intention Recognition

For the task of intention recognition, we make use of a dedicated version of the standard FCN <cit.> where the last two fully connected layers are replaced with three convolutional layers containing decreasing kernel sizes of 7, 5 and 3. The impact of such an alternation is the gradual growth of the receptive field. This property improves network recognition of objects' geometry. Hence, we named this architecture as Lyncean Fully Convolutional Network (LFCN). By a proper use of zero padding, all the extended convolutional layers have the same output size. At the end of the extended part, the aggregated output of the additional layers will be upsampled to the size of the input as elaborated in <cit.>. 


 §.§ Fusion Approach

In order to supplement the result of the interactive segmentation with a proper linguistic commentary, we employ the dense image captioning framework <cit.>. The internal RPN of this architecture provides confidence scores for the existence of the object in proposed regions. After descending sort of the objectness scores, top-ranked region proposals include the most reasonable captions for the objects of the scene. With the comparison of the interactive segmentation result and the bounding boxes, the best match bounding box and the corresponding caption will be obtained (Fig. <ref>).     





§ EXPERIMENTS

Datasets.  
For fine tuning of the LFCN, we used the PASCAL VOC 2012 segmentation dataset <cit.>. The dataset includes 1464 images for training and 1449 images for validation that are distributed in 20 different classes. We used the whole bunch of these samples to generate our special training pairs in the preprocessing step. For the final validation of the model as well as its comparison with state-of-the-art interactive segmentation, we utilized different well-known segmentation benchmarks including Alpha Matting <cit.>, Berkeley segmentation dataset (BSDS500) <cit.>, Weizmann segmentation evaluation database <cit.>, image object segmentation visual quality evaluation database <cit.> and VOC validation subset.  


Preprocessing.
To generate all the necessary training pairs of the interactive segmentation process, we produced positive and negative Voronoi diagrams with respect to each object that is visible in VOC dataset. The positive seeds are selected randomly inside each object while the MCD approach is used to generate three distinct sets of negative seeds with different distances from the intended object. In the last step, each combination of positive/negative Voronoi diagrams, forms a unique training pair. This leads to production of 97,055 interaction patterns. We preserved 7,055 instances for the test and used the rest as the training data.



 §.§ Fine Tuning of the Proposed LFCN Architecture

To reach the best quality for the interactive segmentation, our LFCN is trained in three different levels of granularity as proposed in <cit.>: LFCN32s, LFCN16s and LFCN8s. 
RGB channels of the input image should be concatenated with the corresponding Voronoi diagrams to form a training instance. Consequently, the first convolutional layer of our LFCN contains five channels. During the network initialization, the RGB-related channels will be initialized by the parameters of the original FCN <cit.>. For two extra channels that are associated with Voronoi diagrams, the zero initialization is the best choice as also mentioned in <cit.>. Learning parameters of the finer networks should be initialized from the coarser one. The global learning rates of the networks are 1e-8, 1e-10 and 1e-12, respectively while the extended convolutional layers exploit one hundred times bigger learning rates. The learning policy is fixed and we used the weight decay of 5e-3.



 §.§ Metrics

In order to evaluate UIR localization accuracy of the proposed model, we calculated the well-known measure of Intersection over Union (IoU). To this aim, we computed the IoU of the detected UIR and the corresponding binary label of the validation samples. For the sake of complete comparison between our model and other interactive segmentation techniques, three performance metrics of pixel accuracy, mean accuracy and mean IoU are computed.


The segmentation task of the proposed approach can be considered as a binary segmentation where the classes are limited to foreground (UIR) and background. So, we used binary interpretation of the semantic segmentation metrics that are proposed by Long et al. <cit.>:


  * Pixel Accuracy (Pixel Acc.):

This measure represents the proportion of the correctly classified foreground (C_f) and background (C_b) pixels (true positive rates) to the total number of ground truth pixels in foreground (F) and background (B).

    C_f+C_b/F+B

Unfortunately, this metric can be easily influenced by the class imbalance. Hence, high pixel accuracy does not necessarily mean that the accuracy is acceptable when one of the classes is too small or too large.


  * Mean Pixel Accuracy (Mean Acc.):

This measure is computed as the mean of the separate foreground and background pixel accuracies:

    C_f/F+C_b/B/2

This metric alleviates the imbalance problem but can be still misleading. For example when the great majority of pixels are background, a method that predicts all the pixels as background can still have seemingly good performance.

  * Mean Intersection over Union (Mean IoU):

Intersection over union is the matching ratio between the result of object localization process and the corresponding ground truth label. This metric is the average of the computed intersection over union for the foreground and background regions:

    (C_f/C_f+FP+FN)_f+(C_b/C_b+FP+FN)_b/2

Here FP and FN denote the number of the false positive and false negative rates of each class, respectively. This metric solves the previously described issues.
    


 §.§ Results

Test of localization accuracy. 
In the first step of evaluation, we test our model with a random subset of unseen samples in validation datasets. The response of the model to some instances is shown in Fig. <ref>. It can be noticed that the output of our approach achieves a considerable rate of accuracy regarding the similarity of the model output with the corresponding ground truth. Furthermore, the confusing output of the dense image captioning is replaced with an explicit situation where the segmented UIR and its description are easily distinguishable.



Fig. <ref> (left diagram) presents a comparison between the localization accuracy of the proposed method and the internal RPN of the DenseCap <cit.> in terms of the obtained IoU for the samples presented in Fig. <ref>. As illustrated, our model provides a significant improvement regarding the localization accuracy. These results also demonstrate the proficiency of our model in combining interactive segmentation, region proposal and image captioning techniques.





Sensitivity analysis. 
In this part, we analysed variations of the model output quality against the number of user interactions. As it is shown in Fig <ref>, although IoU can be improved by applying more user interactions that facilitate boundary detection, our model still provides very good results even by minimum number of clicks. We also provided mean IoU accuracy of the proposed model for five different datasets in Fig. <ref> (right diagram) that confirms satisfying performance of our model in the case of low interactive information. This noticeable property of our approach makes it convenient to be applied in real-world applications. During our experiments, the proposed method clearly achieves a satisfying segmentation outcome with just one click.



Comparison of the segmentation quality. 
In this part we performed an extensive evaluation on segmentation capabilities of the proposed method versus some prevalent segmentation techniques such as Geodesic Matting (GM) <cit.>, GrowCut <cit.>, Grabcut <cit.>, Boykov Jolly (BJ) interactive graph cuts <cit.>, Geodesic Star Convexity (GSC) <cit.>, Geodesic Star Convexity with sequential constraints (GSCSEQ), Random Walker (RW) segmentation <cit.>, Shortest Path-based interactive segmentation (SP) <cit.> and Matching Attributed Relational Graphs (MARG) <cit.>. In all the experiments we generated five positive and five negative clicks randomly. For some of the approaches where the user interactions were defined as points or scribbles, we determined click positions with five-pixel-wide circles. To observe the impact of the extended part of the LFCN on the output quality, we also report all the accuracy measures for the normal version of the FCN as well. Table <ref> and <ref> present quantitative results that confirm our approach superiority over several other segmentation techniques on five different benchmarks. As a qualitative comparison, Fig. <ref> represents final segmentation output of the methods in Table <ref> for two different samples. As it can be seen, our approach provides the most accurate segmentation result with respect to semantic interpretation of the scene using same number of interactions.









Dense interactive region captioning. In the final part of our experiments, we verified the ability of the proposed model to caption several regions of the image. Fig. <ref> shows the result of such an experiment where multiple objects in different scales are detected via user interactions and described properly.





§ CONCLUSION

In this paper, we presented a novel hybrid deep learning framework which is capable of targeted segmentation and captioning as a response to user interactive actions. A wide variety of experiments confirmed our model superiority over various state-of-the-art interactive segmentation approaches. In addition, further experiments demonstrated our model capability to caption an arbitrary region of the image with one or few clicks, which is especially convenient for real-world interactive applications. 


