Semi-convolutional Operators for Instance Segmentation
David Novotny Samuel Albanie Diane Larlus Andrea Vedaldi

 


Novotny et al.: Semi-convolutional Operators for Instance Segmentation









Novotny D., Albanie S., Larlus D., Vedaldi A.




	Visual Geometry Group, Department of Engineering Science, University of Oxford 

	{david,albanie,vedaldi}@robots.ox.ac.uk 
 Computer Vision Group, NAVER LABS Europe 

	diane.larlus@naverlabs.com


Semi-convolutional Operators for 
 Instance Segmentation
    
David NovotnyEqual contribution1,2 [1]1 2 1
 

    December 30, 2023
========================================================





Object detection and instance segmentation are dominated by region-based methods such as Mask RCNN. However, there is a growing interest in reducing these problems to pixel labeling tasks, as the latter could be more efficient, could be integrated seamlessly in image-to-image network architectures as used in many other tasks, and could be more accurate for objects that are not well approximated by bounding boxes. In this paper we show theoretically and empirically that constructing dense pixel embeddings that can separate object instances cannot be easily achieved using convolutional operators. At the same time, we show that simple modifications, which we call semi-convolutional, have a much better chance of succeeding at this task. We use the latter to show a connection to Hough voting as well as to a variant of the bilateral kernel that is spatially steered by a convolutional network. We demonstrate that these operators can also be used to improve approaches such as Mask RCNN, demonstrating better segmentation of complex biological shapes and PASCAL VOC categories than achievable by Mask RCNN alone.





§ INTRODUCTION




State-of-the-art methods for detecting objects in images, such as R-CNN <cit.>, YOLO <cit.>, and SSD <cit.>, can be seen as variants of the same paradigm: a certain number of candidate image regions are proposed, either dynamically or from a fixed pool, and then a convolutional neural network (CNN) is used to decide which of these regions tightly enclose an instance of the object of interest. An important advantage of this strategy, which we call propose & verify (P&V), is that it works particularly well with standard CNNs. However, P&V also has several significant  shortcomings, starting from the fact that rectangular proposals can only approximate the actual shape of objects; segmenting objects, in particular, requires a two-step approach where, as in Mask R-CNN <cit.>, one first detects object instances using simple shapes such as rectangles, and only then refines the detections to pixel-accurate segmentations.

An alternative to P&V that can overcome such limitations is to label directly individual pixels with an identifier of the corresponding object occurrence. This approach, which we call instance coloring (IC), can efficiently represent any number of objects of arbitrary shape by predicting a single label map. Thus IC is in principle much more efficient than P&V. Another appeal of IC is that it can be formulated as an image-to-image regression problem, similar to other image understanding tasks such as denoising, depth and normal estimation, and semantic segmentation. Thus this strategy may allow to more easily build unified architectures such as <cit.> that can solve instance segmentations together with other problems.

Despite the theoretical benefits of IC, however, P&V methods currently dominate in terms of overall accuracy. 
The goal of this paper is to explore some of the reasons for this gap and to suggest workarounds. 
Part of the problem may be in the nature of the dense labels. The most obvious way of coloring objects is to number them and “paint” them with their corresponding number. However, the latter is a global operation as it requires to be aware of all the objects in the image. CNNs, which are local and translation invariant, may therefore be ill-suited for direct enumeration. Several authors have thus explored alternative coloring schemes  more suitable for convolutional networks. A popular approach is to assign an arbitrary color (often in the guise of a real vector) to each object occurrence, with the only requirement that different colors should be used for different objects <cit.>. The resulting color affinities can then be used to easily enumerate object a posteriori via a non-convolutional algorithm.

In this paper, we argue that even the latter technique is insufficient to make IC amenable to computation by CNNs. The reason is that, since CNNs are translation invariant, they must still assign the same color to identical copies of an object, making replicas indistinguishable by convolutional coloring. This argument, which is developed rigorously in <ref>, holds in the limit since in practice the receptive field size of most CNNs is nearly as large as the whole image; however, it suggests that the convolutional structure of the network is at least an unnatural fit for IC.

In order to overcome this issue, we suggest that an architecture used for IC should not be translation invariant; while this may appear to be a significant departure from convolutional networks, we also show that a small modification of standard CNNs can overcome the problem. We do so by defining semi-convolutional operators which mix information extracted from a convolutional network with information about the global location of a pixel (<ref>). We train the latter (<ref>) so that the response of the operator is similar for all pixels that belong to the same object instance, making this embedding naturally suited for IC. We show that, if the mixing function is additive, then the resulting operator bears some resemblance to Hough voting and related detection approaches. After extending the embedding to incorporate standard convolutional responses that capture appearance cues (<ref>), we use it to induce pixel affinities and show how the latter can be interpreted as a steered version of a bilateral kernel (<ref>). Finally, we show how such affinities can also be integrated in methods such as Mask RCNN (<ref>).




We assess our method with several experiments. We start by investigating the limit properties of our approach on simple synthetic data. Then, we show that our semi-convolutional feature extractor can be successfully combined with state-of-the-art approaches to tackle parsing of biological images containing overlapping and articulated organisms (<ref>). Finally, we apply the latter to a standard instance segmentation benchmark PASCAL VOC (<ref>). We show in all such cases that the use of semi-convolutional features can improve the performance of state-of-the-art instance segmentation methods such as Mask RCNN.






§ RELATED WORK




The past years have seen large improvements in object detection, thanks to powerful baselines such as Faster-RCNN <cit.>, SSD <cit.> or other similar approaches  <cit.>, all from the propose & verify strategy.

Following the success of object detection and semantic segmentation, the challenging task of instance-level segmentation has received increasing attention. Several very different families of approaches have been proposed.

Proposal-based instance segmentation
While earlier methods relied on bottom-up segmentations <cit.>, the vast majority of recent instance-level approaches combine segment proposals together with powerful object classifiers.
In general, they implement a multi-stage pipeline that first generates region proposals or class agnostic boxes, and then classifies them <cit.>.
For instance DeepMask <cit.> and follow-up approaches <cit.> learn to propose segment candidates that are then classified.
The MNC approach <cit.>, based on Faster-RCNN <cit.>, repeats this process twice <cit.> while <cit.> does it multiple times. <cit.> extends <cit.> to model the shape of objects.
The fully convolutional instance segmentation method of <cit.> also combines segmentation proposal and object detection using a position sensitive score map.

Some methods start with semantic segmentation first, and then cut the regions obtained for each category into multiple instances <cit.>, possibly involving higher-order CRFs <cit.>.

Among the most successful methods to date, Mask-RCNN <cit.> extends Faster R-CNN <cit.> with a small fully convolutional network branch <cit.> producing segmentation masks for each region of interest predicted by the detection branch. Despite its outstanding results, Mask-RCNN does not come without shortcomings: it relies on a small and predefined set of region proposals and non-maximum suppression, making it less robust to strong occlusions, crowded scenes, or objects with fundamentally non-rectangular shapes (see detailed discussion in <ref>).

Instance-sensitive embeddings
Some works have explored the use of pixel-level embeddings in the context of clustering tasks, employing them as a soft, differentiable proxy for cluster assignments <cit.>. This is reminiscent of unsupervised image segmentation approaches <cit.>. It has been used for body joints <cit.>, semantic segmentation <cit.> and optical flow <cit.>, and, more relevant to our work, to instance segmentation <cit.>.  

The goal of this type of approaches is to bring points that belong to the same instance close to each other in an embedding space, so that the decision for two pixels to belong to the same instance can be directly measured by a simple distance function. Such an embedding requires a high degree of invariance to the interior appearance of objects. 

Among the most recent methods, <cit.> combines the embedding with a greedy mechanism to select seed pixels, that are used as starting points to construct instance segments. 
<cit.> connects embeddings, low rank matrices and densely connected random fields.
<cit.> embeds the pixels and then groups them into instances with a variant of mean-shift that is implemented as a recurrent neural network.

All these approaches are based on convolutions, that are local and translation invariant by construction, and consequently are inherently ill-suited to distinguish several identical instances of the same object (see more details about the convolutional coloring dilemma in <ref>). A recent work <cit.> employs position sensitive convolutional embeddings that regress the location of the centroid of each pixel's instance. We mainly differ by allowing embeddings to regress an unconstrained representative point of each instance.

Among other approaches using a clustering component, <cit.> leverages a coverage loss and <cit.> make use of depth information.
In particular, <cit.> trains a network to predict each pixel direction towards its instance center along with monocular depth and semantic labeling. Then template matching and proposal fusion techniques are applied.

Other instance segmentation approaches

Several methods <cit.> move away from box proposals and use Faster-RCNN <cit.> to produce “centerness” scores on each pixel instead. They directly predict the mask of each object in a second stage. An issue with such approaches is that objects do not necessarily fit in the receptive fields. 

Recurrent approaches sequentially generate a list of individual segments. For instance, <cit.> uses an LSTM for detection with a permutation invariant loss while <cit.> uses an LSTM to produce binary segmentation masks for each instance. <cit.> extends <cit.> by refining segmentations in each window using a box network.
These approaches are slow and do not scale to large and crowded images.

Some approaches use watershed algorithms. <cit.> predicts pixel-level energy values and then partition the image with a watershed algorithm. <cit.> combines a watershed algorithm with an instance aware boundary map. Such methods create disconnected regions, especially in the presence of occlusion.  


























































































































































































§ METHOD




 §.§ Semi-convolutional networks for instance coloring


Let ∈𝒳=ℝ^H× W× 3 be an image and u ∈Ω = {1,…,H}×{1,…,W} a pixel. In instance segmentation, the goal is to map the image to a collection 𝒮_={S_1,…,S_K_}⊂ 2^Ω of image regions, each representing an occurrence of an object of interest. The symbol S_0 = Ω - ∪_k S_k will denote the complementary region, representing background. The regions as well as their number are a function of the image and the goal is to predict both.

In this paper, we are interested in methods that reduce instance segmentation to a pixel-labeling problem. Namely, we seek to learn a function Φ :𝒳→ℒ^Ω that associates to each pixel u a certain label Φ_u() ∈ℒ so that, as a whole, labels encode the segmentation 𝒮_. Intuitively, this can be done by painting different regions with different “colors” (aka pixel labels) making objects easy to recover in post-processing. We call this process instance coloring (IC).

A popular IC approach is to use real vectors ℒ=ℝ^d as colors, and then require that the colors of different regions are sufficiently well separated. Formally, there should be a margin M > 0 such that:

    ∀ u,v ∈Ω:
       Φ_u() - Φ_v()≤ 1 - M,    ∃ k : u,v∈ S_k,
    Φ_u() - Φ_v()≥ 1 + M,    otherwise.

If this is the case, clustering colors trivially reconstructs the regions.

Unfortunately, it is difficult for a convolutional operator Φ to satisfy constraint (<ref>) or analogous ones. While this is demonstrated formally in <ref>, for now an intuition suffices: if the image contains replicas of the same object, then a convolutional network, which is translation invariant, must assign the same color to each copy.

If convolutional operators are inappropriate, then, we must abandon them in favor of non-convolutional ones. While this sounds complex, we suggest that very simple modifications of convolutional operators, which we call semi-convolutional, may suffice. In particular, if Φ_u() is the output of a convolutional operator at pixel u, then we can construct a non-convolutional response by mixing it with information about the pixel location. Mathematically, we can define a semi-convolutional operator as:

    Ψ_u() = f(Φ_u(),u)

where f : ℒ×Ω→ℒ' is a suitable mixing function. As our main example of such an operator, we consider a particularly simple type of mixing function, namely addition. With it, <ref> specializes to:

    Ψ_u() = Φ_u() + u,     Φ_u() ∈ℒ = ℝ^2.

While this choice is restrictive, it has the benefit of having a very simple interpretation. Suppose in fact that the resulting embedding can perfectly separate instances, in the sense that Ψ_u() = Ψ_v() ⇔∃ k : (u,v)∈ S_k. Then for all the pixels of the region S_k we can write in particular:

    ∀ u ∈ S_k :   Φ_u() + u = c_k

where c_k∈ℝ^2 is an instance-specific point. In other words, we see that the effect of learning this semi-convolutional embedding for instance segmentation is to predict a displacement field Φ() that maps all pixels of an object instance to an instance-specific centroid c_k. An illustration of the displacement field can be found <ref>.



  §.§.§ Relation to Hough voting and implicit shape models.
 <Ref> are reminiscent of well known detection methods in computer vision: Hough voting <cit.> and implicit shape model (ISM) <cit.>. Recall that both of these methods map image patches to votes for the parameters θ of possible object occurrences. In simple cases, θ∈ℝ^2 can be the centroid of an object, and casting votes may have a form similar to <ref>.

This establishes, a clear link between voting-based methods for object detection and coloring methods for instance segmentation. At the same time, there are significant differences. First, the goal here is to group pixels, not to reconstruct the parameters of an object instance (such as its centroid and scale). <Ref> may have this interpretation, but the more general version <ref> does not. Second, in methods such as Hough or ISM the centroid is defined a-priori as the actual center of the object; here the centroid c_k has no explicit meaning, but is automatically inferred as a useful but arbitrary reference point. Third, in traditional voting schemes voting integrates local information extracted from individual patches; here the receptive field size of Φ_u() may be enough to comprise the whole object, or more. The goal of <ref> is not to pool local information, but to solve a representational issue.



 §.§ Learning additive semi-convolutional features


Learning the semi-convolutional features of <ref> can be formulated in many different ways. Here we adopt a simple direct formulation inspired by <cit.> and build a loss by considering, for each image  and instance S∈𝒮 in its segmentation, the distance between the embedding of each pixel u∈ S and the segment-wise mean of these embeddings:

    ℒ(Ψ|,𝒮)
    =
    ∑_S∈𝒮1/|S|∑_u ∈ SΨ_u()-
     1/|S|∑_u ∈ SΨ_u()
    .

Note that while this quantity resembles the variance of the embedding values for each segment, it is not as the distance is not squared; this was found to be more robust. 

Note also that this loss is simpler than the margin condition (<ref>) and than the losses proposed in <cit.>, which resemble (<ref>) more closely. In particular, this loss only includes an “attractive” force which encourages embeddings for each segment to be all equal to a certain mean value, but does not explicitly encourage different segments to be assigned different embedding values. While this can be done too, empirically we found that minimizing <ref> is sufficient to learn good additive semi-convolutional embeddings.



 §.§ Coloring instances using individuals' traits


In practice, very rarely an image contains exact replicas of a certain object. Instead, it is more typical for different occurrences to have some distinctive individual traits. For example, different people are generally dressed in different ways, including wearing different colors. In instance segmentation, one can use such cues to tell right away an instance from another. Furthermore, these cues can be extracted by conventional convolutional operators.

In order to incorporate such cues in our additive semi-convolutional formulation, we still consider the expression Ψ_u(x) = û + Φ_u(). However, we relax Φ_u()∈ℝ^d to have more than two dimensions d > 2. Furthermore, we define û as the pixel coordinates of u, u_x and u_y, extended by zero padding:

    û = [ u_x u_y   0       0 ]^⊤∈ℝ^d.

In this manner, the last d-2 dimensions of the embedding work as conventional convolutional features and can extract instance-specific traits normally.





 §.§ Steered bilateral kernels


The pixel embedding vectors Ψ_u() must ultimately be decoded as a set of image regions. Again, there are several possible strategies, starting from simple K-means clustering, that can be used to do so. In this section, we consider transforming embeddings in an affinity matrix between two pixels, as the latter can be used in numerous algorithms.

In order to define the affinity between pixels u,v∈Ω, consider first the Gaussian kernel

    K(u,v) = exp( - Ψ_u() - Ψ_v()^2/2).

If the augmented embedding <ref> is used in the definition of Ψ_u() = û + Φ_u(), we can split Φ_u() into a geometric part Φ^g_u() ∈ℝ^2 and an appearance part Φ^a_u()∈ℝ^d-2 and expand this kernel as follows:

    K(u,v) =
    exp(
    - (u + Φ^g_u()) - (v + Φ^g_v())^2/2)
    exp(
    - Φ_u^a() - Φ_v^a()^2/2).

It is interesting to compare this definition to the one of the bilateral kernel:[In the bilateral kernel, a common choice is to set Φ^a_u() = _u ∈ℝ^3 as the RGB triplet for the appearance features.]

    K_bil(u,v) =
    exp(- u - v^2/2)
    exp(
    - Φ_u^a() - Φ_v^a()^2/2).

The bilateral kernel is very popular in many applications, including image filtering and mean shift clustering. The idea of the bilateral kernel is to consider pixels to be similar if they are close in both space and appearance. Here we have shown that kernel (<ref>) and hence kernel (<ref>) can be interpreted as a generalization of this kernel where spatial locations are steered (distorted) by the network to move pixels that belong to the same underlying object instance closer together.

In a practical implementation of these kernels, vectors should be rescaled before being compared, for example in order to balance spatial and appearance components. In our case, since embeddings are trained end-to-end, the network can learn to perform this balancing automatically, but for the fact that (<ref>) implicitly defines the scaling of the spatial component of the kernel. Hence, we modify <ref> in two ways: by introducing a learnable scalar parameter σ and by considering a Laplacian rather than a Gaussian kernel:

    K_σ(u,v) = exp( - Ψ_u() - Ψ_v()/σ).

This kernel is more robust to outliers (as it uses the Euclidean distance rather than its square) and is still positive definite <cit.>. In the next section we show an example of how this kernel can be used to perform instance coloring.  



 §.§ Semi-convolutional Mask-RCNN


The semi-convolutional framework we proposed in <ref> is very generic and can be combined with many existing approaches.
Here, we describe how it can be combined with the Mask-RCNN (MRCNN) framework <cit.>, the current state-of-the-art in instance segmentation.

MRCNN is based on the RCNN propose & verify strategy and first produces a set of rectangular regions ℛ, where each rectangle R ∈ℛ tightly encloses an instance candidate.
Then a fully convolutional network (FCN) produces foreground/background segmentation inside each region candidate. In practice, it labels every pixel u_i in R with a foreground score logit s(u_i)∈ℝ.
However, this is not an optimal strategy for articulated objects or occluded scenes (as validated in <ref>), as it is difficult for a standard FCN to perform individual foreground/background predictions. Hence we leverage our pixel-level translation sensitive embeddings in order to improve the quality of the  predictions s(u_i).

Extending MRCNN   
Our approach is based on two intuitions: first, some points are easier to be recognized as foreground than others, and, second, once one such seed point has been determined, its affinity with other pixels can be used to cut out the foreground region.

In practice, we first identify a seed pixel u_s in each region R using the MRCNN foreground confidence score map 𝐬 = [ s(u_1), …, s(u_|R|)]. We select the most confident seed point as u_s = argmax_1≤ i ≤ |R| s(u_i), evaluate the steered bilateral kernel K_σ(u_s,u) after extracting the embeddings Ψ_u_s for the seed and Ψ_u_i of each pixel u_i in the region, and then defining updated scores ŝ(u_i) as  
ŝ(u_i) = s(u_i) + logK_σ(u_s,u_i). The combination of the scores and the kernel is performed in the log-space
due to improved numerical stability.
The final per-pixel foreground probabilities are obtained as in <cit.> with sigmoid(ŝ(u_i)).

The entire architecture —the region selection mechanism, the foreground prediction, and the pixel-level embedding —is trained end-to-end. For differentiability, this requires the following modifications: 
we replace the maximum operator with a soft maximum over the scores _s = softmax(𝐬) and we obtain the seed embedding Ψ_u_s as the expectation over the embeddings Ψ_u
under the probability density _s.
The network optimizer minimizes, together with the MRCNN losses, the image-level embedding loss ℒ(Ψ|,𝒮) 
and further attaches a secondary binary cross entropy loss that, similar to the MRCNN mask predictor,
minimizes binary cross entropy between the kernel output K_σ(u_s,u_i) and the ground truth instance masks.

The predictors of our semi-convolutional features Ψ_u were implemented as an output of a shallow subnetwork, shared between all the FPN layers. This subnet consists of a 256-channel 1×1 convolutional filter followed by ReLU and a final 3×3 convolutional filter producing D=8 dimensional embedding Ψ_u. 
Due to an excessive sensitivity of the RPN component to perturbations of the underlying FPN representation, we downscale the gradients that are generated by the shallow subnetwork and received by the shared FPN tensors by a factor of 10.



 §.§ The convolutional coloring dilemma


In this section, we prove some properties of convolutional operators in relation to solving instance segmentation problems. In order to do this, we need to start by formalizing the problem.

We consider signals (images) of the type : Ω→ℝ, where the domain Ω is either ℤ^m or ℝ^m.[We assume that the domain extends to infinity to avoid having to deal explicitly with boundary conditions.] In segmentation, we are given a family ∈𝒳 of such signals, each of which is associated to a certain partition 𝒮_ = {S_1,…,S_K_} of the domain Ω. The goal is to construct a segmentation algorithm  𝒜 : ↦𝒮_ that computes this function. We look in particular at algorithms that pre-process the signal by assigning a label Φ_u()∈ℒ to each point u∈Ω of the domain. Furthermore, we assume that this labeling operator Φ is local and translation invariant[We say that Φ is translation invariant if Φ_u((· - τ))= Φ_u-τ() for all translations τ∈Ω.
We say that it is also local if there exists a constant M > 0 such that x_u=x_u' for all |u-u'|<M implies that Φ_u() = Φ_u'().] so as to be implementable with a convolutional neural network.

There are two families of algorithms that can be used to segment signals in this manner, discussed next.

Propose & verify The first family of algorithms submits all possible regions S_r ⊂Ω, indexed for convenience by a variable r, to a labeling function Φ_r() ∈{0,1} that verifies which ones belong to the segmentation 𝒮_ (i.e. Φ_r() =1 ⇔ S_r ∈𝒮_). Since in practice it is not possible to test all possible subsets of Ω, such an algorithm must focus on a smaller set of proposal regions. A typical choice is to consider all translated squares (or rectangles) S_u = [-H,H]^m + u. Since the index variable u∈Ω is now a translation, the operator Φ_u() has the form discussed above, although it is not necessarily local or translation invariant.

Instance coloring The second family of approaches directly colors (labels) pixels with the index of the corresponding region, i.e. 
Φ_u() = k ⇔ u ∈ S_k.
Differently from P&V, this can efficiently represent arbitrary shapes. However, the map Φ needs implicitly to decide which number to assign to each region, which is a global operation. Several authors have sought to make this more amenable to convolutional networks. 
A popular approach <cit.> is to color pixels arbitrarily (for example using vector embeddings) so that similar colors are assigned to pixels in the same region and different colors are used between regions, as already detailed in <ref>.



r0pt
[scale=0.5]
  in -2,-1,0,1,2  (2*-1,0) – (2*,1); (2*,1) – (2*+1,0); ;
    in -2,-1,0,1,2 
      [thick,arrows=to-to] (2*-1,-.3) – node[below] S_(2*+1,-.3) ;;
  [thick,-latex](-5,0) – (5,0) node[right]u;
  [thick,-latex](0,0) – (0,1.4);



Convolutional coloring dilemma Here we show that, even with the variants discussed above, IC cannot be approached with convolutional operators even for cases where these would work with P&V.
 
We do so by considering a simple 1D example. Let  be a signal of period 2 (i.e. x_u+2 = x_u) where for u∈[-1,1] the signal is given by x_u = min(1-u,1+u). Suppose that the segmentation associated to  is 𝒮={[-1,1]+2k,k∈ℤ}. If we assume that a necessary condition for a coloring-based algorithm is that at least some of the regions are assigned different colors, we see that this cannot be achieved by a convolutional operator. In fact, due to the periodicity of , any translation invariant function will assign exactly the same color to pixels 2k,k∈ℤ. Thus all regions have at least one point with the same color.

On the other hand, this problem can be solved by P&V using the proposal set {[-1,1]+u,u∈Ω}  and the local and translation invariant verification function Φ_u() = [x_u = 1], which detects the center of each region.

The latter is an extreme example of a convolutional coloring dilemma: namely, a local and translation invariant operator will naturally assign the same color to identical copies of an object even if when they are distinct occurrences (c.f. interesting concurrent work that explores related convolutional dilemmas <cit.>).

Solving the dilemma  Solving the coloring dilemma can be achieved by using operators that are not translation invariant. In the counterexample above, this can be done by using the semi-convolutional function Φ_u(x) = u + (1 - x_u) ẋ_u. It is easy to show that Φ_u(x)=2k colors each pixel u ∈ S_k = [-1,1]+2k with twice the index of the corresponding region by moving each point u to the center of the closest region. This works because such displacements can be computed by looking only locally, based on the shape of the signal.








§ EXPERIMENTS


We first conduct experiments on synthetic data in order to clearly demonstrate inherent limitations of convolutional operators for the task of instance segmentation. In the ensuing parts we demonstrate benefits of the semi-convolutional operators on a challenging scenario with a high number of  overlapping articulated instances and finally we compare to the competition on a standard instance segmentation benchmark.







 §.§ Synthetic experiments



In <ref> we suggested that convolution operators are unsuitable for instance segmentation via coloring, but that semi-convolutional ones can do. These experiments illustrate this point by learning a deep neural network to segment a synthetic image x_S where object instances correspond to identical dots arranged in a regular grid (<ref> (a)). 

We use a network consisting of a pretrained ResNet50 model truncated after the Res2c layer, followed by a set of 1×1 filters that, for each pixel u, produce 8-dimensional pixel embeddings Φ_u(x_S) or Ψ_u(x_S).  We optimize the network by minimizing the loss from <ref> with stochastic gradient descent. Then, the embeddings corresponding to the foreground  regions are extracted and clustered with the k-means algorithm into K  clusters, where K is the true number of dots present in the synthetic image.

<Ref> visualizes the results. Clustering the features consisting of the position invariant convolutional embedding Φ_u(x_S) results in nearly random clusters (<ref> (c)). On the contrary, the semi-convolutional embedding Ψ_u(x_S)=Φ_u(x_S)+u allows to separate the different instances almost perfectly when compared to the ground truth segmentation masks (<ref> (d)).


 
 









 §.§ Parsing biological images


The second set of experiments considers the parsing of biological images. Organisms to be segmented present non-rigid pose variations, and frequently form clusters of overlapping instances, making the parsing of such images challenging. Yet, this scenario is of crucial importance for many biological studies. 

Dataset and evaluation

We evaluate our approach on the C. Elegans dataset (illustrated <ref>), a subset of the Broad Biomedical Benchmark collection <cit.>. The dataset consists of 100 bright-field microscopy images.  Following standard practice <cit.>, we operate on the binary segmentation of the microscopy images. However, since there is no publicly defined evaluation protocol for this dataset, a fair numerical comparison with previously published experiments is infeasible.  We therefore compare our method against a very strong baseline (MRCNN) and adopt the methodology introduced by <cit.> in which the dataset is divided into 50 training and 50 test images. We evaluate the segmentation using average precision (AP) computed using the standard COCO evaluation criteria <cit.>.
We compare our method against the MRCNN FPN-101 model from <cit.> which attains results on par with state of the art on the challenging COCO instance segmentation task. 

Results
The results are given in <ref>. 
We observe that the semi-convolutional embedding Ψ_u
brings improvements in all considered instance segmentation metrics.
The improvement is more significant at higher IoU thresholds which underlines the importance
of utilizing position sensitive embedding in order to precisely delineate an instance
within an MRCNN crop.





 §.§ Instance segmentation
 

 




 


 
The final experiment compares our method to competition on the instance segmentation task on a standard large scale dataset, PASCAL VOC 2012 <cit.>.

As in the previous section, we base our method on the MRCNN FPN-101 model. Because we observed that the RPN component is extremely sensitive to changes in the base architecture, we employed a multistage training strategy. First, MRCNN FPN-101 model is trained until convergence and then our embeddings are attached and fine-tuned with the rest of the network . We follow <cit.> and learn using 24 SGD epochs, lowering the initial learning rate of 0.0025 tenfold after the first 12 epochs. Following other approaches, we train on the training set of VOC 2012 and test on the validation set.

Results
The results are given in <ref>. Our method attains state of the art on PASCAL VOC 2012 which validates our approach. We further compare in detail against MRCNN in <ref> using the standard COCO instance segmentation metrics from <cit.>. Our method outperforms MRCNN on the considered metrics, confirming the contribution of the proposed semi-convolutional embedding.





§ CONCLUSIONS


In this paper, we have considered dense pixel embeddings for the task of instance-level segmentation. Departing from standard approaches that rely on translation invariant convolutional neural networks, we have proposed semi-convolutional operators which can be easily obtained with simple modifications of the convolutional ones. On top of their theoretical advantages, we have shown empirically that they are much more suited to distinguish several identical instances of the same object, and are complementary to the standard Mask-RCNN approach.


Acknowledgments
We gratefully acknowledge the support of Naver, EPSRC AIMS CDT, AWS ML Research Award, and ERC 677195-IDIU.

splncs04



