Three Birds One Stone: A General Architecture for Salient Object Segmentation, Edge Detection and Skeleton Extraction
Qibin Hou Jiang-Jiang Liu Ming-Ming Cheng Ali Borji Philip H. S. Torr





































Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals


























Three Birds One Stone: A General Architecture for Salient Object Segmentation, Edge Detection and Skeleton Extraction
    Qibin Hou,
        Jiang-Jiang Liu,
        Ming-Ming Cheng,
        Ali Borji,
        and Philip H.S. Torr, Senior Member, IEEE



Q. Hou, M.M. Cheng, and J. Liu  are with CCCE Nankai University.
M.M. Cheng is the corresponding author (cmm@nankai.edu.cn).
Ali Borji is with the University of Central Florida.
P.H.S. Torr is with the University of Oxford.
Manuscript received April 19, 2005; revised August 26, 2015.
    
=====================================================================================================================================================================================================================================================================================================================================================================================================================================





In this paper, we aim at solving pixel-wise binary problems, including
salient object segmentation, skeleton extraction, and edge detection, 
by introducing a general architecture.

Previous works have proposed tailored methods for solving each of the 
three tasks independently.

Here, we show that these tasks share some similarities that 
can be exploited for developing a general architecture.

In particular, we introduce a horizontal cascade of encoders so as to 
gradually advance the feature representations from the original CNN trunks.

To better fuse feature at different levels, 
the inputs of each encoder in our architecture
is densely connected to the outputs of its previous encoder.

Stringing these encoders together allows us to effectively exploit 
features across different levels hierarchically 
to effectively address multiple pixel-wise binary regression tasks.

To assess the performance of our proposed network on these tasks, 
we carry out exhaustive evaluations on multiple representative datasets.

Although these tasks are inherently very different, 
we show that our approach performs very well on all of them 
and works far better than current single-purpose state-of-the-art methods.

We also conduct sufficient ablation analysis to let readers 
better understand how to design encoders for different tasks.

The source code in this paper will be publicly available after acceptance.




Salient object segmentation, edge detection, skeleton extraction, general architecture.






§ INTRODUCTION
 

Convolutional neural networks (CNNs) have been widely used in most 
fundamental computer vision tasks (semantic segmentation <cit.>, 
edge detection <cit.>,
salient object segmentation <cit.>,
skeleton extraction <cit.>.)
and have achieved unprecedented performance on many tasks.

To date, most of the existing methods are designed only for a single task
because different tasks often favor different types of features.

Their design criterion is single-purpose, 
greatly restricting their applicability to other tasks <cit.>.

For example, the Holistically-nested Edge Detector (HED) <cit.> 
works well for the edge detection task but does not perform well for 
salient object detection <cit.>
and skeleton extraction <cit.>.

The reason is that the architecture proposed in HED does not consider 
how to capture homogeneous region information and scale-variant 
(either thick or thin) skeletons.


In this paper, our goal is to present a general architecture for 
solving three important binary problems,
including salient object segmentation, edge detection, and skeleton extraction.

As popular low-level vision tasks, all of them have been widely studied recently.

In fig:teaser, 
we illustrate a 2D space representing features that these tasks favor.

Specifically, salient object segmentation, 
as addressed in many existing works <cit.>,
requires the ability to extract homogeneous regions and hence relies more 
on high-level features (fig:arch_compc and <ref>f).

Edge detection aims at detecting accurate boundaries, 
thus it needs more low-level features to sharpen the coarse edge maps 
produced by deeper layers <cit.>
(fig:arch_compd and <ref>e).

Skeleton extraction <cit.>, on the other hand, prefers high-level semantic information to detect scale-variant (either thick or thin) skeletons.

From the standpoint of the network architecture, in spite of three different tasks,
all of them require multi-level features in varying degrees (See fig:teaser).






Consequently, a natural question is whether it is possible to combine multi-level features
in a proper way such that stronger and more general feature representations can be
constructed for solving all of these tasks.










To solve the above question, 
rather than simply combining the multi-level features extracted
from the trunk of CNNs as done in most existing works
<cit.>, 
we propose to horizontally construct a cascade of encoders 
to gradually encode signals from the CNN trunks (fig:archa)
to make the final representations more powerful.

Each encoder is composed of multiple transition nodes, 
each of which gets input from its former encoder,
enabling subsequent encoders to efficiently select features 
from the backbone in a dense manner.

As the signals from the backbone pass through the encoders sequentially, 
more and more advanced feature representations can be built 
that can be applied to different tasks.

As shown in fig:arch_comp, 
our approach is more general compared to existing relevant methods.

To evaluate the performance of the proposed architecture, 
we apply it to three binary tasks—
salient object detection, edge detection, and skeleton extraction.

Experimental results show that our approach outperforms existing methods 
on multiple widely used benchmarks.

Specifically, for salient object detection, 
compared to previous state-of-the-art works,
our method has a performance gain of nearly 2% on average on 5 popular datasets.

For skeleton extraction, we also significantly improve the state-of-the-art results 
by more than 2% in terms of F-measure.

Furthermore, to let readers better understand the proposed approach, 
we conduct a series of ablation experiments for all three tasks.


To sum up, the contributions of this paper can be summarized as follows:

  
  * First, we analyse the similarities as well as differences 
  	among salient object segmentation, edge detection, and skeleton extraction 
  	and design a general architecture to solve all these tasks;
  
  * Second, we propose to construct a horizontal cascade of encoders
    to progressively extract more general feature representations 
    that can be competent for all these tasks.



The rest of the paper is organized as follows.
sec:relatedWorks reviews a number of recent works that are
strongly related to our tasks and meanwhile analyzes the differences 
among different CNN-based skip-layer architectures.
section:architecture  presents the observations of this paper 
and describe the architecture of our proposed approach in detail.
sec:sod-<ref> compares the proposed approach with other 
state-of-the-art results and at the same time provide sufficient ablation analysis 
to let readers better understand how each component works in our architecture. 
Finally, sec:discussion concludes the whole paper and highlights
some potential research directions.







§ RELATED WORKS


In this section, we first review a number of popular methods that are strongly related to
the tasks we solve and then analyze CNN-based skip-layer architectures
that have been proposed recently.



 §.§ Salient Object Detection


Earlier salient object detection methods mostly rely on hand-crafted features, 
including either local contrast cues <cit.> 
or global contrast cues
<cit.>.

Interested readers may refer to some notable review and benchmark papers <cit.> for detailed descriptions.

Apart from the classic methods, a number of deep learning based methods have recently emerged.

In <cit.>, He presented a superpixel-wise
convolutional neural network architectures to extract hierarchical contrast features
for predicting the saliency value of each region.

Li <cit.> proposed to feed different levels of image segmentation
into three CNN branches and aggregate the multi-scale output features
to predict whether each region is salient.

Wang <cit.> considered both local and global information
by designing two different networks.

The first one is used to learn local patch features to provide each
pixel a saliency value and then, multiple kinds of information are merged
together as the input to the second network to predict the
saliency score of each region.

Similarly, in <cit.>, Zhao designed two different CNNs
to independently capture the global and local context information of each segment patch,
and then fed them into a regressor to output a saliency score for each patch.


The above methods took as input image patches and then used CNN features to predict the
saliency of each input region (either a bounding box <cit.> or
a superpixel <cit.>).

Later works, benefiting from the high efficiency of 
fully convolutional networks <cit.> (FCNs),
utilize the strategy in which spatial information is processed in CNNs, 
and hence produce remarkable results.

Lee <cit.> combined both high-level semantic features
extracted from CNNs and hand-crafted features and then utilized a unified fully-connected
neural network to estimate saliency score of each query region.

Liu <cit.> refine the details of the prediction maps progressively
by harnessing recurrent fully convolutional networks.

In <cit.>, Li combined a pixel-level fully convolutional stream and
a segment-wise spatial pooling stream into one network to better leverage the
contrast information of the input images.

Hou <cit.> and Li <cit.> hierarchically fused
multi-level features in a top-down manner.

More recently, Zhang <cit.> designed a generic architecture to
aggregating multi-level CNN features.

In <cit.>, Zhang embedded R-Dropout and hybrid upsampling layers
into an encoder-decoder structure to better localize the salient objects.



 §.§ Edge Detection.


Early edge detection works <cit.> 
mostly relied on various gradient operators.

Later works, such as <cit.>, 
were driven by manually-designed features and were able to 
improve the performance compared to gradient-based works.

Recently, with the emergence of large scale datasets, learning-based methods <cit.> 
gradually became the main stream for edge detection.

Further, recent CNN-based methods
<cit.> 
have started a new wave in edge detection research.

Ganin <cit.> proposed to use both CNN features and 
the nearest neighbour search to detect edges.

In <cit.>, the contour data was separated into different subclasses 
and different model parameters were learned for each subclass.

Hwang <cit.> viewed edge detection as a per-pixel classification problem by learning
a feature vector for each pixel.


Different from pixel/patch level analysis methods,
Xie and Tu <cit.> designed a holistically-nested edge detector (HED) 
by introducing the concept of side supervision into a fully convolutional networks 
to merge features from different levels.

In <cit.>, multiple cues are considered to improve the precision of edges.

Liu <cit.> advanced the HED architecture by extracting richer 
convolutional features from CNNs.






 §.§ Skeleton Extraction.


Earlier methods <cit.> mainly relied on 
gradient intensity maps of natural images to extract skeletons.

Later learning-based methods viewed skeleton extraction as a per-pixel classification problem.

In <cit.>, Tsogkas and Kokkinos calculated hand-crafted multi-scale and
multi-orientation features for each pixel and utilized multiple instance learning to predict
the score of each pixel.

Sironi <cit.> attempted to learn distance to the closest skeleton
segment for each pixel.

There are also some approaches <cit.> computing the similarity between superpixels and combined them
by clustering or filtering schemes.

Recent skeleton detection methods <cit.>
are mainly based on the holistically-nested edge detector (HED).

In <cit.>, Shen et al. introduced supervision 
in different blocks by guiding the scale-associated side outputs 
toward ground-truth skeletons at different scales and then fused
multiple scale-associated side outputs in a scale-specific manner 
to localize skeleton pixels at multiple scales.

Ke et al. <cit.> added multiple shortcuts 
from deeper blocks to shallower ones based on
the HED architecture such that high-level semantic information 
can be effectively transmitted to lower side outputs, yielding stronger features.




 §.§ CNN-Based Skip-Layer Architectures


Unlike most classification tasks which adopt the classic bottom-up structures 
(fig:arch_compa),
region segmentation and edge detection tasks depend on how homogeneous regions 
are extracted and how edges are sharpened.

Intuitively, considering the fact that lower network layers are capable 
of capturing local details while higher layers capture high-level contextual details, 
a good solution to satisfy the above needs might be introducing 
skip-layer architectures <cit.>.

One of the recent successful CNN-based skip-layer structures is the HED architecture
<cit.>, which learns rich hierarchical features by means
of adding side supervision to each side output (fig:arch_compd).

This architecture treats features at different levels equally, 
and therefore allows enough edge details to be
captured through lower side outputs.

Afterwards, several follow-up works 
<cit.>
adopted similar structures (e.g., by introducing deep supervision) 
to capture richer feature representations by fusing features at different levels.


There are also some other work modifying this structure by adding 
short connections <cit.>
from upper layers to lower ones to better leverage multi-level features 
for salient object detection and skeleton extraction or 
gradually refine the coarse-level features in a top-down manner 
<cit.>.

These approaches, however, only attempt to simply combine multi-level features.

There is still a large room for extracting richer feature representations 
for these pixel-wise binary regression problems.






§ THE PROPOSED GENERAL ARCHITECTURE



In this section, we elaborate on the similar characteristics 
shared by salient object detection, skeleton extraction,
and edge detection tasks and propose a general architecture that can treat them all.




 §.§ Key Observations
 





































Previous works leverage multi-level features by either fusing simple features 
in a top-down manner (fig:arch_compc) or introducing shortcuts 
followed by side supervision (fig:arch_compd and <ref>f).

What they have in common is that each layer in the decoder 
(the right part of each diagram in fig:arch_comp) can only 
receive features from the backbone or its upper layers in the decoder.

These types of designs may work well for salient object detection but 
may fail when applied to edge detection and skeleton extraction (and vice versa).

The fundamental reason behind this is the fact that 
the feature representations formed in the decoders
are not powerful enough to deal with all of these tasks.


Taking into account the nature of salient object detection, 
edge detection, and skeleton extraction,
a straightforward way to build more advanced feature representations 
is to add a couple of groups of transition nodes such that multi-level features 
from the backbone can be sequentially
combined multiple times until the representations are strong enough.

In the following subsections, we will show how to construct a general structure
that can include all the features each task favors.






 §.§ Overview of Our Proposed Architecture



An illustrative diagram of our network is shown in fig:archa.

Structurally, our architecture can be decoupled into multiple components 
{S_i} (0≤ i ≤ N), 
each of which performs different functions.

Each component can be either an encoder ℰ, 
encoding the feature representations from its previous
component, or a decoder 𝒟 that decode its inputs to the final results.

Thus, the output ℛ for an input ℐ can be obtained by

    ℛ = 𝒟(ℰ_N-1(⋯ℰ_1(ℰ_0(ℐ, θ_0),θ_1) ⋯,θ_N-1), θ_N),


where {θ_i} (0≤ i ≤ N) are the learnable 
parameters for {S_i} (0≤ i ≤ N), respectively.

In our architecture, the first encoder ℰ_0 corresponds to the backbone of 
some classification network (VGGNet <cit.> here).

ℰ_0 is mainly used to extract the first-tier multi-level and multi-scale features
from the input images, similarly to most CNN-based architectures.

In the following sections, for convenience, we view ℰ_1 as our first encoder.

The decoder 𝒟 receives signals from the last encoder ℰ_N-1 and
can have different forms depending on the task at hand.

The responsibility of the decoder is to decode the multi-level features from the 
last encoder ℰ_N-1 and output the final results.

Each middle encoder is composed of multiple transition nodes T, 
each of which receives input from its last
encoder and sends responses to the next component for rebuilding higher-level features.



For notational convenience, each block[
The definition of block here refers to the layers that share the same resolution 
in a baseline model (VGGNet <cit.>).] 
in ℰ_0 is treated as a transition node as well.

A sequence of components forms the so-called horizontal cascade, 
which transmits the multi-level features from the backbone to the decoder horizontally.

Our architecture, obviously, can be treated as the generalized case of previous work 
(See fig:arch_comp) and hence is quite different from them.





 §.§ Encoders






Each middle encoder ℰ_i (1≤ i ≤ N-1) encodes features from its previous one
and is composed of multiple transition nodes that are used to fuse multi-level input features
in different ways.


Side Path. 
To better interpret the architecture of our proposed approach, 
we introduce the concept of side path in this paragraph, 
which is similar to the notation of side output in
<cit.>.

Each side path, in our case, starts from the end of a block in CNNs
and ends before the decoder.

A typical example can be found in fig:arch, 
which is represented by the dark green dash arrow with an enhanced 
thickness for highlighting.

Obviously, there are totally four standard side paths in fig:arch 
plus a short one which is connected to the last block in ℰ_0.

At the beginning of each side path, we can optionally add a stack of consecutive 
convolutional layers followed by ReLU layers on each according to different tasks.

This is inspired by <cit.>, who have shown that adding
more convolutional layers improves salient object detection.

In what follows, we neglect the specific number of these convolutional layers 
for the sake of convenience.

Detailed settings of each side path can be found in sec:impl_details.




















Transition Node.
Formally, for any positive integer k (k < N), 
let T_m^k denote the mth transition node in ℰ_k.

Transition node T_m^k is able to selectively receive signals (features) 
from transition nodes that are not shallower than itself 
in its previous encoder ℰ_k-1.

In this way, upper transition nodes can only get inputs from deeper side paths, 
preserving the original high-level features that are informative 
for generating homogeneous regions.

Additionally, lower transition nodes receive features from multiple levels, 
allowing these features to
be merged efficiently to produce even more advanced representations.

fig:archa provides an illustration,
in which transition nodes are represented by colorful solid circles, and
fig:archb shows a representative structure of a transition node.




Notice that, in fig:archa, we only show a case where the transition nodes between adjacent
encoders are densely connected.

In fact, the connection patterns can be decided according to different kinds of tasks.


Internal Structure.
The multi-level feature maps extracted from the backbone model 
usually contains different channel numbers and resolutions.

To ensure that features from different levels can be fused together in our architecture,
each input of a transition node is passed through a convolutional layer with the
same number of channels, followed by an upsampling layer to
make sure that all the feature maps share the same size.

The hyper-parameters of upsampling layers can be easily inferred from 
the context of our network, 
which will be elaborated in the experiments sections.

For fusion, all feature maps with the same size are merged together by simple summation.

Concatenation operation can also be used here but we empirically found that performance gain 
is negligible in all tasks.

An extra convolutional layer is added to eliminate the aliasing effect caused 
by the summation operation.

Furthermore, we also consider introducing an optional identity 
path as shown in fig:archb.

In this way, each transition node is allowed to automatically learn whether the signals 
from other side paths are redundant, allowing our architecture more flexible.



 §.§ Horizontal Hierarchy

A number of recent works have leveraged multi-level features by introducing a series of
top-down paths based on the backbone of a classification network.

In our network, each transition node is able to optionally receive signals from
its previous encoder.

Unlike the fusing strategy in <cit.> which only combines the score maps
from different side paths, our architecture allows more signals to be transmitted to
the next encoder or to the decoder.

In this way, each encoder is allowed to fuse features at
different levels from the backbone, allowing the output features to reach higher levels.

By adding a stack of encoders, as shown in
fig:archa, we are able to further advance the feature levels.

Therefore, when the number of encoders increases, a horizontal hierarchy is formed.




Horizontal Signal Flow.
Let ℰ_k be the kth encoder.




The set of all transition nodes forms a multi-tree, a special directed acyclic graph (DAG)
whose vertices and edges are composed of all the transition nodes and the connections
between each pair of transition nodes.

With these definitions, a horizontal signal flow in the DAG starts from the end of an arbitrary
transition node in ℰ_0 and ends before the decoder.

Furthermore, the vertices through which it passes should not be lower than its starting transition node.

The dark green thick arrow in fig:archa depicts a representative horizontal signal flow.

When applied to different tasks, the edges can be selectively discarded.

In the following experiments sections, we will further elaborate on this and discuss
how to better leverage the horizontal signal flows for different types of binary vision tasks.





 §.§ Diverse Decoders
 

The form of the decoder is also very important when facing different tasks.

To date, many decoders (fig:arch_comp) with various structures have been developed.

Here, we describe two of them which we found to work very well for the three binary tasks to be solved
in this paper, respectively.

The first one corresponds to the structure in fig:archa, which has been adopted by many segment
detection related tasks.

This structure gradually fuses the features from different side paths in a top-down manner.

Since the feature channels from different side paths may vary, to perform the summation operation,
a convolutional layer with kernel size 3 × 3 is used, if needed.

In spite of only one loss layer, we found that one loss layer performs better than the structure in
fig:archd which introduces the concept of side supervision <cit.> (See tab:group_decoder).

We will provide more details on the behaviors of them in the experiments sections.

For edge detection and skeleton extraction, we employ the same form of decoder as in <cit.> (fig:archc).

Edge detection and skeleton extraction require the ability of sharpening thick lines detected and
thereby rely more on low-level features compared to segmentation.

Adding side supervision to the end of each side path allows more detailed edge information to be
emphasized.

On the other hand, the side predictions can also be merged with the weighted-fusion layer as the final output,
providing better performance.









 §.§ Implementation Details
 



As most prior works chose VGGNet <cit.> as their pre-trained model, for fair comparison, we base our model on this architecture too.

For salient object detection, we replace 3 fully-connected layers with 3 convolutional layers (conv6),
the same structure to conv5.

We change the stride of pool5 to 1 and the dilatation of conv6 to 2 for large receptive fields,
and add 2 convolutional layers at the beginning of each side path following <cit.>.

From side path 1 to 6, the strides are 1, 2, 4, 8, 16, and 16, respectively, and the corresponding
channel numbers of convolutional layers are set to 32, 64, 128, 256, 512, and 512, respectively.

We adopt the architecture shown in Table <ref> as our default setting 
and the decoder in fig:decodersa as our default decoder in our experiments.

For edge detection, we change the stride of pool4 layer to 1 and set the dilation rate of convolutional layers
in conv5 to 2 as in <cit.>.

The convolutional layers in each transition node are all with 16 channels by default.

The connection patterns can be found in Table <ref>. We adopt the same decoder as in <cit.> (fig:decodersc).

For skeleton extraction, the network structure is the same as in Table <ref>
aside from the channel numbers in side paths which correspond to 32, 64, 128, and 256 from side path 1
to 4, respectively.

Thus, the strides of side paths 1 to 4 are 1, 2, 4, and 8, respectively.

All the convolutional layers mentioned here are with kernel size 3 and stride 1.






§ APPLICATION I: SALIENT OBJECT DETECTION


In this section, we apply the proposed approach to salient object segmentation. 
Salient object segmentation is an important pixel-wise binary problem 
and has attracted a lot of attention recently. We first describe the importance of
each component in our architecture by a series of ablation experiments and then
compare our method with other state-of-the-art methods.







 §.§ Evaluation Measures and Datasets


Here, we use two universally-agreed, standard, and easy-to-understand
measures <cit.> for evaluating the existing deep saliency models.

We first report the F-measure score, which simultaneously considers recall and precision, the overlapping area
between the subjective ground truth annotation and the resulting prediction maps.

The second measure we use is the mean absolute error (MAE) between the estimated saliency map and ground-truth annotation.



We perform evaluations on 5 datasets, including MSRA-B <cit.>,
ECSSD <cit.>, HKU-IS <cit.>,
SOD <cit.>, and DUT-OMRON <cit.>.

For training, we first use the 2,500 training images from MSRA-B.

We also try a larger training set incorporating another 2,500 images from HKU-IS as done in <cit.>.

Notice that all the numbers reported here are from the results
the authors have presented or the results we obtained by running their publicly available code.





 §.§ Ablation Studies







The hyper-parameters are as follows: weight decay (0.0005), momentum (0.9), and
mini-batch size (10).

The initial learning rate is set to 5e-3 and is divided by 10 after 8,000 iterations.

We run the network for 12,000 iterations and choose our best model according to the performance
on the validation set <cit.>.

Further, we also use the fully connected CRF model <cit.> which is the same as
in <cit.> as a post-processing tool for maintaining spatial coherence.



The Number of Encoders.
The number of encoders plays an important role in our approach.

Some prior works (e.g., <cit.>) have shown good results with the decoders
directly connected to the backbone.

However, when we add the first encoder, a small improvement is achieved in terms of
F-measure score.

Quantitative results can be found in Table <ref>.

When we add another encoder, further improvements on the F-measure and MAE scores are obtained.

We also added additional encoders but introducing more encoders yield no
further  improvements. 

This might be due to the fact that feature representations after two times fusion have already
reached the top level of our architecture.

More Training Data.
The amount of training data is essential for CNN-based methods.

Besides training on the MSRA-B dataset as done in <cit.>, we also
attempt to add more training data as in <cit.>.

In Table <ref>, we show the results using different training sets.

With another 2,500 training images from <cit.>, our performance can be
further improved about 1% in terms of F-measure on average. 

Therefore, we believe more high-quality training data can help. 

The Effect of Horizontal Signal Flows.
By default, for salient object detection, we use a dense way to connect each pair of transition nodes
from adjacent encoders.

To show the effect of horizontal signal flows, we attempt to simplify our network by reducing the inputs of
each transition node (i.e., the dash arrows in fig:archa).

In Table <ref>, we list four different patterns of our proposed architecture.




As can be seen, when we gradually reduce the number of horizontal signal flows, 
the performance decreases accordingly.

This phenomenon indicates that more top-down connections between
transition nodes helps segmentation type tasks.

The Roles of Different Decoders.
The structure of the decoder also affects the performance of our approach.

We try two different structures (Figs. <ref>b and <ref>d) as our decoders.

Although the structure in fig:archb was helpful in <cit.>, we obtain no performance
gain by such a structure but a slight decrease in F-measure (See tab:group_decoder).


This phenomenon reveals that introducing side supervision as in <cit.> is 
not always a good strategy for salient object segmentation.

Different network architectures may favor different types of decoders.














 §.§ Comparison with the State-of-the-Art


We exhaustively compare our proposed approach with 14 existing state-of-the-art salient object detection
methods including 2 classic methods (GC <cit.> and DRFI <cit.>) and 10 recent CNN-based methods (LEGS <cit.>, MC <cit.>, MDF <cit.>, DCL <cit.>, RFCN <cit.>, DHS <cit.>, ELD <cit.>, DISC <cit.>, MSRNet <cit.>, DSS <cit.>, UCF <cit.>, Amulet <cit.>).

Notice that MSRNet <cit.> and DSS <cit.> are two of the best models to date.

Here, our best results are shown at the bottom of Table <ref>. 



F-measure and MAE Scores.
Here, we compare our approach with the aforementioned approaches in terms of F-measure and MAE (See Table <ref>).



As can be seen, our model trained on the MSRA-B dataset already outperforms all of the existing methods
in terms of F-measure.

With more training data, the results are improved further by a large margin (1% on average).

This phenomenon is more pronounced when testing on the HKU-IS dataset.

Notice that our approach does better than the best existing model <cit.>
using F-measure (2% improvement on average).

Similar patterns can also be observed using the MAE score.


Visual Comparisons.
In fig:visual_results, we show the visual comparisons with several previous state-of-the-art
approaches. In the top row, the source image have salient objects
with complex textures. As can be seen, our approach is able to successfully segment all salient objects in images
with boundaries being accurately highlighted. Some other methods such as DSS <cit.> and
DCL <cit.> also produce high quality segmentation maps, although inferior to our results. A similar phenomenon also happens when processing images where contrast between foreground and
background is low or when salient objects are tiny and irregular. Compared with existing methods, our approach performs
much better in both cases. See for example the case shown at the bottom row of fig:visual_results. These results demonstrate that our proposed horizontal hierarchy is capable of
capturing rich and robust feature representations when applied to salient object detection.








§ APPLICATION II: EDGE DETECTION




In this section, we apply our GearNet to edge detection, 
one of the popular and basic low-level tasks in computer vision.































The hyper-parameters used in our experiment include mini-batch size set to 10, momentum set to 0.9, weight decay set to 2e-4, and initial learning rate set to 1e-6 which is divided by 10 after 23,000 iterations.

Our network is trained for 30,000 iterations.

We evaluate our GearNet on the Berkeley Segmentation Dataset and Benchmark (BSDS 500) <cit.>, which is one of notable benchmarks in the edge detection field.

This dataset contains 200 training, 100 validation, and 200 testing images, each with accurately
annotated boundaries.

Besides, our training set also incorporates the images from the PASCAL Context Dataset <cit.> and performs data augmentation
as in <cit.> for fair comparisons.

Similar to previous works, we use the fixed contour threshold (ODS) and per-image best threshold (OIS)
as our measures.

Before evaluation, we apply the standard non-maximal suppression algorithm to get thinned edges.



 §.§ Ablation Analysis


The Number of Encoders.
The number of encoders also plays an important role in the edge detection task.

We consider the RCF network <cit.> as a special case of GearNet with 0 encoders.

From Table <ref>, we observe that adding 1 encoder (the bottom row) based on
the RCF architecture helps us obtain an increase of 0.4% in terms of ODS.

When we add 2 middle encoders, the ODS score can be further improved by 0.3 points.

We observed no significant improvement when adding more than two encoders.








The Number of Channels.
As stated in Sec. <ref>, the convolutional layers in each side path are all with 16 channels.

To explore how the channel numbers used in each side path effect the performance of our architecture,
we also attempt to increase the channel numbers as done in <cit.> (21 channels).

However, the results show that more channels in each side path gives worse performance, leading to
a decrease of around 0.2 points in terms of ODS.

Similar phenomenon was also encountered when decreasing the number of channels.

 

The Effect of Horizontal Signal Flows.

We also analyze the number of horizontal signal flows in this paragraph.

While more horizontal signal flows helps salient object segmentation,
we found that this operation does not boost the performance in edge detection.

In Table <ref>, we attempt to simplify our network by 
reducing the inputs of each transition node (Patterns 1 and 2).

According to the experimental results, these two patterns degrade the performance by 
nearly 0.4 and 0.6 points in ODS, respectively.

Furthermore, when we try to increase the number of horizontal signal flows in our architecture
as in Pattern 3 in Table <ref>, the performance decreases as well.

This demonstrates different kinds of low-level features are essential to edge detection.

However, too many high-level features also harm the quality of the predicted edges because of
the lack of rich detailed information.



















 §.§ Comparison with the State-of-the-Art


We compare our results with results from 12 existing methods, including gPb-owt-ucm <cit.>, SE-Var <cit.>, MCG <cit.>, DeepEdge <cit.>, DeepContour <cit.>, HED <cit.>, CEDN <cit.>, RDS <cit.>, COB <cit.>, CED <cit.>, DCNN+sPb <cit.>, and RCF-MS <cit.>, most of which are CNN-based methods.



 

Quantitative Analysis. In Table <ref>, we show the quantitative results
of 12 previous works as well as ours.

With only one encoder, our method achieves ODS of 0.815 and OIS of 0.834, which are
already better than the most of the previous works.

This indicates that fusing features from different blocks of VGGNet performs better than
combining only the feature maps from the same block <cit.>.

On the other hand, more side supervision does help learning rich feature representations
<cit.>.

Furthermore, when we add another encoder to our architecture as in Table <ref>, our results
can be further enhanced from ODS of 0.815 to ODS of 0.818 (+0.003).

For OIS, a similar phenomenon can also be found in Table <ref>.

Visual Analysis. In fig:edge_vis, we show some visual comparisons between our approach and a leading representative method <cit.>.

As can be observed, our approach performs better in detecting the boundaries compared to the other one.

In fig:edge_visb, it is apparent that the real boundaries of the plants are highlighted well.

In addition, thanks to the fusion mechanism in our approach, the features learned by our network
are much more powerful compared to <cit.>.

This is because the areas with no edges are rendered much cleaner, especially in Figs. <ref>a and
<ref>b.

To sum up, in spite of less than 1 point improvement compared to <cit.>,
the quality of our results is much higher visually.

PR Curve Comparisons.
 The precision-recall curves of some selected methods can be found in fig:edge_pr.
 
 One can observe that the PR-Curve produced by our approach is already better than
 humans in some certain cases and is better than all previous methods.

























































§ APPLICATION III: SKELETON EXTRACTION


In this section, we apply our GearNet to skeleton extraction.




We will show that our method substantially outperforms priors works by a large margin.





 §.§ Ablation Analysis

The hyper-parameters we use are as follows: weight decay set to 0.0002, momentum set to 0.9, mini-batch size set to 10, and initial learning rate of 1e-6 which is divided by 10 after 20,000 iterations.

We run the network for 30,000 iterations.

A standard non-maximal suppression algorithm is used to obtain thinned skeletons before evaluation.

We use the same training set to <cit.>. 

The Number of Encoders. As in salient object detection and edge detection, reducing the number
of encoders when carrying out skeleton detection degrades the performance.

When we remove the second middle encoder ℰ_2, the F-measure score drops by more than 2 points.

Adding more encoders in our approach also leads to no performance gain.

The Number of Channels.
In our experiment, similarly to edge detection, we also try to reduce the number of channels 
in each side path to half of each and observe that the results slightly decrease
by 0.5 points.

When we further reduce the channel numbers to a quarter of each, the performance drops dramatically
(by more than 5 points).

This phenomenon indicates that skeleton extraction relies on more information from the backbone.

This can be achieved by adding proper number of channels in each side path.

In addition, we also attempt to increase the number of channels by doubling them but 
find no performance gain.




The Effect of Horizontal Signal Flows.
The number of horizontal signal flows also affects the skeleton results.

We reduce the number of optional connections between ℰ_1 and ℰ_2 
from 3 to 2 but the F-measure score decreases by around 2 points.

Reducing the number of optional connections between ℰ_0 and
ℰ_1 leads to a similar phenomenon.

This indicates introducing connections between higher and lower layers is essential
for skeleton detection.



 



 §.§ Comparison with the State-of-the-Arts

We compare our GearNet with 4 recent CNN-based methods (HED <cit.>, FSDS <cit.>, LMSDS <cit.>, and SRN <cit.>) on 2 popular and challenging datasets including SK-LARGE <cit.> and WH-SYMMAX <cit.>.

Similar to <cit.>, we use the F-measure score to evaluate the quality of prediction maps.

In Table <ref>, we show quantitative comparisons with existing methods.

As can be seen, our method wins dramatically by a large margin (3.4 points) on the SK-LARGE dataset <cit.>.

There is also an improvement of 2.1 points on the WH-SYMMAX dataset <cit.>.




In fig:sk_vis, we also show some visual illustrations of three approaches (more can be found in
our supplementary materials).

Owing to the advanced features extracted from our GearNet, our method is able to more accurately locate
the exact positions of the skeletons.

This point can also be substantiated by the fact that our prediction maps are also much thinner than other works.

Both quantitative and visual results unveil that our horizontal hierarchy provides a better way
to combine different-level features for skeleton extraction.




PR Curve Comparisons
In fig:skel_sklarge_pr, we also show the precision-recall curses on two datasets, including SK-LARGE <cit.> and WH-SYMMAX <cit.>.

As can be seen, quantitatively, our approach on both datasets substantially outperforms other existing methods, which verify the statement we explain above.






§ DISCUSSION AND CONCLUSION


In this paper, we present a novel architecture for binary vision tasks and apply it to
three drastically different example tasks including salient object segmentation, edge detection, and skeleton extraction.

Notice, however, that our approach is not limited to these tasks and can be potentially applied to a wide variety of binary pixel labeling tasks in computer vision.

In order to take more advantage of CNNs, we introduce the concept of transition node, which receives signals from different-level features maps.







Exhaustive evaluations and comparisons with recent notable state of the art methods on widely used datasets shows that our framework outperforms all of them in all three tasks, testifying the power of our proposed framework.

Further, we structurally analyze our proposed architecture using several ablation experiments in each task and investigate the roles of different design choices in our approach. We hope that our work will encourage subsequent research to design universal architectures for computer vision tasks.





























§ ACKNOWLEDGMENTS

This research was supported by the National Youth Talent Support Program,
NSFC (NO. 61620106008, 61572264),
Tianjin Science Fund for Distinguished Young Scholars (17JCJQJC43700),
and Huawei Innovation Research Program.





































IEEEtran






[
    < g r a p h i c s >
]
Qibin Hou is currently a Ph.D. Candidate with College of Computer Science and
Control Engineering, Nankai University, under the supervision of Prof. Ming-Ming Cheng.
His research interests include deep learning, image processing, and computer vision.




[
    < g r a p h i c s >
]
Ming-Ming Cheng received his PhD degree from Tsinghua University in 2012.
Then he did 2 years research fellow, with Prof. Philip Torr in Oxford.
He is now a professor at Nankai University, leading the Media Computing Lab.
His research interests includes computer graphics, computer vision, and image processing.
He received research awards including ACM China Rising Star Award,
IBM Global SUR Award, CCF-Intel Young Faculty Researcher Program, etc.




[
    < g r a p h i c s >
]
Jiangjiang Liu is currently a Master student
with College of Computer Science and
Control Engineering, Nankai University,
under the supervision of Prof. Ming-Ming Cheng.
His research interests include deep learning,
image processing, and computer vision.




[
    < g r a p h i c s >
]
Ali Borji
received the PhD degree in cognitive
neurosciences from the Institute for Studies in
Fundamental Sciences (IPM), 2009.
He is currently an assistant professor at
Center for Research in Computer Vision,
University of Central Florida.
His research interests include visual
attention, visual search, machine learning, neurosciences,
and biologically plausible vision models.





[
    < g r a p h i c s >
]
Philip H.S. Torr received the PhD degree from Oxford University.
After working for another 3 years at Oxford,
he worked for 6 years as a research scientist for Microsoft Research,
first in Redmond, then in Cambridge,
founding the vision side of the Machine Learning and Perception Group.
He is now a professor at Oxford University.
He has won awards from several top vision conferences,
including ICCV, CVPR, ECCV, NIPS, etc.
He is a Royal Society Wolfson Research Merit Award holder.



