A Classification Refinement Strategy for Semantic Segmentation

arXiv:1801.07674v1 [cs.CV] 23 Jan 2018

James W. Davis, Christopher Menart, Muhammad Akbar
Dept. of Comp. Sci. & Eng., Ohio State University
Columbus, OH USA
Email: {davis.1719, menart.1, akbar.24}@osu.edu

Roman Ilin
RYAP, Air Force Research Laboratory
Wright Patterson AFB, OH USA
Email: rilin325@gmail.com

Abstract—Based on the observation that semantic segmentation errors are partially predictable, we propose a compact
formulation using confusion statistics of the trained classifier
to refine (re-estimate) the initial pixel label hypotheses. The
proposed strategy is contingent upon computing the classifier
confusion probabilities for a given dataset and estimating a
relevant prior on the object classes present in the image to
be classified. We provide a procedure to robustly estimate the
confusion probabilities and explore multiple prior definitions.
Experiments are shown comparing performances on multiple
challenging datasets using different priors to improve a stateof-the-art semantic segmentation classifier. This study demonstrates the potential to significantly improve semantic labeling
and motivates future work for reliable label prior estimation
from images.
Keywords-semantic segmentation; confusion probabilities;
probabilistic refinement

I. I NTRODUCTION
Semantic segmentation is a challenging computer vision
problem wherein a class label is assigned to each pixel in
an image. This provides much richer scene information than
traditional image classification, and therefore is inherently
a more difficult task. While image classification requires
the detection of the primary object, semantic segmentation
requires precise, localized pixel-wise detections. Annotating
training images for semantic segmentation in a supervised
manner is more expensive and time-consuming, hence most
public datasets are relatively small in comparison to those
for image classification (e.g., ImageNet [1]).
Current semantic segmentation methods show promising results on multiple datasets, however systematic errors
in their output can be readily found. For example, RefineNet [2] mis-classifies ground as sidewalk so often
in the PASCAL-Context dataset [3] that pixels classified as
sidewalk are 60% likely to actually be ground. In this
example, pixel accuracy would be improved by simply relabeling all sidewalk pixels as ground. However, it is
more preferable to use this confusion knowledge to better
reason about certain situations in a probabilistic manner.
We categorize labeling errors into one of two types of
mistakes. An “in-context” labeling error is when a pixel is
assigned an incorrect label from the set of actual labels for
the given image (e.g., a foreground object label is assigned
to an incorrect location). An “out-of-context” labeling error

Figure 1. Example contextual errors of RefineNet on PASCAL-Context.
In-Context: The sofa back in the left image is classified as floor, which
is incorrect even though the label correctly appears elsewhere in the image.
Out-of-Context: A portion of the right image is classified as motorbike,
which is not a valid label anywhere in the image.

is an invalid pixel label assignment that is outside of the
actual label set for the image (e.g., an indoor object label is
assigned to a pixel in an image of an outdoor scene). Actual
examples of these errors are shown in Fig. 1.
Given that a classifier will have similar performance characteristics on related images, analysis of the classification
errors and label confusions across a dataset should support a
secondary (post-processing) refinement stage to re-estimate
the output label confidences to better classify the pixels,
reducing both in-context and out-of-context errors. The
proposed approach is based on these underlying motivations.
Our method is derived from a direct marginalization of
p(l|d), the probability of label l given input data d, resulting
in a decomposition of the formulation into classifier output
label probabilities and learned classifier/truth confusions.
Applied together, the framework treats the confidence levels
of the original classifier output as partial evidence, and
incorporates the confusion information to determine the final
probability of witnessing each object label at each location
in an image. In a sense, this confusion approach can be considered a form of context-aware re-estimation. We present
a robust method to compute the confusion probabilities and
also outline various label priors used to bias the refinement.

Upper bound performances of the framework with various
priors are reported for three challenging datasets to justify
the approach.
The rest of this paper is organized as follows. Section
II describes related work in semantic segmentation. Section
III describes our framework for classifier refinement using
label confusion probabilities and priors. Section IV presents
experimental results showing the performance improvements
of the approach on multiple datasets, followed by a conclusion in Sect. V
II. R ELATED W ORK
Convolutional Neural Networks (CNNs) such as [2], [4]–
[6] have achieved unprecedented results in semantic segmentation. However, they also introduce a tension between
precise localization and inclusion of broader image context
[7]. Many recent innovations in network architecture were
motivated by this issue, including dilated convolution [5],
multi-scale prediction [8], zoom-out features [9], recursive
convolutional networks [10], and convolutional upsampling
[2], [11].
Context is exploited using additional techniques, such
as K-nearest-neighbors methods [12], conditional random
fields (CRFs) [13], [14], and precision matrices [15]. These
approaches remain relevant in part due to their flexibility.
Research has continued to investigate estimating and employing co-occurrence statistics for CRFs [16], and some
recent methods have attempted to integrate CRFs directly
into neural models [17], [18]. Our proposed method provides
a Bayesian framework to incorporate context formed by
confusion probabilities and label priors to influence pixellevel decisions.
The most related work to the proposed framework is the
approach of LabelBank [19]. Both methods process secondary information to adjust the confidence values produced
by a base classifier. LabelBank uses a CNN architecture
to estimate which classes are present in the image, then
essentially filters out labels in the output of the base semantic
segmentation classifier that do not belong to this list. The
method therefore focuses on the removal of out-of-context
errors. Our strategy is more flexible, as we include additional
confusion probabilities and relax the requirement of binary
estimation of the classes, which together allow the method
to be more general to remove both in-context and outof-context errors. Furthermore, our approach is supported
within a Bayesian framework, rather than applying an ad
hoc masking technique. We will show in the experiments
that the upper bound performance capability of our approach
is above the LabelBank optimum.
As we describe our procedure to refine the classifier
output label probabilities, one may be tempted to view this
as a form of boosting. Similar to boosting, our approach is a
staged process, first running the base classifier then refining
the output hypotheses (two stages). However, boosting trains

a series of multiple (weak) classifiers using re-weighted
examples based on the errors from the previous stage(s).
III. A PPROACH
Let C be a given semantic segmentation classifier that
generates pixel-wise label hypotheses for images. The output
of C at each pixel site provides a confidence/probability
for every possible class (typically attained using softmax).
For image data di at each pixel site i for each possible
label l ∈ L, we denote the classifier output probabilities
as P (C(di ) = l|di ). Note that we differentiate this from
the typical classifier output notation of P (l|di ). The final
predictions are usually made by choosing the label that is
assigned the highest confidence/probability at each site.
We propose a Bayesian method to exploit the confusions
exhibited by classifier C to probabilistically refine and
improve the estimate of the best label at each site. We begin
by examining P (ligt |di ), the probability of site i having its
ground truth label (ligt ) given the image data di . Note that
this probability is not the classifier output probability, and
hence is currently independent of any particular classifier.
A marginalization of this probability over the possible
classifier outputs yields
X
P (ligt |di ) =
P (ligt , C(di ) = c|di )
(1)
c∈L

=

X

P (ligt |C(di ) = c, di ) · P (C(di ) = c|di )(2)

c∈L

=

X

P (ligt |C(di ) = c) · P (C(di ) = c|di )

(3)

c∈L

The first term in the two-part decomposition of Eqn. 3 represents the confusion relationship between the ground truth
label and the classifier C outputs. The second term contains
the classifier’s output probabilities, as initially described.
Applying Bayes’ Rule to the first term in Eqn. 3 yields
P (ligt |C(di ) = c) =

P (C(di ) = c|ligt ) · P (ligt )
P (C(di ) = c)

(4)

where the denominator P (C(di ) = c) is computed from the
integration of the numerator over all possible labels in the set
L. We relax these probabilities to be both site-independent
and data-independent for ease in learning and for generalization. Hence the probability for any pixel (regardless of
site location or image) having any label l ∈ L given the
classifier’s selected class label c is
P (l|C(di ) = c) =

P (C = c|l) · P (l)
P (C = c)

(5)

where P (C = c|l) is the generalized confusion (or mismatch) probability between a given/known label and the
classifier’s output, and P (l) is a prior on the label l.
Merging Eqn. 5 into Eqn. 3 yields the refinement, or
re-estimation, formulation of the original site-dependent

classifier output probabilities P (C(di ) = c|di ), ∀c ∈ L into
more confusion-aware probabilities. The probability of any
output label l ∈ L given the input data is now described by
X P (C = c|l) · P (l) · P (C(di ) = c|di )
P (l|di ) =
(6)
P (C = c)
c∈L

Hence Eqn. 6 can be used to determine the new refined
probabilities for each possible label l ∈ L at pixel site i.
We can efficiently formulate this process employing a
single transformation matrix with all possible label combinations. An |L| × |L| refinement matrix R is defined
as R(c1 , c2 ) = P (c1 |C = c2 ), given by all c1 , c2 ∈ L
instantiations within Eqn. 5. For pixel site i in the test image,
we construct the vector of site-dependent base classifier
output probabilities as Xi (c) = P (C(di ) = c|di ), ∀c ∈ L.
Following Eqn. 6, ∀l ∈ L, and employing matrix R, it is
equivalent to refine the original classifier output probabilities
with the linear transformation
X̂i = R · Xi

(7)

As usual, the selected class label for site i will be the
label/entry in X̂i (instead of Xi ) that is maximal.
This refinement process is conducted for each site in the
test image being classified. Since the matrix R is computed
only once (and fixed) for the entire image using siteindependent data (with Eqn. 5), the re-estimation procedure
is quite fast and computationally efficient.
The refinement framework is straightforward to derive,
but as we will show, can be used to significantly improve
performance. We next present the means to robustly compute
confusion probabilities and detail various forms of the prior
expectations needed for this process.
A. Confusion Probabilities
The site-independent classifier/truth confusion (error)
probabilities P (C = c|l) required in Eqn. 6 can be readily
learned from data by collecting all sites (for all images
employed) having a specific ground truth label and determining the frequency of those sites the classifier assigned label
c ∈ L. To ensure all probabilities are defined, a minimum
value of 10−4 is used for any confusion pairing (before
the final normalization of counts into probabilities) if the
number of sites is 0 for a particular classification output
given the ground truth label.
Region Border Artifacts. Region border labeling artifacts
are common to all semantic segmentation classifiers due to
imprecise manual ground truth labeling along the borders
of objects, spatial pooling, etc. For example, in Fig. 2(a)
we show an example training image, and in Fig. 2(b) we
present its ground truth region labels. The ground truth
region borders (i.e., edge map) are shown in Fig. 2(c). The
pixel labeling errors on this image made by the segmentation
classifier RefineNet-Res101 [2] (trained on the dataset that
includes this image) are highlighted in Fig. 2(d). As is

(a) Image

(b) Truth

(c) Truth borders

(d) Classifier errors

Figure 2. Region border classification errors. (a) Input image. (b) Ground
truth regions. (c) Region borders. (d) Pixels mislabeled by the classifier.

typical with other semantic segmentation classifiers, we
consistently find labeling errors along the borders of regions.
In this example, the classifier performed particularly well for
the image as a whole, with nearly all of the errors occurring
only at the region borders (other images will show additional
larger errors within regions).
When learning the classifier/truth confusions, these border
pixel errors should be ignored as they can pollute the
desired statistics. They are merely artifacts that should not
be considered. Therefore we limit the computation of the
P (C = c|l) confusions to only non-border pixel sites to
avoid negatively influencing the confusion probabilities.
To accomplish this, we first find all ground truth region
border pixels in each image, then perform a spatial dilation
of those border pixels. Only the complement of this pixel
set in the image is employed for the computation of the
confusion probabilities. The resultant confusion probabilities
are computed across all of the images to be analyzed and
provides a single robust confusion representation for the
entire dataset (not per image).
We additionally note that this underlying labeling issue
at region borders therefore suggests that pixel-wise evaluations and comparisons of semantic segmentation approaches
would be more appropriate if ignoring these pixels (which
they currently do not discount).
Training vs. Test/Validation Data. It is also important to
compute the confusion probabilities from data not used to
train the base classifier C. The confusions from training data
will not be well representative of the types of confusions that
will occur on additional (validation, test) data.
To illustrate, we begin by representing the confusion
probabilities P (C = c|l) as an |L| × |L| matrix with the
rows representing the class labels c ∈ L given by the base

5

10

10

15

15

20

20

Classifier Ouput Label

Classifier Ouput Label

B. Defining Prior Expectations
5

25
30
35
40

25
30
35
40

45

45

50

50

55

55
5

10

15

20

25

30

35

Ground Truth Label

40

45

50

55

(a) Training confusions

5

10

15

20

25

30

35

Ground Truth Label

40

45

50

55

(b) Testing confusions

Figure 3.
Confusion probability matrices constructed from PASCALContext using RefineNet-101 on (a) training and (b) testing data.

classifier output (best class hypothesis) and the columns
signifying the ground truth classes l ∈ L (same set of
classes). Therefore each (m, n) cell in the matrix holds
the value P (C = m|n). In a scenario with absolutely no
confusions, this matrix would be strictly diagonal. Any offdiagonal entries denote errors that exist in the output of the
classifier. Note that this matrix is different from the earlier
refinement matrix R.
To demonstrate the need for computing these confusion
probabilities from separate validation data (instead of the
training data itself), we show in Fig. 3(a) the confusion probability matrix generated from the PASCAL-Context training
set (∼5K images) using RefineNet-Res101 (trained on this
data). Notice the strong diagonal structure, showing that the
classifier did indeed fit the training data well. However,
to compare, that same trained classifier was employed on
the PASCAL-Context test set (∼5K images, with ground
truth available) and the resulting confusion probabilities
are shown in Fig. 3(b). These results show much greater
off-diagonal confusions, demonstrating that the classifier is
prone to making more and qualitatively different types of
mistakes on unseen data. Hence, these confusion probability
differences demonstrate the need to learn the confusions
from examples separate from the training data, ideally from
a validation set if one is available
However, we have found that many semantic segmentation
approaches do not follow the standard machine learning
practice of withholding a separate validation set out of the
training data. They instead tend to use all available training
data and validate directly on the test set (as ground truth for
test data is typically available). As previously stated, it is
recommended for the proposed approach to retain a small
set of validation data (not used to train the base classifier)
to learn the confusion probabilities. If no validation data is
available when using a pre-trained model, a small random
sample of test data could be employed.

With the confusion probabilities now computed, the only
remaining component necessary for classification refinement
is the label prior P (l). This term represents a prior expectation of a given class label l, and can be defined in several
ways.
Uniform Prior: The simplest naive approach would be
to employ a non-informative uniform prior. We refer to this
version as Punif orm (l) = 1/|L|. With this prior all effort of
refining the classifier probabilities would be placed onto the
learned confusions.
Global Prior: The prior could also be interpreted globally
as the probability of observing class l for any randomly
selected pixel across the dataset (using ground truth information). This prior can be computed simply by L1-normalizing
the histogram of class labels collected throughout the entire
training dataset. We refer to this prior as Pglobal (l). However, this formulation may bias the refinement to the most
dominant class labels in the dataset.
Binary Prior: Alternatively, we could specify the prior
locally, estimating it for each image individually. A binary
prior has a constant value for those classes present in the
image and zero elsewhere (with the L1-norm equal to 1).
We refer to this prior as Pbinary (l). It should be easier to
estimate this image-level prior than dense per-pixel label
assignments (i.e., semantic segmentation). This prior has
also been explored in LabelBank [19].
Histogram Prior: Similar to the binary prior, we can
construct a more informative non-binary prior using the L1normalized histogram of classes present in the image. We
refer to this prior as Phist (l). We posit that estimating this
class-frequency histogram of labels for an image will be
more difficult than for the binary prior, but similarly should
be easier to estimate than the full per-pixel label assignments
of an image.
Unconstrained Prior: Lastly, the prior could be viewed
more abstractly, separated from any binary or frequencybased constraint on its interpretation. In this case, for a
given collection of annotated and classified sites in an image
along with the confusion probabilities, the best possible prior
could be solved using non-linear multi-variate techniques by
minimizing the negative log-loss of the final refined class
probabilities for the target ground truth labels. Based on
Eqn. 6, this is given by solving for P (l) within the loss
Loss = −

#sites
X



log max P (ligt |di ), 

(8)

i=1

where  = 10−10 . In our experiments with this prior, we used
Matlab’s fmincon solver with the interior-point algorithm.
We refer to the resulting (solved) unconstrained prior as
Puc (l).
We will examine and compare these different prior formulations in Sect. IV, though we note that other defini-

tions/interpretations of the priors may be possible.

5
20

10
15

A. Confusion Probabilities
We first present the computed confusion probability matrices for the three datasets. As discussed in Sect. III,
the use of training (instead of validation) data to compute
the confusion probabilities P (C = c|l) can lead to poor
performance. Since validation hold-out sets were not used
by RefineNet for any of the datasets, we instead used the
suggested random sample of test data (as pseudo-validation)
to compute the confusion probabilities for each dataset.
The number of randomly sampled test images for each
dataset was 500 (of 2000) for ADE20K, 1000 (of 5105)
for PASCAL-Context, and 200 (of 654) for NYUDv2.
The resulting confusion probability matrices using
the approach outlined in Sect. III-A are shown in
Fig. 4. There are clearly many off-diagonal confusions present in each dataset. Some example confusion situations include {animal/rug, river/water}
in ADE20K, {sofa/computer, ground/sidewalk} in
PASCAL-Context, and {door/wall, box/other-prop}
in NYUDv2. We also note the similarity of the sampled
confusion probabilities for PASCAL-Context in Fig. 4(b) to
the confusions derived from the entire test set (shown in
Fig. 3(b)).

Classifier Output Label

60

80

100

20
25
30
35
40
45

120

50
55

140
20

40

60

80

Ground Truth Label

100

120

140

5

(a) ADE20K

10

15

20

25

30

35

Ground Truth Label

40

45

50

55

(b) PASCAL-Context

5

10

Classifier Output Label

We present experimental results and analyses of the
proposed refinement approach across 3 challenging datasets
commonly used for semantic segmentation. ADE20K [20]
is a recent dataset of varied scenes with 20K training and
2000 testing images having a dense labeling of 150 classes.
PASCAL-Context [3] is a densely-labeled dataset of various
scenes with 59 labels and is comprised of 4998 training
and 5105 testing images. NYUDv2 [21] contains RGB-D
images (RGB only is analyzed) of indoor rooms (bedrooms,
kitchens, bathrooms, etc.) with 795 training and 654 testing
images (we employed the typical 425 × 560 image crop, as
provided by [22]).
We selected the RefineNet [2] CNN semantic segmentation model as the base classifier to be refined with our
approach. RefineNet is a ResNet-based architecture that employs residual chained pooling to localize coarse predictions
using high-resolution features from multiple paths. This
model has been previously trained on the aforementioned
datasets and has been publicly-released online (RefineNetRes101 trained on PASCAL-Context & NYUDv2, and
RefineNet-Res152 trained on ADE20K). Note that RefineNet includes the background (void) class as a valid
output label, but in this work we removed background label
from their outputs and renormalized the remaining label
confidences.

Classifier Output Label

40

IV. E XPERIMENTS

15

20

25

30

35

40

5

10

15

20

25

Ground Truth Label

30

35

40

(c) NYUDv2
Figure 4. Confusion probability matrices generated using test data samples
for the examined datasets.

B. Refinement with Different Priors
We analyzed the standard score metrics of pixel accuracy and mean Intersection-over-Union (IoU) for various
refinements of the base classifier using the different priors
described in Sect. III-B. The resulting scores from the base
classifier on the three datasets are presented in the top row
of Tables I & II.
Results using the non-informative uniform prior
Punif orm (l) = 1/|L| and global prior Pglobal (l) with the
corresponding dataset confusion probability matrix (from
Fig. 4) are provided in the second and third rows of the
tables. The two methods are generally within ±1% of the
base classifier, hence there is no significant change after
refinement (as expected). The pixel accuracy scores for
Pglobal (l) are higher than Punif orm (l), which is a result of
the bias to more dominate classes in the dataset. The lower
mean IoU scores for Pglobal (l) also reflect this bias.
We next examined the use of the more informative imagebased priors, where we present the ceiling (best possible
performance) metrics of refinement across the datasets. We
used the ground truth data available to compute the optimal
priors for each definition, leaving estimation of these particular priors to future work. This study is used to demonstrate
the full potential of the proposed strategy and motivate future
efforts.
For a relevant baseline of the image-based priors, we first
show (in the fourth row of the tables) the upper bound ceiling
results possible with the LabelBank [19] method. This ap-

Base Classifier (RefineNet [2])
Uniform Prior Punif orm (l)
Global Prior Pglobal (l)
LabelBank [19] (upper bound)
Binary Priors Pbinary (l) (upper bound)
Histogram Priors Phist (l) (upper bound)
Unconstrained Priors Puc (l) (upper bound)

ADE20K
78.6%
77.0%
80.0%
83.7%
85.9%
88.1%
89.0%

PASCAL-Context
78.2%
64.6%
77.7%
85.7%
87.0%
88.4%
90.2%

NYUDv2
72.9%
72.7%
72.9%
76.8%
78.1%
80.3%
82.6%

Table I
P IXEL ACCURACY SCORES ON TEST DATA BEFORE / AFTER VARIOUS REFINEMENTS .

Base Classifier (RefineNet [2])
Uniform Prior Punif orm (l)
Global Prior Pglobal (l)
LabelBank [19] (upper bound)
Binary Priors Pbinary (l) (upper bound)
Histogram Priors Phist (l) (upper bound)
Unconstrained Priors Puc (l) (upper bound)

ADE20K
34.0%
35.1%
34.6%
44.9%
53.6%
57.7%
59.3%

PASCAL-Context
50.5%
49.8%
49.1%
62.0%
65.1%
68.7%
72.2%

NYUDv2
45.2%
45.5%
43.8%
53.6%
55.9%
59.2%
62.5%

Table II
M EAN I O U SCORES ON TEST DATA BEFORE / AFTER VARIOUS REFINEMENTS .

proach removes (only) the out-of-context label errors using
the optimal binary mask on the output label probabilities. We
see a significant improvement in the ceilings of both pixel
accuracy (3.9-7.5%) and mean IoU (8.4-11.5%) over the
base classifier. We note that this method is equivalent to our
refinement approach using the optimal/known binary priors
Pbinary (l) with an identity confusion probability matrix (no
confusions).
Employing the binary priors Pbinary (l), as above, but
now with the computed confusion matrix, performance is
improved. The fifth row of the table shows an increase of
1.3-2.2% for pixel accuracy and 2.3-8.7% for mean IoU
above the previous optimal LabelBank method (uses no
confusion information). This shows that use of the confusion
probabilities indeed can be used to increase the performance
by additionally removing many in-context errors (not possible with the LabelBank masking method).
Using the histogram priors Phist (l) provides a further
increase above the Pbinary (l) results, with an additional
increase of 1.4-2.2% for pixel accuracy and 1.6-3.6% in
mean IoU. This is expected, given the prior is better adjusted
to reflect the true distribution of labels in each image.
Lastly, the unconstrained priors Puc (l) give the expected
best upper bound performance of all the prior versions,
yielding an additional increase over Phist (l) of 0.9-2.3% in
pixel accuracy and 1.6-3.3% in mean IoU. As these priors
are solved to optimize the given refinement formulation, the
improvements should be close to the maximum theoretically
possible results.

C. Discussion and Future Directions
Overall, we see possible refinement gains over the base
classifier of 9.7-12.0% in pixel accuracy and 17.3-25.3% for
mean IoU across the datasets. These upper bound results
show that there is still much room for improvement in the
classifier that can be achieved using refinement with the
confusion probabilities and appropriate priors. When the
prior is defined globally (independent of the image), as
in Punif orm (l) or Pglobal (l), and used together with the
confusion probabilities to refine the test images, there is
minimal benefit (or even a slight loss in some datasets). The
notable gains are achieved when the priors are computed
independently for each test image. The presented results
with the image-based priors Pbinary (l), Phist (l), and Puc (l)
demonstrate that increasingly higher gains are possible in
both pixel accuracy and mean IoU with the proposed strategy.
Based on these results, future work will address in more
detail how particular prior expectations can be reliably
estimated from image data. One may consider constructing the priors from the base classifier’s existing semantic
segmentation results (e.g., determine Pbinary (l) or Phist (l)
from the histogram of labels given by the classifier for a
test image), but this refinement yields no improvement as
it is obviously biased to the base classifier results. One
possible method may be to find other similar training images
for a test image and use a weighted combination of their
priors, as images that appear similar likely have related
label priors. Another path to explore is a specialized deep
learning technique to estimate the prior directly from the
input image itself. Unlike the semantic segmentation task

(or even image classification) which is typically a trained to
1-hot categorical vectors of class labels, the target outputs
here are multinomial distributions.
Though an approach already exists in [19] to estimate
Pbinary (l), the normalized label histogram prior Phist (l)
shows a higher performance ceiling and hence a method to
estimate this prior would be more beneficial. We expect the
unconstrained priors Puc (l), which resulted in the highest
performance metrics, will be more difficult to estimate from
test images as the priors are a function of both the image
labels present and the base classifier confusion probabilities.
Hence we suggest that estimation of the histogram-based
priors Phist (l) may be the best focus for future research.
As the proposed method continues to produce label probabilities/confidences, it can therefore be easily integrated
with other processing strategies or further context-based
reasoning. For example, instead of probabilistically refining
each pixel site independently of other sites, a more holistic
approach, such as formulating a spatial random field (e.g.,
CRF), could be employed to produce a more spatiallyconsistent solution.
V. C ONCLUSION
Semantic segmentation is a complex visual task requiring
reliable and consistent pixel-wise label predictions. Current
models produce systematic errors which can be exploited to
refine the initial confidences of the classifier. We presented a
general Bayesian strategy for classifier refinement, deriving a
straight-forward and compact description comprised of confusion probabilities, label priors, and classifier confidences.
By understanding the types of mistakes made in the classifier
label assignments, we demonstrated the strong potential of
the proposed method by successfully refining a state-of-theart classification model on multiple datasets. The results
further motivate future research into reliably estimating prior
expectations from images.

[4] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional
networks for semantic segmentation,” in CVPR, 2015.
[5] L.-C. Chen, G. Papanderou, I. Kokkinos, K. Murphy, and
A. L. Yuille, “DeepLab: Semantic image segmentation with
deep convolutional nets, atrous convolution, and fully connected CRFs,” arXiv:1606.00915, 2016.
[6] G. Lin, C. Shen, A. van den Hengel, and I. Reid, “Efficient
piecewise training of deep structured models for semantic
segmentation,” in CVPR, 2016.
[7] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. VillenaMartinez, and J. Garcia-Rodriguez, “A review on deep
learning techniques applied to semantic segmentation,”
arXiv:1704.06857, 2017.
[8] A. Raj, D. Maturana, and S. Scherer, “Multi-scale convolutional architecture for semantic segmentation,” Robotics
Institute, Carnegie Mellon University, Tech. Rep. CMU-RITR-15-21, October 2015.
[9] M. Mostajabi, P. Yadollahpour, and G. Shakhnarovich, “Feedforward semantic segmentation with zoom-out features,” in
CVPR, 2015.
[10] F. Visin, M. Ciccone, A. Romero, K. Kastner, K. Cho, Y. Bengio, M. Matteucci, and A. Courville, “Reseg: A recurrent
neural network-based model for semantic segmentation,” in
CVPR, 2016.
[11] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” Medical
Image Computing and Computer-Assisted Intervention, pp.
234–241, 2015.
[12] J. Tighe and S. Lazebnik, “Scalable nonparametric image
parsing with superpixels,” Int. J. Comp. Vis., vol. 101, no. 2,
pp. 329–349, 2013.

ACKNOWLEDGMENT

[13] J. Shotton, J. Winn, C. Rother, and A. Criminisi, “TextonBoost: Joint appearance, shape and context modeling for
multi-class object recognition and segmentation,” in ECCV,
2006.

This research was supported by the United States
Air Force Research Laboratories under contract No.
GRT00044839/60056946, Subcontract 60056946.

[14] P. Krähenbül and V. Koltun, “Efficient inference in fully
connected CRFs with Gaussian edge potentials,” in NIPS,
2011.

R EFERENCES

[15] N. Souly and M. Shah, “Scene labeling using sparse precision
matrix,” in CVPR, 2016.

[1] Z. Deng, S. Todorovic, and L. J. Latecki, “Semantic segmentation of RGBD image with mutex constraints,” in ICCV,
2015.
[2] G. Lin, A. Milan, C. Shen, and I. Reid, “Refinenet: Multi-path
refinement networks for high-resolution semantic segmentation,” in CVPR, 2017.
[3] R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille, “The role of context for object
detection and semantic segmentation in the wild,” in CVPR,
2014.

[16] W. Zhang, S. Zeng, D. Wang, and X. Xue, “Weakly supervised semantic segmentation for social images,” in CVPR,
2015.
[17] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet,
Z. Su, D. Du, C. Huang, and P. H. Torr, “Conditional random
fields as recurrent neural networks,” in ICCV, 2015.
[18] A. Arnab, S. Jayasumana, S. Zheng, and P. H. Torr, “Higher
order conditional random fields in deep neural networks,” in
ECCV, 2016.

[19] H. Hu, Z. Deng, G.-T. Zhou, F. Sha, and G. Mori, “LabelBank: Revisiting global perspectives for semantic segmentation,” arXiv:1703.09891, 2017.
[20] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and
A. Torralba, “Scene parsing through ADE20K dataset,” in
CVPR, 2017.
[21] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor
segmentation and support inference from RGBD images,” in
ECCV, 2012.
[22] S. Gupta, P. Arbeláez, and J. Malik, “Perceptual organization
and recognition of indoor scenes from RGB-D images,” in
CVPR, 2013.

