WebSeg: Learning Semantic Segmentation from Web Searches
Qibin Hou Ming-Ming Cheng Jiangjiang Liu Philip H. S. Torr




headings


 

WebSeg: Learning Semantic Segmentation from Web Searches

Hou 


[1]Nankai University   [2]University of Oxford 

andrewhoux@gmail.com, cmm@nankai.edu.cn



WebSeg: Learning Semantic Segmentation from Web Searches
    Qibin Hou[1]   Ming-Ming Cheng[1]   Jiangjiang Liu[1]   Philip H.S. Torr[2]
    December 30, 2023
===============================================================================






In this paper, we improve semantic segmentation by
automatically learning from Flickr images associated with a particular keyword,
without relying on any explicit user annotations,
thus substantially alleviating the dependence on accurate annotations when
compared to previous weakly supervised methods.

To solve such a challenging problem, 
we leverage several low-level cues (such as saliency, edges, etc.)
to help generate a proxy ground truth.
Due to the diversity of web-crawled images,
we anticipate a large amount of `label noise' in which
other objects might be present.
We design an online noise filtering scheme
which is able to deal with this label noise, especially in cluttered images.
We use this filtering strategy as an auxiliary module
to help assist the segmentation network in learning cleaner proxy
annotations.
Extensive experiments on the popular PASCAL VOC 2012
semantic segmentation benchmark show surprising good results
in both our WebSeg (mIoU = 57.0%)
and weakly supervised (mIoU = 63.3%) settings.








§ INTRODUCTION




Semantic segmentation, as one of the fundamental computer vision tasks,
has been widely studied.

Significant progress has been achieved
 
on challenging benchmarks,
PASCAL VOC <cit.> and Cityscapes <cit.>.


Existing semantic segmentation algorithms
<cit.>
rely on large-scale pixel-accurate human annotations,
which is very expensive to collect.

To address this problem, recent works focused on
semi-supervised/weakly-supervised semantic segmentation
using user annotations in terms of bounding boxes
<cit.>,
scribbles <cit.>,
points <cit.>, or even keywords <cit.>.

However, using these techniques to learn 
new categories remains a challenging task 
that requires manually
collecting large sets of annotated data. 

Even the the simplest image level keywords annotation might
take a few seconds for a single example <cit.>,
which involves substantial human labour, if we consider exploring
millions/billions of images and hundreds of new categories

in the context of never ending learning <cit.>.














We, as humans, are experts at quickly learning how to identify object regions

by browsing example images searched using the corresponding keywords.


Identifying object regions/boundaries and salient object regions
of unknown categories
easily helps us obtain enough pixel-wise `proxy ground truth' annotations.

Motivated by this phenomenon, 


this paper addresses the following question:
could a machine vision system automatically learn
semantic segmentation of unseen categories
from web images that are the result of search keyword,
without relying on any explicit user annotations?

Very recently, some works, such as <cit.>,
have proposed various web-based learners attempting to solve the semantic segmentation
problem.

These methods indeed leverage web-crawled images/videos.

However, they assume that each image/frame is associated with correct labels
instead of considering how to solve the problem of label noise presenting 
in the web-crawled images/videos.

Our task vs. weakly supervised semantic segmentation.
Conceptually, our task is quite different from
traditional weakly supervised semantic segmentation as we must also deal with label noise.

On one hand, in our task, only a set of target categories are provided
rather than specific images with precise image-level annotations.

This allows our task to be easily extended to cases
where hundreds/thousands of target categories are given
without collecting a large number of images with precise image-level labels.

But this comes at a cost. Due to the typical diversity of web-crawled images,
some of them may contain objects that are
inconsistent with their corresponding query keywords (See fig:web_images).

Therefore, how to deal with this label noise <cit.>
becomes particularly important, which is one of our contributions.










As demonstrated in fig:teaser,
we show some keyword-retrieved web images containing rich appearance varieties
that can be used as training samples for semantic segmentation.

With the help of 
low-level cues
e.g., saliency, over-segmentation and edges,
we are able to solve the semantic segmentation problem 
without relying on precise image-level noise free labels
required by previous weakly supervised methods
<cit.>, which
required attention cues <cit.>.

For instance, edge information provides potential locations for object boundaries.

Saliency maps tell us where the regions of interest are located.

With these heuristics, the object regions that correspond to the query keywords
become quite unambiguous for many images, making the unannotated web images a valuable source to
be used as proxy ground truth for training semantic segmentation models.

All the aforementioned low-level cues are category-agnostic,
making automatically segmenting unseen categories possible.











Given the heuristic maps produced by imperfect heuristic generators,
to overcome the noisy regions,
previous weakly supervised semantic segmentation
methods <cit.>
mostly harness image-level annotations
(e.g. using PASCAL VOC dataset <cit.>)
to correct wrong predictions for each heuristic map. 

In this paper, we consider this ticklish situation from a new perspective,
which attempts to eliminate the negative effect caused by noisy proxy annotations.

To do so, we propose to filter the irrelevant noisy regions by introducing an
online noise filtering module.

As a light weight branch,
we embed it into a mature architecture (Deeplab <cit.>),
to filter those regions with potentially wrong labels.

We show that as long as the most regions are clean,
by learning from the large amount of web data,
we can still obtain good results.

To verify the effectiveness of our approach,
we conduct extensive experiment comparisons on
PASCAL VOC 2012 <cit.> dataset.

Given 20 PASCAL VOC object category names,
our WebSeg system 
retrieves 33,000 web images
and immediately learn from these data without any manual assistant.

The experimental results suggested that our WebSeg system,
which does not use manually labeled image,
is able to produce surprisingly good semantic segmentation results (mIOU = %57.0).

Our results are comparable with recently published
state-of-the-art weakly supervised methods,
which use tens of thousands of manually labeled images with precise keywords annotations.

When additional keyword level weak supervision is provided,
our WebSeg system could achieve an mIoU score of %63.3,
which significantly outperforms
the existing weakly supervised semantic segmentation methods.

We also carefully performed a serials of ablation studies to
verify the importance of each component in our approach.



























In summary, our contributions include:

  
  * an attractive computer vision task,
    which aims at performing semantic segmentation
    by automatically learning from keyword-retrieved web images.
    
  
  * an online noise filtering module (NFM),
  	which is able to effectively filter potentially noisy region labels
    caused by the diversity of web-crawled images.













§ INTERNET IMAGES AND SEVERAL LOW-LEVEL CUES
 

The cheapest way to obtain training images is to
download them from the Internet (Flickr).

Given a keyword or more, we are able to easily obtain
a massive number of images that are very likely to be relevant,
without reliable annotations at keyword or pixel level.



 §.§ Crawling Training Images From the Internet
 

Given a collection of keyword, we first download 2,000 images for each category from the Flickr
website according to relevance.

Regarding the diversity of the fetched images, for example, some of them may have
very complex background and some of them may also contain more than
one semantic category (Fig. <ref>),
we design a series of filtering strategies to
automatically discard those images with low quality.

To measure the complexity of the web images, we adopt the following schemes.

First, we use the variance of the Laplacian <cit.> to determine
whether the input images are blurry.

By convolving an image with the Laplace operator and computing the variance of the resulting map,
we can obtain a score indicating how blurry the image is.

In our experiment, we set a threshold 50 to throw away all the images that are larger than it.

Second, we discard all the images with larger saturation and brightness values by transforming the input images
from RGB color space to HSV space.

This is reasonable as images with lower saturation and brightness are even difficult for humans to recognize.

Specifically, we calculate the mean values of the H and V channels for each image.

If either value is lower than 20, then the corresponding image will be abandoned.

Finally, we get around 33,000 images for training.





 §.§ Low-Level Cues
 

Web-crawled images do not naturally carry object region information
with them.

However, a three-year-old child could easily find precise object regions for most of these images,
even if the corresponding object category has not been explicitly explained to him/her.


This is mostly because of the capacity of humans processing low-level vision tasks.

Inspired by this fact, we propose to leverage several
low-level cues for the proposed task by mimicking the working mechanism of our vision system.







In this paper, we take into account two types of low-level cues, including saliency and edge,
which are also essential in our visual system when dealing with high-level vision tasks.

Salient object detection models provide the locations of
foreground objects as shown in fig:teaser(d) but weak knowledge on boundaries.

As a remedy, edge detection models contain rich information about the boundaries, which
can be used to improve the spatial coherence of our proxy ground truths.

Specifically, for saliency detection, we use the DSS salient object detector
<cit.> to generate saliency maps as done in <cit.>.

For edge detection, we select RCF <cit.> as our edge detector which we found works
better than the HED edge detector <cit.>.

Furthermore, inspired by <cit.>, we apply the MCG method <cit.>
to the edge maps to get high-quality regions.

Some visual results can be found in fig:teaser.

As can be observed, with the help of edge information, some undetected salient regions
can be segmented out.

We will show some quantitative results when different heuristic cues are used in sec:experiments.








Formally, let I denote an image with image labels 𝐲, a subset of a predefined set
ℒ={l_1, l_2, …, l_L}, where L = |ℒ| denotes the number of semantic labels.

We also use l_0 to denote the “background” category, so we have ℒ̂={l_0, ℒ}.

Further, let us denote S and R as the corresponding saliency and region maps of I, respectively.




Then for each region R_m ∈ R, we perform the following snapping operation with respect to S

    H_j = 1/|R_m|∑_j ∈ R_mS_j,

where H is the resulting heuristic map (the proxy ground truth annotation) used as supervision.

Some visual results can be found in fig:teaser(e).



































§ ONLINE NOISE FILTERING


Due to the diversity of the downloaded images,
it is very difficult to extract the desired information from them.


This section is dedicated to presenting a promising mechanism to solve these problems,
namely online noise filtering module (NFM).



 §.§ Observations




Regarding the web-crawled images in our task, like weakly supervised
semantic segmentation, one of the main issues is the noisy labels,
which are from various weak cues.

These low-level cues possibly contain parts of the semantic objects
and even false predictions (fig:illu_ignore),
severely influencing the learning process of CNNs and
leading to low-quality predictions.

Furthermore, web-crawled images may contain more than one semantic category.

For example, the images in the `bicycle' category often cover the `person' category
and hence both of them will be predicted to salient regions
with category `bicycle' (fig:illu_ignorea).

Previous methods <cit.> solved this problem
using attention models <cit.>.


However, the attention models themselves require the supervision of
image-level labels that is impossible to be applied to our task.




Regarding this challenge, a promising way to overcome this is to discard
the noisy regions in the heuristic maps
but meanwhile keep reliable ones unchanged.

To achieve such a goal, in this section,
we present an online noise filtering mechanism
to intelligently filter those noisy regions.








Overview.
The pipeline of our proposed approach can be found in fig:arch.


As the name implies, our proposed scheme is able to filter noisy labels online by introducing
a noise filtering module (NFM) and then uses the generated heuristic maps as the supervision of
the semantic segmentation module (SSM).































 §.§ Online Noise Filtering


Noise Filtering Module (NFM).
Given an image I and its region map R,
it is easy for us to predict the label of each region R_m.

Specifically, the first part of our NFM consists of two
convolutional layers for building higher-level
feature representations, both of which have 512 channels, kernel size 3, and stride 1.

Then, we introduce a channel-wise region pooling layer following
the formulation of eqn:snapping
to extract equal number of features from each region.



Suppose there were totally M regions in image I, the dimensions of the output of the
channel-wise region pooling layer would be (M, 512, 1, 1), the mini-batch is M.

Finally, two fully connected layers are added as the classifier, which are with
1,024 and |ℒ| neurons, respectively.

Notice that all the weighted layers in our NFM are followed by ReLU layers for non-linear
transformation apart from the last one.

Please note that designing more complex structure might be more powerful here
but this is beyond the scope of this paper as our goal is to show how to learn to discard those regions
with potentially noisy labels.

Learning to Filter Noisy Labels.


Given an image I with image-level label 𝐲 and its corresponding region map R
and heuristic map H, for any region R_m, if ∑_j ∈ R_mH_j > 0 then
we say that region R_m belongs to foreground (otherwise background).

During training, the ground truth label of each foreground region should be 𝐲.





For background regions, we give them a special label
l_s (l_s ∉ℒ̂), indicating that
these regions should be ignored during training as we only care about foreground in NFM.



Let us denote f^m(I) as the activations corresponding to R_m in the score layer of NFM.


Notice that we omit the network parameters here for notational convenience.

The predicted label of R_m can be obtained by
C_m = _l ∈ℒf^m_l(I).

If C_m ∉𝐲, then H_j, j ∈ R_m  = l_s.



The resulting H will be used as the annotation of SSM.

The bottom row in fig:illu_ignore shows some typical examples of H.

Compared to the middle row in fig:illu_ignore, we can observe that
our online noise filtering module indeed helps a lot when dealing with images
with interferential categories.




Semantic Segmentation Module (SSM).


The proxy annotations sent to SSM are gray-level continuous maps, each pixel of which indicates
the probability of being foreground.

Therefore, following <cit.>, we use the following cross-entropy loss
to optimize our SSM

    ℰ(θ) = ∑_n=1^N
    ( p̂_n^0 log p_n(l_0|I; θ) + p̂_n^c log p_n(𝐲|I; θ)
    ),

where N is the number of elements in I, θ is the network
parameters, p̂_n^c = H_n, denoting the probability of the nth
element being salient, p̂_n^0 = 1 - p̂_n^c, and p_n(𝐲|I; θ) the probability of
the nth element belonging to 𝐲, which is from the network prediction scores.

Training. During the training phase, we found that the NFM module always tends to over-fit
when it reaches convergence.

The classifier will predict the same label to almost all the foreground regions influenced by the noisy labels,
which weakens the role of our NFM.

To address this problem, we change the learning rate of the weighted layers in NFM by multiplying
a fixed factor 0.1 to convolutional layers and 0.01 to fully connected layers.

We empirically found that such an under-fitting state of NFM
makes our whole model work the best.





During the inference phase,
we discard NFM while only keep the original layers in Deeplab.

Therefore, we do not introduce any additional computation into the
testing phase of Deeplab model.



 §.§ Refinement







With the above CNN trained based on heuristic maps,
it is possible for us to iteratively train more powerful CNNs by
leveraging the supervision from the keywords as done in <cit.>.

Suppose in the r-th iteration of the learning process the network parameter
is denoted by θ^r.

We can use θ^r to generate the prediction scores g^r(I) of SSM.

Given an image I, let 𝐲 be its image-level labels.

The segmentation map T^r of I can be computed according to Alg. <ref>.

With T^r, we are able to optimize another CNN (the Deeplab model here),
which may give us better results.

Notice that after the first-round iteration, we do not use the NFM here any more
as T^r already provides us more reliable `ground truth.'

By carrying out the above procedures iteratively, we can gradually refine the
segmentation results.



§ EXPERIMENTS
 










Our autonomous web learning of the semantic segmentation task is similar
to previous weakly supervised semantic segmentation task but now
we can deal with label noise.
To better verify the effectiveness of our proposed method, in this section, we compare our proposed approach
with existing weakly supervised semantic segmentation methods and meanwhile analyze the
importance of each component in our approach by ablation experiments.

Differently from our new task which leverages only category-independent cues,
traditional weakly supervised semantic segmentation methods also considers the knowledge of precise image-level labels
(e.g., attention cues).

For fair comparisons, we separate our experiments into two groups, which respectively represent
weakly supervised semantic segmentation (including attention cues) and semantic segmentation
with only category-independent cues.




 §.§ Implementation Details


Datasets.
In our experiment, we only use one keyword for each search.

We denote the collected dataset with only web images as 𝒟(W),

thus each image in 𝒟(W) is supposed to have only one semantic category.

Furthermore, to compare our approach with existing weakly supervised methods,
we choose the same datasets as in <cit.>, which contain two parts.

The first dataset is similar to our 𝒟(W), containing images
with only single image-level labels, which are originally
from the ImageNet dataset <cit.>.

Here, we denote it as 𝒟(S).

The second part, denoted as 𝒟(C), has only images from the PASCAL VOC 2012
benchmark <cit.> plus its augmented set <cit.>.

More details can be found in tab:datasets.

We evaluate our approach on the PASCAL VOC 2012  benchmark and report the results
on both the 'val' and 'test' sets.


Low-Level Cues. Other than saliency cues and region cues, it is reasonable to
harness attention cues <cit.> for weakly supervised semantic segmentation.

Let us denote A as the attention cues of image I.

We first perform a pixel-wise maximum operation between A and S (saliency cues)
as in <cit.>, which
aims to preserve as many heuristic cues as possible.

Then, we use Eqn. <ref> to perform a snapping operation
with respect to the combined map, yielding
the heuristic cues for training our first-round CNN.









Model Settings and Hyper-Parameters.
We use the publicly available Caffe toolbox <cit.> as our implementation tool.

Like most previous weakly supervised semantic segmentation works,
we use VGGNet <cit.> as our pre-trained model.

The hyper-parameters we used in this paper are as follows: initial learning rate (1e-3),
divided by a factor of 10 after 10 epochs, weight decay (5e-4), momentum (0.9), and mini-batch
size (16 for the first-round CNN and 10 for the second-round CNN).

We train each model for 15 epochs.

We also use the conditional random field (CRF) model proposed in <cit.>
as a post-processing tool to enhance the spacial coherence, in which the graphical model
is built upon regions.

This is because we already have high-quality region cues which are derived from edge knowledge.

Instead of both color and texture histograms as in <cit.>,
we only use the color histograms for measuring the similarities between adjacent regions
as our regions are derived from edge maps which already contain semantic information.

All the other settings are the same to <cit.>.
















































































































"@width 0.6pt







 §.§ Sensitivity Analysis


To analyze the importance of each component of our approach, we perform a number of
ablation experiments in this subsection.


Low-Level Cues.
The proposed learning paradigm starts with learning simple images and then
transitions to general scenes following <cit.>.

In this paragraph, we analyze the roles of different low-level cues
and their combinations in the first-round iteration.

For fair comparisons, we discard the NFM here whereas only keep the SSM.

table:low_level shows the quantitative results of
using different low-level cues.

With only saliency cues,
our first-round CNN achieves a baseline result with a mean IoU score of 52.59%.

With the edge information incorporated, the baseline result can be improved
by 1.8% in terms of mean IoU.

When all three kinds of low-level cues are considered, we achieve the best results,
which is 56.08%, around 3.5% improvement compared to the baseline result.

Similar phenomenon can also be found when only utilizing web images in Table <ref>.

These results verify the effectiveness of the low-level cues we leverage.

By comparing the results on the two tasks in Table <ref>,
we can observe that under the same experiment setting the web-crawled images
are more complex than the images from the ImageNet.

This is reasonable because unlike the ImageNet images the web-crawled images
are not with accurate labels (See fig:web_images).



The Role of NFM.
Table <ref> lists the results when the NFM is used or not during the phase of training the first-round CNN.

One can observe that with small regions provided, using NFM gives a 0.92% improvement
compared to not using it and a 0.38% improvement compared to using large regions.

Actually, the most important difference between whether using NFM lies in the ability of predicting images
with multiple categories.

As shown in fig:visualComp (columns c and d), our NFM is able to erase those noisy
regions caused by the wrong predictions of saliency model and hence produces cleaner results.

This phenomenon is especially obvious when training on web images.

As shown in Table <ref>, there is an improvement of 1.8% on the 'val' set
during the first-round iteration compared to the baseline result without NFM.

Compared to the growth range (+0.92%) with weak supervision, we can observe the noise-erasing
ability of our NFM is essential, especially when handling noisy web images.

From a visual standpoint, fig:illu_ignore provides more intuitive and direct results.

Comparing the bottom two lines of fig:illu_ignore, we can observe that
our method after NFM successfully filters most noisy regions (undesired categories and
stuffs in the background) in the heuristic cues.

This makes our segmentation network learn cleaner knowledge from the heuristic cues.







Region Size in NFM.
The region sizes play an important role in our NFM.

Here, we compare the results when using different region sizes in our NFM.

The regions are directly derived from the edge maps <cit.>.

By building the Ultrametric Contour Maps (UCMs) <cit.> with different thresholds,
we are allowed to obtain regions with different scales.

Here, we show the results when setting threshold to 0, 0.25, and 0.75, which correspond to small, medium, and large
regions, respectively.

The quantitative results can be found in Table <ref>.

As the number of regions increases, our approach achieves higher and higher mean IoU scores
on the PASCAL VOC 2012 validation set.

Training with small regions enables 0.54 percent improvement compared to using large regions.

The reason for this might be that lower thresholds provide more accurate regions, making less regions
stretch over more than one category.







The Role of CRF models.
Conditional random filed models have been widely used in semantic segmentation tasks.

In Table <ref>, we show the results when the CRF is used or not.

Obviously, the CRF model does help in all cases.



The Role of Different Training Sets.
We try to use different training sets to train the second-round CNN
while keep all the setting of the first-round one unchanged.

As can be seen in Table <ref>, combining both 𝒟(S) and 𝒟(C)
allows us to obtain the best results, which is 1.66 points higher than only using 𝒟(S) for
training.

For pure web supervision, incorporating the images of 𝒟(C) is also helpful, which leads to
a 2.61-point extra performance gain.

This indicates that more high-quality training data does help and complex images provide more information
between different categories that would be useful for both weakly supervised semantic segmentation and semantic
segmentation with only web supervision.












 §.§ Comparisons with Existing Works


In this subsection, we compare our approach with the existing state-of-the-art weakly supervised semantic segmentation works.

All the methods compared here are based on the Deeplab-Large-FOV baseline model <cit.>
except special declarations.

Table <ref> lists the results of prior methods and ours on both `val' and `test' sets.

As can be found, with weak supervision, our approach achieves a mean IoU score of 60.5% on the `val'
set without using CRF.

This value is already far better than the results of all prior works no matter whether the CRFs are used for them.

By using CRFs as post-processing tools, our method realizes the best results on the `val' and `test' sets,
which are both higher than 63.0%.



When considering training on web images with only category-agnostic cues, our results on
both the `val' and `test' sets are also better than most previous works and comparable
to the state-of-the-arts.


This reflects that despite extremely noisy web images, our approach is robust as well.

This is mostly due to the effectiveness of our proposed architecture, the ability of
eliminating the effect of noisy heuristic cues.








 §.§ Discussion


In Table <ref>, we show the specific number of each category
on both the `val' and `test' sets .

As can be observed, with the supervision of accurate image-level labels, our approach
wins the other existing methods on most categories (the purple line).

However, there are also some unsatisfactory results for a few categories
(`chair' and `table').

This small set of categories are normally with strange shapes and hence difficult to be
detected by current low-level cues extractors.

For instance, our method on the `table' category gets a IoU score of 24.1%,
which is nearly 8% worse than the AS-PSL method <cit.>.

This is mainly due to the fact that AS-PSL mostly relies on the attention cues,
which performs better on detecting the location of semantic objects.

Regarding the results with web supervision, in spite of only several
low-level cues and extremely noisy web images, our approach can get comparable
results to the best weakly supervised methods.

More interestingly, our approach behaves the best on two categories on the `test' set
(`bicycle' and `train').

This implies that leveraging reliable low-level cues provides a promising way
to deal with learning semantic segmentation automatically from the Internet.

Failure Case Analysis.
Besides the successful samples, there are also many failure cases
that have been exhibited on the bottom part of Fig. <ref>.

One of the main reasons leading to this should be the wrong predictions of the heuristic cues.

Although our online noise filtering mechanism, to some extent, is able to eliminate
the interference brought by noises, it still fails when processing some very complicated scenes.

For example, in Fig. <ref>a, it is very hard to distinguish clearly
which regions belong to `bicycle' and which regions belong to `person.'

On the other hand, the simple images cannot incorporate all the scenes in the `test' set
and therefore results in many semantic regions undetected.








§ CONCLUSION AND DISCUSSION


In this paper, we propose an interesting but challenging computer vision problem,
namely WebSeg, which aims at learning semantic segmentation from the free web images
following the learning manner of humans.

Regarding the extremely noisy web images and their imperfect proxy ground-truth annotations,
we design a novel online noise filtering mechanism, a new learning paradigm, to let CNNs
know how to discard undesired noisy regions.

Experiments show that our learning paradigm with only training on web images has already
obtained comparable results compared to previous state-of-the-art methods.

When leveraging more weak cues as in weakly supervised semantic segmentation,
we further improve the results by a large margin.

Moreover, we also perform a series of ablation experiments to show how each component
in our approach works.

Despite this, there is still a large room for improving the results of our task, which
is based on purely web supervision.

According to Table <ref>, the mIoU scores of a few categories are still low.

Thereby, how to download good web images, improve the quality of heuristic cues, and design
useful noise-filtering mechanisms are interesting future directions.

This new topic, as mentioned above, covers a series of interesting but difficult techniques,
offering many valuable research directions which need to be further delved into.

In summary, automatically learning knowledge from the unrestricted Internet
resources substantially reduce the intervention of humans.

We hope such a interesting vision task could drive the rapid development of its relevant research topics as well.




splncs


