Milking CowMask for Semi-Supervised Image Classification
Geoff French Avital Oliver Tim Salimans
cs.CV


FlexRiLoG — A SageMath Package for Motions of Graphs
    Georg GraseggerJohann Radon Institute for Computational and Applied Mathematics (RICAM), Austrian Academy of Sciences
Jan LegerskýJohannes Kepler University Linz, Research Institute for Symbolic Computation (RISC)
Department of Applied Mathematics, Faculty of Information Technology, Czech Technical University in Prague
    
====================================================================================================================================================================================================================================================================================================================================




Consistency regularization is a technique for semi-supervised learning that underlies a number of strong results for classification with few labeled data. It works by encouraging a learned model to be robust to perturbations on unlabeled data. Here, we present a novel mask-based augmentation method called CowMask. Using it to provide perturbations for semi-supervised consistency regularization, we achieve a state-of-the-art result on ImageNet with 10% labeled data, with a top-5 error of 8.76% and top-1 error of 26.06%. Moreover, we do so with a method that is much simpler than many alternatives. We further investigate the behavior of CowMask for semi-supervised learning by running many smaller scale experiments on the SVHN, CIFAR-10 and CIFAR-100 data sets, where we achieve results competitive with the state of the art, indicating that CowMask is widely applicable. We open source our code at
<https://github.com/google-research/google-research/tree/master/milking_cowmask>.










§ INTRODUCTION



Training accurate deep neural network based image classifiers requires large quantities of
training data. While images are often readily available in many problem domains, producing
ground truth annotations is usually a laborious and expensive task that can act as a
bottleneck. Semi-supervised learning offers the tantalising possibility of reducing the
amount of annotated data required by learning from a dataset that is only partially
annotated.

Semi-supervised learning algorithms based on consistency
regularization <cit.> have proved to be simple while effective,
yielding a number of state of the art results over the last few years.
Consistency regularization is driven by encouraging consistent predictions for unsupervised
samples under stochastic augmentation.
Using CutOut <cit.> – in which a rectangular region of an image
is masked to zero – as the augmentation
has proved to be highly effective, making significant
contributions to the effectiveness of rich augmentation strategies <cit.>.

In this paper, we introduce a simple masking strategy that we call CowMask, whose shapes and appearance
are more varied than the rectangular masks used by CutOut and RandErase <cit.>.
When used to erase parts of an image in a similar fashion to RandErase, CowMask
outperforms rectangular masks in the majority of semi-supervised image classifications tasks that we tested.







We extend the Interpolation Consistency Training (ICT)
algorithm <cit.> to use mask-based mixing, using both rectangular masks as in CutMix <cit.> and CowMask. Both CutMix and CowMask exhibit strong semi-supervised learning
performance, with CowMask outperforming rectangular mask based mixing in the majority
of cases. CowMask based mixing achieves a state-of-the-art result[The concurrent SimCLR <cit.> work achieves a better result using a larger architecture] on semi-supervised Imagenet, and results comparable to state-of-the-art on multiple small image datasets, all without the need for previously proposed complex training procedures.



In Section <ref> we discuss related work that forms the basis of our approach, alongside other semi-supervised learning algorithms for comparison. In Section <ref> we present CowMask, the novel ingredient to our semi-supervised learning algorithm, that is described in Section <ref>. We present our experiments and results in Section <ref>. Finally we discuss our work and conclude in Section <ref>.








§ BACKGROUND







 §.§ Semi-supervised classification


A variety of semi-supervised deep neural network image classification approaches have been
proposed over the last several years, including the use of
auto-encoders <cit.>,
GANs <cit.>,
curriculum learning <cit.> and
self-supervised learning <cit.>.

Many recent approaches are based on 
consistency regularization <cit.>, a simple approach exemplified by
the π-model <cit.> and the Mean Teacher model <cit.>.
Two loss terms are minimized; standard
cross-entropy loss and consistency loss for supervised and unsupervised samples
respectively. Consistency loss measures the difference between predictions resulting
from differently perturbed variants of an unsupervised sample.
The π-model perturbs samples twice using stochastic augmentation and
minimises the squared difference between class probability predictions. The Mean
Teacher model builds on the π-model by using two networks; a teacher and a student.
The student is trained using gradient descent as normal while the weights of the teacher
are an exponential moving average of those of the student. The consistency loss term
measures the difference in predictions between the student and the teacher
under different stochastic augmentation.

A variety of types of perturbation have been explored. Sajjadi  <cit.> employed
richer data augmentation including affine transformations,
while <cit.> and <cit.> used standard augmentation strategies
such as random crop and noise for small image datasets.
Virtual Adversarial Training (VAT) uses adversarial perturbations that
maximise the consistency loss term.


































































































 §.§ Mixing regularization


Recent works have demonstrated that blending pairs of images and corresponding ground truths
can act as an effective regularizer. MixUp <cit.> draws a blending
factor from the Beta distribution that is used to interpolate images and ground truth labels.
Interpolation Consistency Training (ICT) <cit.> extends this approach to work in a
semi-supervised setting by combining it with the Mean Teacher model. The teacher network
is used to predict class probabilities for a pair of images A and B and MixUp is used to
blend the images and the teachers' predictions. The predictions of the student for the blended
image are encouraged to be as close as possible to the blended teacher predictions.

MixMatch <cit.> guesses labels for unsupervised samples by sharpening the
averaged predictions from multiple rounds of standard augmentation and blends images and
corresponding labels (ground truth for supervised samples, guesses for unsupervised) using 
MixUp <cit.>. The blended images and corresponding guessed labels are used to
compute consistency loss.




 §.§ Rich augmentation

AutoAugment <cit.> and RandAugment <cit.> are rich augmentation schemes
that combine a number of image operations provided by the Pillow library <cit.>. AutoAugment learns
an augmentation policy for a specific dataset using re-inforcement learning, requiring a large amount
of computation to do so. RandAugment on the other hand has two hyper-parameters that are chosen via grid
search; the number of operations to apply and a magnitude.

Unsupervised data augmentation (UDA) <cit.> adds employs a combination of CutOut <cit.>
and RandAugment <cit.> in a semi-supervised setting achieving state-of-the-art
results in small image benchmarks such as CIFAR-10. Their approach encourages consistency between
the predictions for the original un-modified image and the same image with RandAugment applied.

ReMixMatch <cit.> builds on MixMatch by adding distribution
alignment and rich data augmentation using CTAugment or RandAugment (depending on the dataset). CTAugment is a variant of AutoAugment
that learns an augmentation policy during training, and RandAugment is a pre-defined set of 15 forms of augmentations with concrete scales. It is worth noting that
ReMixMatch uses predictions from standard 'weak' augmentation as guessed target probabilities
for unsupervised samples and encourages predictions arising from multiple applications
of the richer CTAugment to be close to the guessed target probabilities. The authors found that using
rich augmentation for guessing target probabilities (a la MixMatch) resulted in unstable training.

FixMatch <cit.> is a simple semi-supervised learning approach that uses standard
'weak' augmentation to predict pseudo-labels for unsupervised samples. The same samples are richly
augmented using CTAugment and cross-entropy loss is computed using the pseudo-labels.
Confidence thresholding <cit.> masks the unsupervised
cross-entropy loss to zero for samples whose predicted confidence is below 95%.




 §.§ Mask-based regularization


Erasing a rectangular region of an image by replacing it with zeros – as in Cutout <cit.> –
or noise – as in RandErase <cit.> – has proved to be an effective augmentation strategy
that yields improvements in supervised image classification.

CutOut has proved to be highly effective in semi-supervised classification scenarios.
The UDA authors <cit.> report impressive results, while the FixMatch authors
<cit.> report that CutOut alone is as effective as the combination of
the other 14 image operations used in CTAugment.

CutMix <cit.> replaces the blending factor in MixUp with a rectangular mask and uses it to mix
pairs of images, effectively cutting and pasting a rectangle from one image onto another. This
yielded significant supervised classification performance gains. CutMix was applied by French as part of a consistency regularization based semi-supervised semantic segmentation algorithm <cit.>.







.8


    

-2mm
.13

    < g r a p h i c s >
 
    < g r a p h i c s >
 
    < g r a p h i c s >

 






§ COWMASK



Our exploration of mask-based augmentation for consistency regularization is motivated by the strong performance
of Cutout <cit.> shown in ablation studies in UDA <cit.> and FixMatch <cit.>.
Furthermore, French  show that semantic segmentation problems exhibit a challenging data distribution where
the cluster assumption – identified in prior work <cit.> as important to the success of consistency regularization – does not apply. In spite of this, they
obtain strong results using CutMix, suggesting mask-based mixing as a promising avenue.

Here, we propose CowMask; a simple approach to generating the masks shown on the right of Algorithm  <ref>, so called due to its' Friesian cow-like appearance.
We note that the concurrent work FMix <cit.> uses an inverse Fourier transform to generate masks with a similar visual appearance.

Briefly, a CowMask is generated by applying Gaussian filtering of scale σ to normally distributed noise.
A threshold τ is chosen such that a proportion p of the smooth noise pixels are below τ.
Pixels with a value below τ are assigned a value of 1, or 0 otherwise.
The scale of the mask features is controlled by σ – as seen in the examples on the right of Algorithm <ref> – and is drawn from a log-uniform distribution in
the range (σ_min,σ_max). The proportion p of pixels with a value of 1 is drawn from
a uniform distribution in the range (p_min,p_max). The procedure for generating a CowMask is provided in Algorithm <ref>.











§ SEMI-SUPERVISED LEARNING METHOD


We adopt the Mean Teacher<cit.> framework as the basis of our approach.
We use two networks; the student f_θ(·) and the teacher g_ϕ(·),
both of which generate class probability vectors.



The student is trained by gradient descent as normal.
After every update to the student, the weights of the teacher are updated to be an exponential moving average of those of the student using ϕ' = ϕα + θ(1-α).
The EMA momentum α controls the trade-off between the stability and the speed at which the teacher follows the student.


Our training set consists of a set of supervised samples S consisting of input images s and corresponding target labels t, and a set of unsupervised samples U consisting only of input images u.
Given a labelled dataset we select the supervised subset randomly such that it maintains the class balance of the overall dataset[We use  from Scikit-Learn <cit.>] as is standard practice in the literature.
All available samples are used as unsupervised samples. Our models f_θ are then trained to minimize a combined loss:

    L = L_𝒮(f_θ(s),t) + ω L_𝒰(f_θ(u), g_ϕ(u))

where we use standard cross entropy loss for the supervised loss L_𝒮(·) and consistency loss for the unsupervised loss L_𝒰(·) that is modulated by the unsupervised loss weight ω.

We explore two different types of mask-based consistency regularization: mask-based erasure and mask-based mixing. In mask-based erasure we perturb our input data by erasing the part of the input image corresponding to a randomly sampled mask. In mask-based mixing we blend two input images together, with the blending weights given by the sampled mask.
We follow the nomenclature of Cutout and CutMix, using the terms CowOut and CowMix to refer to CowMask
based erasure and mixing respectively.





0.96cm
.45

    

-2mm
.48


 





 §.§ Mask-based augmentation by erasure


Mask-based erasure can function as an augmentation that can be added to the standard augmentation
scheme used for the dataset at hand, with one caveat.
Similar to prior work <cit.> we found it necessary to split
our augmentation into a 'weak' standard augmentation scheme (e.g. crop and flip) and a 'strong' rich
scheme; RandAugment in the case of the prior works mentioned or CowOut in our work.
Weakly augmented samples are passed to the teacher network, generating predictions that are used as pseudo-targets
that the student is encouraged to match for strongly augmented variants of the same samples.
Using 'strong' erasure augmentation to generate pseudo-targets resulted in unstable training.






The π-model <cit.> and the Mean Teacher model <cit.> both use a Gaussian ramp-up function to modulate the effect of consistency loss during the early stages of training. Reinforcing the random predictions of an untrained network was found to harm performance. In place of a ramp-up we opt to use confidence thresholding <cit.>.
Consistency loss is masked to zero for samples for which the teacher networks' predictions are below a specified threshold. FixMatch <cit.> uses confidence thresholding for similar reasons.



Our procedure for computing unsupervised consistency loss based on erasure is provided in Algorithm <ref>. For our small image experiments we found that the best value for the unsupervised weight factor ω is 1.

























 §.§ Mask-based mixing

Alternatively, we can construct an unsupervised consistency loss by mask-based mixing of images in place of erasure.
Our approach for mixing image pairs using masks is essentially that of interpolation consistency training (ICT) <cit.>.
ICT works by passing the original image pair to the teacher network, the blended image to the student, and encouraging the student networks' prediction to match the blended teacher predictions.
Where ICT draws per-pair blending factors a beta distribution, we mix images using a mask, and mix probability predictions with the mean of that mask (the proportion of pixels with a value of 1).


Confidence thresholding required adaptation for use with mix-based regularization.
Rather than applying confidence thresholding to the blended teacher probability
predictions we opted to blend the confidence values before thresholding as this gave
slightly better results. Further improvements resulted from modulating the consistency
loss by the proportion of samples in the batch whose predictions cross the confidence threshold,
rather masking the loss for each sample individually.




The procedure for computing unsupervised mix consistency loss is provided in Algorithm <ref>.
We found that a higher weight ω was appropriate for mix consistency loss; we used
a value of 30 for our small image experiments.






































§ EXPERIMENTS AND RESULTS


We first evaluate CowMix for semi-supervised consistency regularization on the challenging ImageNet dataset, where we match the state of the art. Next, we examine CowOut and CowMix further and compare with previously proposed methods by trying multiple versions of our approach combined with multiple models on three small image datasets: CIFAR-10, CIFAR-100 and SVHN. The training regimes used for both ImageNet and the small image datasets are sufficiently similar that we used the same codebase for all of our experiments.

Our results are obtained by using the teacher network for evaluation. We report our results as error rates presented
as the mean ± 1 standard deviation computed from the results of 5 runs, each of which uses a different subset of samples as the supervised set. Supervised sets are consistent for all experiments for a given dataset and number of supervised samples.





 §.§ ImageNet 2012


We contrast the following scenarios: a supervised baseline using 10% of the dataset, semi-supervised training with the same 10% of labelled examples using CowMix consistency regularization on all unlabeled examples, and fully supervised training with all 100% labels. 




  §.§.§ Setup


We used the ResNet-152 architecture. We adopted a training regime as similar as possible to a standard ImageNet ResNet training protocol.
We used a batch size of 1024 and SGD with Nesterov Momentum <cit.> set to 0.9 and weight decay (via L2 regularization) set to 0.00025. Our standard augmentation scheme consists of inception crop, random horizontal flip and colour jitter, as in <cit.>.We found that the standard learning rate of 0.1 resulted in unstable training, but were able to stabilise it by reducing the learning rate to 0.04 <cit.>. We found that our approach benefits from training for longer than in supervised settings, so we doubled the number of training epochs to 180 and stretched the learning rate schedule by a factor of 2, reducing the learning rate at epochs 60, 120 and 160 and reduced it by a factor of 0.2 rather than 0.1. We used a teacher EMA momentum α of 0.999.

We obtained our CowMix results using a mix loss weight of 100 and and a confidence threshold of 0.5.
We drew the CowMask σ scale parameter from the range (32, 128).





  §.§.§ Results


Our ImageNet results are presented in Table <ref>.
We match the S^4L MOAM <cit.> top-5 error result and beat their top-1 error result, with a
simple end-to-end approach and a significantly smaller model.
By comparison the S^4L MOAM result is obtained using a 3-stage training and fine-tuning procedure.
The recent SimCLR <cit.> approach (concurrent work) uses self-supervised contrastive learning followed
by a fine tuning stage.
They beat our result when using a much larger model.
We tested our approach with wider models (e.g. ResNet-50×2) but obtained better results from the deeper
and commonly used ResNet-152.








 §.§ Small image experiments

Alongside CowOut and CowMix we implemented and evaluated Mean Teacher, CutOut/RandErase and CutMix, and we compare our method against these using the CIFAR-10, CIFAR-100, and SVHN datasets.

We note the following differences between our implementation and those of CutOut and CutMix: 1. Our boxes are chosen so that they entirely fit within the bounds of the mask, whereas CutOut and CutMix use a fixed or random size respectively and centre the box anywhere within the mask, with some of the box potentially being outside the bounds of the mask. 2. CutOut uses a fixed size box, CutMix randomly chooses an area but constrains the aspect ratio to be that of the mask, we choose both randomly.






  §.§.§ Setup


For the small image experiments we use a 27M parameter Wide ResNet 28-96x2d with shake-shake regularization <cit.>. We note that as a result of a mistake in our implementation we used a 3×3 convolution rather than a 1×1 in the residual shortcut connections that either down-sample or change filter counts, resulting in a slightly higher parameter count.

The standard Wide ResNet training regime <cit.> is very similar to that used for ImageNet.
We used the optimizer, but with weight decay of 0.0005 and a batch size of 256.
As before, the standard learning rate of 0.1 had to be reduced to ensure stability, this time to 0.05.
The small image experiments also benefit from training for longer; 300 epochs instead of the standard 200 used in supervised settings. The adaptations made to the Wide ResNet learning rate schedule were nearly identical to those
made to the ImageNet schedule. We doubled its length and reduced the learning rate by a factor
of 0.2 rather than 0.1. We did however remove the last step; the learning rate is reduced at epochs 120 and
240 rather than epochs 60, 120 and 160 as used in supervised settings.
For erasure experiments we used a teacher EMA momentum α of 0.99 and for mixing experiments we used 0.97.

When using CowOut and CowMix we obtained the best results when the CowMask scale parameter σ is drawn from
the range (4, 16). We note that this corresponds to a range of (18, 12) relative to the 32×32 image size and that the σ range used in our ImageNet experiments bears a nearly identical relationship to the 224×224 image size used there. For erasure experiments using CowOut we obtained the best results when drawing p; the proportion of pixels that are retained from the range (0.25, 1). Intuitively it makes sense to retain at least 25% of the image pixels as encouraging the network to predict the same result for an image and a blank space is unlikely to be useful. For mixing experiments using CowMix we obtained the best results when drawing p from the range (0.2, 0.8).

We performed hyper-parameter tuning on the CIFAR-10 dataset using 1,000 supervised samples and evaluating on 5,000 training samples held out as a validation set. The best hyper-parameters found were used as-is for CIFAR-100 and SVHN.


















  §.§.§ Results


Our results for CIFAR-10, CIFAR-100 and SVHN datasets are presented in Tables <ref>, <ref> and <ref> respectively. Considering the techniques we explore we find that mix-based regularization outperforms erasure based regularization, irrespective of the mask generation method used. 


We would like to note that our 27M parameter model is larger than the 1.5M parameter models used for the majority of results in other works, so we cannot make an apples-to-apples comparison in these cases. Our CIFAR-10 results are competitive with recent work, except in small data regimes of less than 500 samples where EnAET <cit.> and FixMatch <cit.> outperform CowMix. Our CIFAR-100 and SVHN results are competitive with recent approaches but are not state of the art. We note that we did not tune our hyper-parameters for these datasets.



§ DISCUSSION



We explain the effectiveness of CowMix by considering the effects of CowMask and mixing based semi-supervised
learning separately.

DeVries  <cit.> established that Cutout – that uses a box shaped mask similar to RandErase - encourages the network to utilise a wider variety of features in order to overcome the varying combinations of parts of an image being present or masked out. In comparison to a rectangular mask the more flexibly shaped CowMask provides more variety and has less correlation between regions of the mask. This increases in the range of combinations of image regions being left intact or erased enhances its effect.

The MixUp <cit.> and CutMix <cit.> regularizers demonstrated that encouraging network predictions vary smoothly between two images as they are mixed – using either interpolation or mask-based mixing – improved supervised performance, with mask-based mixing offerring the biggest gains. We adapted CutMix – in a similar fashion to ICT – for semi-supervised learning and showed that mask based mixing yields significant gains when used as an unsupervised regularizer. CowMix adds the benefits of flexibly shaped masks into the mix.




§ CONCLUSIONS


We presented and evaluated CowMask for use in semi-supervised consistency regularization, achieving a new state of the art on semi-supervised Imagenet, with a much simpler method than in previously proposed approaches, using standard networks and training procedures.  We examined both erasure-based and mixing-based augmentation using CowMask, and find that the mix-based variant – which we call CowMix – is particularly effective for semi-supervised learning. Further experiments on small image data sets SVHN, CIFAR-10, and CIFAR-100 demonstrate that CowMask is widely applicable.

Research on semi-supervised learning is moving fast, and many new approaches have been proposed over the last year alone that use mask-based perturbation. In future work we would like to further explore the use of CowMask in combination with these other recently proposed methods.












§ BROADER IMPACT


Manual annotation is a laborious and expensive task.
This can act as a bottleneck, slowing or preventing the adoption of machine learning systems.
This is particularly likely to affect organisations such as small businesses and non-commercial entities, for which
the financial cost of annotation could act as a hindrance.
Semi-supervised learing offers the potential of reducing this bottleneck, making machine learning accessible to those
with less access to resources.














plain





§ SUPPLEMENTARY MATERIAL – ILLUSTRATION OF COMPUTATION


Here we provide illustrations that give a simplified overview of our approaches.



 §.§ Mask based erasure


Our mask-based augmentation by erasure algorithm is illustrated in Figure <ref>




 

 §.§ Mask based mixing


Our mask-based augmentation by mixing algorithm is illustrated in Figure <ref>.





