
layout: blog_detail
title: 'Stochastic Weight Averaging in PyTorch'
author: Pavel Izmailov and Andrew Gordon Wilson
redirect_from: /2019/04/29/road-to-1.0.html

In this blogpost we describe the recently proposed Stochastic Weight Averaging (SWA) technique [1, 2], and its new implementation in torchcontrib.  SWA is a simple procedure that improves generalization in deep learning over Stochastic Gradient Descent (SGD) at no additional cost, and can be used as a drop-in replacement for any other optimizer in PyTorch. SWA has a wide range of applications and features:

SWA has been shown to significantly improve generalization in computer vision tasks, including VGG, ResNets, Wide ResNets and DenseNets on ImageNet and CIFAR benchmarks [1, 2].
SWA provides state-of-the-art performance on key benchmarks in semi-supervised learning and domain adaptation [2].
