
Figure 4. Train loss and test error along the line connecting the SWA solution (circle) and SGD solution (square). SWA solution is centered in a wide region of low train loss while the SGD solution lies near the boundary. Because of the shift between train loss and test error surfaces, SWA solution leads to much better generalization.
Examples and Results
We released a GitHub repo here with examples of using the torchcontrib implementation of SWA for training DNNs. For example, these examples can be used to achieve the following results on CIFAR-100:



DNN (Budget)
SGD
SWA 1 Budget
SWA 1.25 Budgets
SWA 1.5 Budgets




VGG16 (200)
72.55 ± 0.10
73.91 ± 0.12
74.17 ± 0.15
74.27 ± 0.25


PreResNet110 (150)
76.77 ± 0.38
78.75 ± 0.16
78.91 ± 0.29
79.10 ± 0.21


