
[UPDATE] Repeated Augmentation
Repeated Augmentation [15], [16] is another technique which can improve the overall accuracy and has been used by other strong recipes such as those at [6], [7]. Tal Ben-Nun, a community contributor, has further improved upon our original recipe by proposing training the model with 4 repetitions. His contribution came after the release of this article.
Below you can see the optimal value used on our recipe:
ra_sampler=True,
ra_reps=4,

The above is the final optimization which improved our accuracy by 0.184 points. 
Optimizations that were tested but not adopted