What GPU size do I need to fine tune BERT base cased?
Speaking about vanilla BERT.
It is currently not possible to fine-tune BERT-Large using a GPU with 12GB - 16GB of RAM, because the maximum batch size that can fit in memory is too small (even with batch size = 1).
The fine-tuning examples which use BERT-Base should be able to run on a GPU that has at least 12GB of RAM using the hyperparameters given on this page.
However, GPU training is single-GPU only.