Deep Learning NLP: "Efficient" BERT-like Implementations?
I work in a legacy corporate setting where I only have 16 core 64GB VM to work with on an NLP project.  I have a multi-label NLP text classification problem where I would really like to utilize a deep representation learning model like BERT, RoBERTa, ALBERT, etc.  

I have approximately 200,000 documents that need to be labeled and I have annotated set of about 2,000 to use as the ground truth for training/testing/fine tuning.  I also have a much larger volume of domain related documents to use for pre-training.  I will need to do the pre-training from scratch most likely, since this in a clinical domain.  I am also open to pre-trained models if they might have a chance working with just fine-tuning like Hugging Face, etc.. 

What models and their implementations that are PyTorch or Keras compatible would folks suggest as a starting point?  Or is this a computational non-starter with my existing compute resources?

If you want to use your current setup, it will have no problem running a transformer model. You can reduce memory use by reducing the batch size, but at the cost of slower runs.

Alternatively, test your algorithm on google Colab which is free. Then open a GCP account, google will provide $300 dollars of free credits. Use this to create a GPU cloud instance and then run your algorithm there. 

You probably want to use Albert or Distilbert from HuggingFace Transformers. Albert and Distilbert are both compute and memory optimized. HuggingFace has lot's of excellent examples. 

Rule of thumb you want to avoid Language Model training from scratch. If possible fine tune the language model or better yet skip it and go straight to the training the classifier. Also, HuggingFace and others have MedicalBert, ScienceBert, and other specialized pretrained models.  
