Resource and useful tips on Transfer Learning in NLP I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data and 60000 unlabeled data available to me. My data type is related to instructions (like- open the door[label-1], give me a cup of water[label-1], give me money[label-0] etc.) In this case, I heard that Transferring knowledge from other models will help me a lot. Can anyone give me some useful resource for transfer learning in NLP domain?

I already did a few experiments. I used GLoVE as a pretrained embeddings. Then test it with my label data. But got around 70% accuracy. Also tried with embedding built using my own data (63k) and then train the model. Got 75% accuracy on the test data. My model architecture is given below-


Q1: I have a quick question will it be referred to as Transfer learning if I use GLOVE embeddings in model?

Any kind of help is welcomed. Even someone has other ideas for building a model without using transfer learning is welcomed.
If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).

Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.

As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.

Hope that helps, good luck!