Masked Language Modeling on Domain-specific Data My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even training a language model(like BERT) from scratch, but unfortunately, my data isn't that big to help the previous model learn new connections, let alone learn the embeddings from scratch.
Now what I have in mind is creating a transformer model with my own vocabulary which consists of words in my domain-specific data (after separating them with spaces and not using transformer tokenizers). This way the vocab size would be smaller and the positions and relations would be learned faster and more easily. Although I'm a bit confused about implementation.
Can I use this architecture (that is for NMT) and give plain text for both the input and output? or should I mask some tokens in the input and give the complete sentence as the label?
Any other suggestions?
First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. Datasets (benchmarks) they used to train and evaluate this model from scratch were WMT 2014 Engligh-to-German, WMT 2014 English-to-French (section 5.1 of the paper). The conclusion is that you cannot train transformers from scratch unless you have your sentence pairs in 2 languages.
MLM on the other hand is something that BERT used for training. So if you want to go in this direction, you can use the pre-trained BERT and fine-tune it with Masked language model head on top of your BERT model using your own dataset. You need to stick to the tokenizer that BERT was originally trained on but at least you are taking advantage of some general context that was learned during training BERT from scratch. If you want to train BERT either from scratch or fine-tune with MLM head you can follow this tutorial from hugging face