Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?
A1. BERT by itself may not be useful for scientific terms. You have 2 choices, either find a pre-trained embedding specific to the scientific text database or use transfer learning and build upon BERT.
A2. Fastext or Glove will have the same issue as BERT
A3. You have mentioned that BERT works well at a high level. So I would not advice you to create embeddings from scratch and instead use transfer learning to enhance the embeddings