What are the good parameter ranges for BERT hyperparameters while finetuning it on a very small dataset? I need to finetune BERT model (from the huggingface repository) on a sentence classification task. However, my dataset is really small.I have 12K sentences and only 10% of them are from positive classes. Does anyone here have any experience on finetuning bert in small datasets? Do you have any suggestions for learning rate, batch size, epoch number warmup steps etc.?
How many classes do you have? Bert can handle a high-quality 12k dataset for binary classification. I recommend duplicating your positive test case 4x and sampling a 5k test cases from your negative class. This will give you a balanced dataset. Then implement BERT in google colab using the original GitHub repository from the google Bert team. You can find a functional collab here: https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb
On collab, TPUs takes about 10 to 15 minutes to run. All you have to do is use write your data processor.
The following parameters are a good place to start.

TRAIN_BATCH_SIZE = 32
EVAL_BATCH_SIZE = 8
PREDICT_BATCH_SIZE = 8
LEARNING_RATE = 2e-5
NUM_TRAIN_EPOCHS = 3.0
MAX_SEQ_LENGTH = 128
# Warmup is a period of time where hte learning rate 
# is small and gradually increases--usually helps training.
WARMUP_PROPORTION = 0.1
# Model configs
SAVE_CHECKPOINTS_STEPS = 1000
SAVE_SUMMARY_STEPS = 500