DPO Trainer for DialoGPT (Train/loss does not decrease / result is not in downward trend)
Hi, our group is currently working on a pest management chatbot. And we are currently testing which model can cater both English and Tagalog.
Right now, we're having problems with our DPO Trainer since the graph is not showing ideal results, specially the train/loss results. I have tweaked our code and parameters for a while now, and the results of the graph for the loss is either linear, increasing, or spiky.
Here is our parameters:
@dataclass
class ScriptArguments:
beta: Optional[float] = field(default=0.1, metadata={"help": "The beta parameter for DPO loss."})
model_name_or_path: Optional[str] = field(default=MULTI_LANG_MODEL, metadata={"help": "Path to the model."})
learning_rate: Optional[float] = field(default=2e-5, metadata={"help": "Optimizer learning rate."})
per_device_train_batch_size: Optional[int] = field(default=16, metadata={"help": "Training batch size per device."})
gradient_accumulation_steps: Optional[int] = field(default=4, metadata={"help": "Gradient accumulation steps."})
warmup_steps: Optional[int] = field(default=1000, metadata={"help": "Learning rate warmup steps."})
max_grad_norm: Optional[float] = field(default=0.1, metadata={"help": "Gradient clipping max norm."})
lora_alpha: Optional[float] = field(default=16, metadata={"help": "Lora alpha for LoRA fine-tuning."})
lora_dropout: Optional[float] = field(default=0.05, metadata={"help": "Lora dropout rate."})
lora_r: Optional[int] = field(default=32, metadata={"help": "Lora R parameter."})
max_prompt_length: Optional[int] = field(default=64, metadata={"help": "Max prompt length for inputs."})
max_length: Optional[int] = field(default=512, metadata={"help": "Max length for model outputs."})
num_train_epochs: Optional[int] = field(default=1, metadata={"help": "Number of training epochs."})
output_dir: Optional[str] = field(default="./dpo_results", metadata={"help": "Directory to save model outputs."})
Datasets = 40k
Can someone please help me with our training? Thank you in advance!