Should you reload the optimizer for transfer learning?
When doing transfer learning it makes sense to have different update policies for "inherited" parameters and the "new" parameters. "Inherited" parameters are pre-trained on dataset1 and they typically form the front end of the deep model. The "new" parameters are trained from scratch and they typically produce the desired predictions on dataset2. It would be sensible to restart the learning schedule for the "new" parameters. However, most often we would avoid doing that for "inherited" parameters in order to avoid catastrophic forgetting.