Difficulty in choosing Hyperparameters for my CNN
I am going to comment on your ideas:


This is generally recommended. There is no proven reason not to do it but many reasons to do it, and you should initialize as much weights as possible. Typically it is done until the last layer.
Initialize everything that you can from pre-trained network.
I would freeze the first layers in the first epochs, and after that unfreeze them. If possible, I would rather use differential learning rates, as this post shows.
Totally, as shown above.
Adam is more widely used, I would use that.
Cosine scheduling with reestarts has been shown to be very effective, but any learning rate with reestarts should be similar.
It does make sense.
I wouldn't worry about that until you are overfitting, and you are not overfitting yet. If you overfit, I would recommend dropout instead of weight decay.