How to structure a cnn for fine-tuning?
I want to fine-tune a model so that I can experiment with various different hyper parameters. For example:

filters
Regularisation
Convolutional filter size
Learning rate
Optimizers

I have chose do do this in PyTorch and have created a base model (see below). However I am unsure of the best way to set up my code in order to do this. Specifically my ConvNet and train function. I will need to make comparisons as I go along using graphs. Could anyone offer any advice on the best way to structure my code/go about this.
class ConvNet(nn.Module):
  def __init__(self, num_classes=10):
    super().__init__()

    self.layer1 = nn.Sequential(
        nn.Conv2d(3, 16, kernel_size=3, stride=1),
        nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )

    self.layer2 = nn.Sequential(
        nn.Conv2d(16, 32, kernel_size=3, stride=1),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )

    self.fcl = nn.Sequential(
        nn.Flatten(),
        nn.Linear(1152, 100)
    )

  def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.fcl(out)

    return out



If you're looking for a simple tutorial, PyTorch has one that is explained well for computer vision here: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html
This is section is most relevant to your question so you can see how fine-tuning is done when the optimizer is adjusted.
model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
# Here the size of each output sample is set to 2.
# Alternatively, it can be generalized to nn.Linear(num_ftrs, len(class_names)).
model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

You have to train your model completely first. Then you can refer to this post: https://discuss.pytorch.org/t/how-to-perform-finetuning-in-pytorch/419/8?u=nullpointer
ignored_params = list(map(id, model.fc.parameters()))
base_params = filter(lambda p: id(p) not in ignored_params,
                     model.parameters())

optimizer = torch.optim.SGD([
            {'params': base_params},
            {'params': model.fc.parameters(), 'lr': opt.lr}
        ], lr=opt.lr*0.1, momentum=0.9)

If you'd like to change the weights of a layer without changing the weights of the network while you're fine-tuning, you can do something like the following:
    model = models.vgg16(pretrained=True)
    print list(list(model.classifier.children())[1].parameters())
    mod = list(model.classifier.children())
    mod.pop()
    mod.append(torch.nn.Linear(4096, 2))
    new_classifier = torch.nn.Sequential(*mod)
    print list(list(new_classifier.children())[1].parameters())
    model.classifier = new_classifier

If you're looking to add layers or filters to the current model:
class MyModel(nn.Module):
    def __init__(self, pretrained_model):
        self.pretrained_model = pretrained_model
        self.last_layer = ... # create layer

    def forward(self, x):
        return self.last_layer(self.pretrained_model(x))

