I am trying to classify flowers with a pretrained network, but for some reason it does not train
I am currently trying to classify flowers from this dataset, using Pytorch.

First of all I started to transfrom my data for the training, validation and testing set.

data_dir = 'flowers'
train_dir = data_dir + '/train'
valid_dir = data_dir + '/valid'
test_dir = data_dir + '/test'

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406], 
                                                            [0.229, 0.224, 0.225])])

test_transforms = transforms.Compose([transforms.Resize(224),
                                      transforms.CenterCrop(224),
                                      transforms.ToTensor(),
                                      transforms.Normalize([0.485, 0.456, 0.406], 
                                                           [0.229, 0.224, 0.225])])


Afterwards I loaded the data with ImageFolder:

trainset = datasets.ImageFolder(train_dir, transform=train_transforms)
testset = datasets.ImageFolder(test_dir, transform=test_transforms)
validationset = datasets.ImageFolder(valid_dir, transform=test_transforms)


Then I defined my DataLoaders:

trainloader = torch.utils.data.DataLoader(trainset, batch_size = 64, shuffle = True)
testloader = torch.utils.data.DataLoader(testset, batch_size = 32)
validationloader = torch.utils.data.DataLoader(validationset, batch_size = 32)


I choose vgg to be my pretrained model:

model = models.vgg16(pretrained = True)


And defined a new classifier:

for param in model.parameters():
    param.requires_grad = False

classifier = nn.Sequential(OrderedDict([
    ('fc1', nn.Linear(25088, 4096)),
    ('relu', nn.ReLU()),
    ('fc2', nn.Linear(4096, 4096)),
    ('relu', nn.ReLU()),
    ('fc3', nn.Linear(4096, 102)),
    ('output', nn.Softmax(dim = 1))

]))

model.classifier = classifier 


This is the code to actually train my NN (on the GPU):

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.classifier.parameters(), lr = 0.005)

epochs = 9
print_every = 10
steps = 0

model.to('cuda')

for e in range(epochs):
    running_loss = 0

    for ii, (inputs, labels) in enumerate(trainloader):
        steps += 1



        inputs, labels = inputs.to('cuda'), labels.to('cuda')

        optimizer.zero_grad()

        # Forward and backward 
        outputs = model.forward(inputs)
        loss = criterion(outputs, labels)



        loss.backward()
        optimizer.step()

        running_loss += loss.item()



        if steps % print_every == 0:
            print("Epoch: {}/{}... ".format(e+1, epochs),
                  "Loss: {:.4f}".format(running_loss/print_every))

            running_loss = 0


But when I run my model, the loss is random and I am not sure why.

Thank you for any kind of help in advance and Greetings from Germany! 

Here are some tips- in the order of which I think they will help:


Try doing some hyper-parameter optimization. (i.e. try 10 learning rates over a domain like 1e-2 to 1e-6) More info on what that is: (http://cs231n.github.io/neural-networks-3/#hyper)
Code and print a accuracy metric (print it with your loss), because you may be surprised how high a pre-trained model accuracy will be.
Try switching to model = models.vgg16_bn(pretrained = True) and perhaps bigger networks like vgg 19 or resnet34


Can you include your accuracy and lost per epoch?

Let me know if any of those tips helped!

(Hello from the USA)
