Neural Network Model using Transfer Learning not learning I am a beginner in Deep Learning and working on Road Crack detection using transfer learning. I am working on binary classification with two classes , crack and no crack.

My distribution of two classes is as follows:

Cracks - 600 images

No cracks - 480 images

I have used data augmentation also :

 train_generator = train_datagen.flow(trainX, trainY, batch_size=16)

 val_generator = test_datagen.flow(testX, testY, batch_size= 16)


I am using VGG16 and I have frozen the lower 4 layers like this :

vgg = vgg16.VGG16(include_top=False, weights='imagenet', 
                                 input_shape=input_shape)

output = vgg.layers[-1].output

output = keras.layers.Flatten()(output)

vgg_model = Model(vgg.input, output)

for layer in vgg_model.layers[:4]:

     layer.trainable = False


After that, I added two hidden layers :

model = Sequential()
model.add(vgg_model)
model.add(Dense(256, activation='relu', input_dim=input_shape))
model.add(Dense(256, activation='relu'))
model.add(Dense(2, activation='softmax')) 
model.compile(loss='binary_crossentropy',
          optimizer=Adam(lr = 1e-6),
          metrics=['accuracy'])


But after 1-2 epochs nothing seems to change, neither validation accuracy nor loss. I tried using SGD optimizer also but that also didn't help. I added more layers also but didn't have any effect on accuracy and loss.The maximum validation accuracy achieved is 62%. 

I tried testing an image from my dataset, for that also model gives wrong prediction. For every test image it predicts as crack, i.e label 1.

Could someone suggest how i can improve this?
Thanks!
Just take 2 images from your training data. One from class 'crack', another one from class 'not crack'. Now, check if your model can get a training accuracy of 100% which means it can overfit on the training dataset or not. If it is not able to do that, something is extremely wrong about the model.