Improving Validation Loss and Accuracy for CNN I am new to CNNs and need some direction as I can't get any improvement in my validation results.

I am trying to do binary image classification on pictures of groups of small plastic pieces to detect defects. Unfortunately, I am unable to share pictures, but each picture is a group of round white pieces on a black background. One class includes pictures with all normal pieces, the other class includes pictures where two pieces in the picture are stuck together - and therefore defective.

I have a small data set: 250 pictures per class for training, 50 per class for validation, 30 per class for testing. The pictures are 256 x 256 pixels, although I can have a different resolution if needed.

Here is my CNN architecture:

classifier = Sequential()
classifier.add(Conv2D(32, (7, 7), padding="same", input_shape=(256, 256, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Dropout(0.5))

classifier.add(Conv2D(64, (5, 5), padding="same", input_shape=(256, 256, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Dropout(0.5))

classifier.add(Flatten())

classifier.add(Dense(units=128, activation='relu'))
classifier.add(Dense(units=1, activation='sigmoid'))

classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Treatment done to images:
train_datagen = ImageDataGenerator(rescale=1./255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)
validation_datagen = ImageDataGenerator(rescale=1./255)

train_batch_size = 10
val_batch_size = 10
num_epochs = 100
train_images = 250
val_images = 50

classifier.fit_generator(training_set,
                             steps_per_epoch=train_images // train_batch_size
                             epochs=num_epochs,
                             validation_data=validation_set,
                             validation_steps=val_images // val_batch_size) 


Here are the results:



It's overfitting and the validation loss increases over time. The validation accuracy is not better than a coin toss, so clearly my model is not learning anything.

I have tried different values of dropout and L1/L2 for both the convolutional and FC layers, but validation accuracy is never better than a coin toss.

I understand that my data set is very small, but even getting a small increase in validation would be acceptable as long as my model seems correct, which it doesn't at this point.

Update:
Switching from binary to multiclass classification helped raise the validation accuracy and reduced the validation loss, but it still grows consistenly:



Any advice would be very appreciated. Thanks in advance!
Your data set is very small, so you definitely should try your luck at transfer learning, if it is an option.

Other than that, you probably should have a dropout layer after the dense-128 layer. Also, it is probably a good idea to remove dropouts after pooling layers. Instead, you can try using SpatialDropout after convolutional layers.