My CNN produces volatile validation_loss and does not converge in python(Keras)
It seems that there is some form of randomness or too many parameters in my model. I am aware that it is currently overfitting, but that should not be the cause of the volatility(?)


It is quite plausible that this behavior is caused by a combination of the model's excessive capacity and the data that you are feeding into it.

As you have pointed out, overfitting is obvious here - the model cannot generalize well and fails on unseen data. If you reduce model capacity (better yet, start off with the bare minimum and incrementally build up from there), then the network will be forced to learn instead of performing memorization. Since you mention that you have already tried tuning rates/layers/sizes, the model alone cannot be the cause, although maybe someone with more expertise in convolutional layers can provide useful insight.

The lack of convergence can be an indicator that there is simply not enough data to learn from - neural networks aren't guaranteed to converge when there is insufficient data. The required amount depends greatly on the specifics of the task.


  my data set consist of about 1800 images...600 of each dog


That's quite a small dataset, considering that:


  Two dogs are very similar and the 3rd is very different.


So roughly two thirds of your images can cause confusion. The more they look alike, the more samples you will need to capture the details.


  The videos are taken on different backgrounds, but +- the same amount on each background for each dog.


The model could then erroneously try to discriminate between the two similar-looking dogs based on the background alone (think worst-case scenario).

In your case, I would look at the data more closely, investigate which images are consistently the easiest/hardest for the model to classify and take full advantage of ImageDataGenerator's image augmentation to improve generalization capabilities. See how the model responds if you place similar-looking dogs under the same label, or have one of them appear exclusively in training, the other in validation. Maybe there are some answers there.