Using three image datasets with different image sizes to train a CNN
Resizing the images will work.But if you significantly reduce the image dimensions information IS lost. Here is a simple example. Suppose you have a 500 X 500 image. That has 250,000 pixels. Assume in these images the object of interest( lets say a bird in a forest) only occupies 10% of the pixels in the image(25,000 pixels). Now assume you reduce the image to 100 X 100 and thus have 10,000 pixels. Your object of interest (the bird) now only occupies 1000 pixels. These are the pixels the neural net will learn from.
Now a better way to do this is to first crop your original 500 X 500 images to maximize the percentage of bird pixels in the cropped image. For example assume the resulting cropped image comes out to be say a 200 X 200 image but in that image the subject of interest(the bird) occupies 50% of the pixels (20,000 pixels). Now if you reduce the cropped image down to a 100 X 100 the object of interest will occupy(5000 pixels).
Cropping images is of course a hassle. For some types like images for example images of people, and you are just interested in the faces there are routines available that will automatically crop the image to just the face. In general you will have to crop the images yourself.
If you want to build high accuracy classifiers you have to start with a "robust" data set.
The larger the percentage the object of interest takes up in the images the better your classifier will perform.