Is the usage of on-line data augmentation a fair comparison between CNN models
In my opinion, you already give part of the answer within your question:

images are augmented on the fly, is this really fair, since each models is getting slightly different images?


For Evaluation / Validation I usually try to provide situations as similar as possible over the different architectures - otherwise you might induce unnecessary bias you are not able to account for.
Also you could reduce computational effort by offline-Augmentation and then directly handing over the same augmented Training samples
I am using on-line data augmentation of images I feed into my Convolutional Neural Network. I am using the Keras ImageDataGenerator for this. The images are augmented in each batch and then the model is trained on these images.
I am comparing different models, but since the images are augmented on the fly, is this really fair, since each models is getting slightly different images?