Is the usage of on-line data augmentation a fair comparison between CNN models
If I understand you correct you are wondering whether the randomness caused by the data augmentation affects the result?
The randomness of the augmentation does not affect the result (at least not to a degree that makes a difference anyway) if you train long enough. The other options you have are (as I think about it):

Augment your data deterministically applying the same transformation to your images before inserting them to your model. Those transformation could be (a) either random ones, e.g. rotate your images by a random degree between some limits, or (b) predetermined ones, e.g. rotate all your images by 1, 3 and 5 degrees.
Don't augment your data at all. Use your initial data to train your model.

The effect of those choices are:

The number of transformation you would apply is limited and even if choice 1a is chosen would be predefined set. If you are willing to increase this dramatically other issues arise like where are you going to store all this data, how are you going to handle it during training etc. So, on the fly has the advantage that the storage of your data does not change, neither is the way you deal with your data. The disadvantage of course being a slower procedure is used (which depending on the transformation could make quite a difference).
This choice to be valid means that you have a lot of data. And by meaning a lot (depending on the problem of course) sometimes a lot is not enough. Since your are (probably) using different data for testing differences appear between your training and testing data in many aspects. For example for human detection (arbitrary choice) differences in poses, colors, light conditions, image clarity, image size, aspect ratio are common. How do you deal with that? You either collect a super huge collection of data or (probably) use data augmentation, right?

To sum it up, it's fair because on the long run it does not make a big difference. Consider the option of early stopping for your model for example. Is it fair to compare models that have stopped their training not in the best iteration? Well, it's not completely fair but it does not make a difference.
I am using on-line data augmentation of images I feed into my Convolutional Neural Network. I am using the Keras ImageDataGenerator for this. The images are augmented in each batch and then the model is trained on these images.
I am comparing different models, but since the images are augmented on the fly, is this really fair, since each models is getting slightly different images?