Setting batch size: training requires twice as much memory as validating
Aside: Tensorflow-GPU currently grabs 100% of GPU memory by design -- so that may may change the challenge a bit.

To add to @hssay on point answer -- with images, it is helpful to look at the entire pipeline.  Optimize the input pipeline, then you will use less memory.  May want to start here: TensorFlow

Test, test test.  You may find that your results do not really change much with smaller image files (less resolution, color, etc).  In some cases, results may even get better (the overtraining/undertraining balance).

[this is really should be a comment, but my rep does not transfer here :( ]