TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?
Tensorflow handles batches differently on distribution strategies if you're using Keras, Estimator, or custom training loops.

Since you are using TF1.15 Estimator with MirroredStrategy in one worker (1 machine), each replica (one per GPU) will receive a batch size of FLAGS.train_batch_size. So, if you have 4 GPUs, then the global batch size will be 4 * FLAGS.train_batch_size.

Here's the explanation:


  In Estimator, however, the user provides an input_fn and have full control over how they want their data to be distributed across workers and devices. We do not do automatic splitting of batch, nor automatically shard the data across different workers. The provided input_fn is called once per worker, thus giving one dataset per worker. Then one batch from that dataset is fed to one replica on that worker, thereby consuming N batches for N replicas on 1 worker. In other words, the dataset returned by the input_fn should provide batches of size PER_REPLICA_BATCH_SIZE. And the global batch size for a step can be obtained as PER_REPLICA_BATCH_SIZE * strategy.num_replicas_in_sync.


Source: TF1.X Distribution Strategy Notebook