Speed of training decrease by adding more GPUs
Using GPUs can accelerate training but as you increase the number of GPUs your training as to be distributed which means your data has to be moved to several GPUs which can cost in term of bandwidth. I would profile the training and see the time spent to move the data to GPUs and et results back. Plus it's harder to syncronize training like this.

If you are using tensorflow 1.14+ try to change the distribution method to "MirroredStrategy" . This tend to work better with multiple GPUs I found.