How do you know when you are using a multi gpu?
If you are using Linux and Nvidia GPUs, you can do the following in a terminal

nvidia-smi


Which will show you some stats about the GPUs available on your system.

You can run it automatically every 2 seconds like this to see how power/memory usage changes during training:

watch -n 2 nvidia-smi


If you see nothing or the nvidia-smi command fails, you likely don't have the correct drivers installed.

If theya re showing, but Keras/Tensorflow is not finding them, have a look at this thread for more checks for Tensorflow backend.