Display configuration problems with Mac Pro
Apparently, there are three buses on the Mac Pro, which can handle either two normal (p.e. Full HD) displays or one 4k display.
[1]     [2]
[1]     [2]
[0]     [0]

The numbers indicate to which bus those ports are connected. Moreover, the Mac Pro has got an HDMI output, which is also connected to bus 0.
Apparently however, if you want to get three 4K displays to work, you have to plug one of them into a port from bus 1, one into a port from bus 2 and one directly into the HDMI-port for some reason. (I assume that this will not be a problem as you are using Thunderbolt to HDMI adapters anyway)
For more detailed information, please take a look at: How do I connect a 3rd 4K monitor to my 2013 Mac Pro?
Good luck!