Xbee series 2 - router or end device firmware?
I don't think the router or end device firmware makes a difference. I always use the router firmware, since I have enough power available, and never had such delays as you are experiencing. 


Are you maybe using them in an area with lots of interference?
Are you using broadcasts to send data? Broadcasts can cause an overload of the network that could easily create those delays and even more.
How big are the packets you are sending? An XBee packet has a relatively small size. If your data needs to be split, then any delays due to other factors will be multiplied.
Are your microcontrollers/data sources set to the same serial speed and parity etc. as the XBees?


As a first step I suggest to enable acknowledgements for the packets you send and see how many retries were needed (can be done in API mode). Then you'll know better what to exclude.