Choosing wireless tech for lowest possible lag
I've used NRF24L01+ chipset 2.4GHz wireless modules with Arduino before, and found them to be great, and super cheap (~$10 for 10 of them on ebay!). They have 3 modes of transmission: 250kbps, 1Mbps, and 2Mbps. The range decreases accordingly with higher bitrate, but the time spent sending a message does too. There are multiple Arduino libraries (e.g. RF24, Mirf, RadioHead) and tutorials for using the modules (see http://playground.arduino.cc/InterfacingWithHardware/Nrf24L01). They also have built-in mesh networking modes which may or may not be of use.

Versions with an RP-SMA antenna connector and an advertised range of up to 1000m are also available for about $5 each. I would recommend using one of these for the referee's button at least, depending on the range you need.

According to Charles Hallard of hallard.me, The non-amplified chips can get 30m line-of-sight range in 250kbps mode, and I have verified this in my own testing.

According to user sporadic on diychristmas.org, sending a 32 byte payload in 250kbps mode takes 1432µs from the start of Tx mode on the transmitter to receiving the full message on receiver. This decreases to 444µs on 1Mbps mode and 283µs on 2Mbps mode.

Given these statistics, they fall well within 2ms latency for triggering, suitable for your application, and they use very little power. It's up to you to work out the best balance of latency, range and expense.