How can I get two Arduinos to "talk" to each other?
Wired

Communication speed is not a factor - you only have to transmit an on/off signal. The simplest method would be to run a 4-wire cable, like a telephone cable, between the two.

The wires would be


GND 
5V (maybe) 
Ground floor output -> fourth floor input
Fourth floor output -> ground floor input


The steps are


Both wire 3 and 4 start LOW When ground floor button is pressed, ground floor led goes on, wire 3 is set HIGH. 
Fourth floor is monitoring wire 3, when it goes HIGH, fourth floor led goes on, and    wire 4 is set HIGH 
Ground floor is monitoring wire 4, when it goes HIGH, ground floor led goes off, and wire 3 is set LOW 
Fourth floor is monitoring wire 3, when it goes LOW, fourth floor led goes off,    and wire 4 is set LOW 
Ground floor is monitoring wire 4, when it    goes LOW, it begins waiting for a button press again.


Wire 2 can be used for 5V (or 3.3V depending on the Arduino) so that one of the devices can be powered off the other.

Wireless

The other option is to attach an RF transceiver to each Arduino, something like this would be good as it has 100m line-of-site range which you might need to get through 4 floors of concrete.

Learning

Start off with the simple wired method shown above. When it is working, connect wire 3 and 4 into the RX and TX pins and try to communicate using the serial port. Finally, move on to the wireless transceivers.