Can Power Functions components work with the RCX?
The RCX uses "electric system 9V" style connectors. These are not compatible with power functions connectors. The "electric system 9V" connectors have the connections built into regular lego studs (with the electric part of each stud carefully arranged to allow connections to be made between parts placed at right angles) while the power functions connectors have a special connector section that is not compatible with regular lego studs in any way.

As the other answer mentions the power functions extension cables (8886 for regular length, 8871 for extra-long) can be used as an adapter between "power functions" and "electric system 9V". This cable has two different connectors, one end is a regular power functions connector with "power functions" connections on the top and bottom. The other end is a special connector with "power functions" connections on the top but "electric system 9V" connections on the bottom.

There are a few caveats though.


The power functions only end of the extension cable can be a bit of a pain to mount, because the electrical positions won't stack on top of regular lego studs.
The RCX has a 1-plate high step next to the motor ports. This was not a problem for electric system 9V wires which had the wire exit near the top of the brick but the power functions connectors have a much larger cable exit lump. This means that the power functions adapter cable can't be placed on a RCX in the normal orientation. For the side motor ports you can exit sideways, but for the center motor port you either have to exit over the screen, use an "electric system 9V" cable between the RCX and the adapter cable (so now you have three sections of cable to deal with....) or use a 1x2 electric plate (part 4755) to build-up the connector on the RCX.
"Power functions" is a 4-wire system while "electric system 9" is only two wire. So while you can use basic power functions motors with the adapter cable things like the servo motor will not work and you can't supply power to things like the IR receiver from an "electric system 9V source. 
For smaller models actually building your model may be a pain because of the extra wire length from the adapter and because the RCX was designed with old-style technic in mind while power functions was designed with new-style technic in mind.


As for connecting multiple motors you can do so physically but if the motors are under heavy load you may hit the current limits in the motor drivers, heck even a single "RC race buggy motor" can overload the drivers in the RCX.