Which wireless controller is more compatible with Macs: Xbox 360 or PS3?
Having tried both the Xbox controller and the PS3 controller on my Mac I can say: the PS3 controller is somewhat easier to get working with the Mac. It communicates via BlueTooth and pairing it is mostly painless. It's also wireless without requiring any other devices to be purchased. You can get the PS3 controller drivers from here.

The Xbox controller requires an additional USB dongle from Microsoft to get working if you want to use a wireless controller. Otherwise, you'll have to stick to using it as a wired controller. You can get the Xbox 360 controller drivers from here or here is an alternative driver.

I thought the drivers for the PS3 controller were a little more stable and that the stick response was snappier than the 360 controller. But I much prefer the 360 controllers layout, weight and overall tactile feel. Personal preference there really.