Can I use my Macbook Pro Retina as a monitor for PS3?
No, the video ports (HDMI and Thunderbolt) are output only. The MacBooks aren't designed to function as monitors.

You could try getting a video capture device that supports live viewing, but the performance and quality are likely to be far from optimal—such solutions are designed for recording video, not live playback or gaming.