CTRL+ALT+FX Equivalent in OSX
There are no virtual consoles like in Linux; there is no way to interact with the system using the connected keyboard/mouse/display outside of the regular GUI which you have already found to be unresponsive.


To bail out with a minimum of needed interaction, there is a shortcut to kill the frontmost application directly: ⌥⇧⌘⎋, Command-Option-Shift-Escape. This is precisely the same action as the usual Force Quit dialog, except that it does not ask you to choose an application.

(Tip: If you press modifier keys while a menu is open, in this case press Shift while the Apple menu is open, shortcut variants like this one will be displayed.)
If you have another machine or even a phone around, another thing you can do is connect via SSH (turn on Remote Login in Sharing preferences) and use that to kill the runaway process. However, if your runaway program is hogging the RAM then I recommend you SSH in before you start work, as making a new SSH connection while the system is stuck in swapping will be just as sluggish, but using an existing connection is less problematic.


(Note that in the unfortunate case where your OpenGL program is hitting glitches in the graphics driver, as opposed to merely consuming resources itself, you may find that the system continues to be unusable even after your process exits, since the entire GUI rendering is based on hardware-accelerated graphics processing.)