How to record the screen of an OpenGL application without stuttering?
The reason for the “stuttering” could be a combination of different factors.

First is of course your computer’s power and video card. The more power you have, the more the number of FPSs your computer can “record”. 

Despite that, QuickTime is not known for its “fast” recording, so I suggest you give a try to different alternatives. This comparison is a little bit “outdated” because some (if not most) of the applications mentioned have been updated and have fixed or changed some of the problems they had, however, it should be a starting point for your final QUickTime Replacement. This other link has a very similar list, but rather than a review is a list with a short description of the products.

For reference, the most important Screen Capture tools for OS X (or to put it in another way, the “most used ones”) are (in no particular order): iShowU, Snapz Pro X, ScreenFlow and one that is missing from those lists (because it’s a new player in the Mac world): Camtasia for Mac (which I’ve been betatesting and it’s quite good and in par with the ones I’ve mentioned). Some features missing from Camtasia 1.0 have been addressed in an update and I know they are working on a newer version after getting tons of feedback in the Beta forums (which I can’t disclose because I participated).

I’ve personally tried those four I’ve mentioned (i have a Snapz Pro license) and I think they more or less are the same, but some were (when I tried them) faster than others but lacked certain features. All in all, the “best” doesn’t exist, they all have strengths and weaknesses so your best bet is to give them a try and see if one does the job.

Last but not least, remember that quality also plays an important factor in CPU overhead when recording, try to keep your CPU load low and reduce the recording quality as much as you can afford.