How can I specify an area to render based on mask image in Cycles?
Yes, a Holdout shader is what you're looking for. I frequently use this technique when I want to re-render part of a scene and composite it into a previous version.

Your second idea of a mesh in front of the camera is exactly what I do. You can put a plane in front of the camera and make a hole in it whatever shape you need. Give the plane a Holdout shader, and everything behind it will be totally ignored. It'll only render what you can see though the hole. The hole can be any shape you want.