Is it possible to specify a complex rendering area in Blender?
You could put your model on a renderlayer and then model a plane with a custom shape hole on another layer and then tick exclude and mask for that layer on the model-layer.