How to specify castellations in gerber files
Castellations are special, custom, and highly fab-dependent.  

I've done it with holes that are cut in half by the outline layer.  But I supplied an extra drawing explaining what I wanted.  The castellations came out fine after some negotiation.  The board vendors tend to do some massaging of the CAD data anyway, so whatever extra work was required for castellations did not bother them.

The plating is an extra step that has to be done after routing, so expect to spend rather more money than a standard-spec board would cost.