How to insert a keyframe for Mask Point coordinates
It seems like keyframe insertion for masks is exclusively handled by bpy.ops.mask.shape_key_insert() and cannot be done from python in a different manner. This operator only works in the right context (mask open in clip editor or image editor, or mask strip selected in sequence editor) and if the points of your mask are currently selected.