How to convert selected vertices to mask?
import bpy
import bmesh

Om = bpy.context.object.data
vertm = [v.select for v in Om.vertices]
bm = bmesh.new()
bm.from_mesh(Om)
if not bm.verts.layers.paint_mask:
    m = bm.verts.layers.paint_mask.new()
else:
    m = bm.verts.layers.paint_mask[0]
for i, i2 in zip(bm.verts, vertm):
    i[m] = i2
bm.to_mesh(Om)
bm.clear()
Om.update()


this will work in Object and Sculpt modes. Not while in Edit mode.