Modifying keyframes for a mask created in the UV editor
I am not 100% sure, what you mean by "created a mask in the UV editor", because as far as I know the only way to make masks in Blender is in either the image or the movie clip editor.
But maybe I am missing something.

If you mean that you want to animate an edge of the unwrap of an object, then the short answer is that Blender cant do that.
A really quick google search unearthed this addon, though: https://www.blendernation.com/2011/11/12/animating-uv-textures/

If you are already using it and are wondering why these properties still dont show up in the graph editor, you should probably try to contact the developer directly.