How to stop modal mode from unregister function?
I would not recommend starting your modal operator in the register() function.  Typically you want to add a button or something to a UI panel, which the user can use to invoke your operator.

But if you're sure you want to start and stop your operator using register() and unregister(), then you could try something like this:

#insert this after your imports:
still_running = True

#insert this in your modal() method:
global still_running
if still_running == False:
    return {'FINISHED'}

#insert this in your unregister() method:
global still_running
still_running = False