How to save a vote of an anonymous user only after registering using modal window?
You could store anonymous votes in a cookie, then on login/register, check to see if that cookie exists and cast the votes accordingly.


When an anon user tries to vote, use user_cookie_save() to save the Entity Type and Entity ID of the Entity that the user was trying to vote on in a cookie. Name it something like pending_votes. At the same time, you can trigger your modal login window. It's possible that a user won't login/register at that time and it's also possible that they will attempt to vote on other Entities before finally logging in. So you'll probably want to store the Entity Type and ID as a serialized array.
Then you can hook into a user logging in with hook_user_login() and check for the existence of the pending_votes cookie. If it's there, use it's contents along with votingapi_set_votes() to cast the pending votes. (delete the pending_votes cookie afterwards so they don't cast more votes each time they log in)