Update views result after unset() in views_pre_render
You may want to look at doing this in Views' hook_views_post_execute():


  This hook is called right after the execute process. The query has been executed, but the pre_render() phase has not yet happened for handlers.


as the paging happens in the handler phase, and hook_views_pre_render()


  This hook is called right before the render process. The query has been executed, and the pre_render() phase has already happened for handlers, so all data should be available.


All about the various Views hooks is available here:  

https://api.drupal.org/api/views/views.api.php/group/views_hooks/7

Also, some interesting conversation about this process and potential alternate suggestions is available here in a drupal.org issue queue "update pager count after chaning results in hook_views_pre_render" as well.