user_delete function is very slow
If you want to maintain data integrity, there's not much you can do to speed the process up except upgrade your server's hardware.

user_delete calls user_delete_multiple, and it has plenty to do for each record; 
 not just deleting the user record, it deletes fields/data, invokes hooks, clears caches, etc. All of that needs to happen or you're likely to find problems further down the line.

If you're finding that the slowdown is causing the script to timeout, you can process the list with the Batch API and solve that at least.

Incidentally, if you don't care about data integrity, you can just cherry pick the lines from user_delete_multiple that you do need, and execute those directly instead.