DROP USER takes too long when there are many users
The resolution to this issue was enabling service broker on the database as so.

ALTER DATABASE [Database_name] SET NEW_BROKER WITH ROLLBACK IMMEDIATE;


After enabling service broker for the database, drop users were practically instantaneous. 

Kin asked if the service broker was enabled in a previous comment which sent me searching in the right direction.