Am I able to remove unneeded field/migration tables from the database manually?
Yes, use

drush migrate-deregister --orphans


This should get rid of all migration tracking tables that are not longer being referred to.