How to clean up multiple Drush Composer installs?
composer global remove drush/drush will get rid of either the ~/.composer or the ~/.config copy, depending on your current configuration.  However, if you are no longer using the components that you installed via composer global, you can just use rm -rf to delete the directories you don't need, which is vendor and bin in ~/.composer and ~/.config.