Why field tables are not getting removed when I remove the field?
I feel like having these many unused tables in database might impact the performance


That's very unlikely, unless the extra disk space used will somehow affect your server's performance.

Deleted fields and data are cleared up on cron runs (by field_purge_batch()), so running cron a few times will probably get rid of them. Exactly how many runs it takes will depend on the number of fields, amount of data in each, etc.

I'd advise against removing it manually unless you're quite familiar with the internals of the field API, and understand the implications.