Do Fields Slow Down Drupal
If you have Tokens module enabled and also if you have a lot of fields, then it'll take a lot of time for the token ui to load up. This was the problem in my site. It took me a long time to figure this out. If there is a performance hit, only when you try to edit or create a content type then it might be caused by the tokens module. It's worth checking it out.