Drupal 7 Taxonomy term display page
I think what you would like to achieve is like a glossary.

You can by default create a glossary with views. How to do this is best explained or shown in this video. 

If you want to show each of these on one page, i think you need to have some kind of a special field to group your results by. For this, try to the Views PHP module. Make a field where you grab the first letter of each term, and group by this (for example).