How to customize pagination for the taxonomy term view page for different vocabulary?
I've used the Taxonomy Views Integrator module which helped me set custom pagination and view modes by creating taxonomy view page for different taxonomy terms and vocabularies. Some details about this module (from its project page):

The TVI (Taxonomy Views Integrator) module allows selective overriding of taxonomy terms and/or vocabulary with the view of your choice. Using TVI you can easily create custom views to output all terms in X vocabulary.
TVI will try not to interfere with views or taxonomy/term page displays if no TVI views exist for the current page request.