Different custom view per taxonomy
You can use the option Specify validation criteria of the Contextual filter to restrict your Views display to one or more Vocabularies, as you've heard before.
Then clone the display and do the same for different combinations of Vocabularies. You can change the path of each new page to avoid conflicts between different Views and calling multiple queries.
I don't know how Taxonomy display module works, I'm not sure if it improves performance, maybe you end up with the same performance metrics in your custom solution. The nodes still have to be matched with taxonomy in another database table, no matter which options you choose. I'm not a SQL expert, maybe someone else will be able to enlighten about the performance and how it affect Taxonomy term queries.