When should I create multiple Displays of a View instead of multiple Views? Is it simply a matter of convenience to create multiple Displays of a View instead of multiple Views, or is there a performance benefit?

Does having any of the following items present in some Displays but not others affect the View as a whole?


contextual filters
content type filters
aggregation


For example, I'm curious whether having a display show just Article content type results, another showing Forum results, another showing a custom Photo content type makes each display run slower.  Or if the Views all share the same content type, but differ only in that one has a contextual filter for nid, and the others don't, do they all slow down?

If I have 30 Displays for a single view, should I refactor it for performance benefit, or just because the UI wasn't meant to show that many Displays.
Code-wise, both Views and displays are represented by objects. When a View is being processed, it loads all it's child displays into memory, regardless if you are rendering only one View in the page you are looking at. While database queries are executed only for the active display, the properties of the inactive displays are still loaded into memory.

This means that if you have 50 displays in one View, 50 displays will be loaded into memory, regardless if you are only displaying only one of them. So the answer to your question is yes, typically the more displays in a view you have, the more inefficient the View becomes memory-wise. 

With that being said, I personally think that having anywhere between 2-10 displays in a View should prove negligible memory-wise. Having a discreet amount of displays in a View does serve it's practical purposes, after all (otherwise, why would they be there in the first place?).

As to your second question (performance impact of additional filters, etc), yes, adding certain parameters to your View will affect performance. If you want to know by exactly how much, you can go to admin/structure/views/settings (if you are in D7) and select Show information and statistics about the view during live preview, Show the SQL query, and Show performance statistics. Here is a screenshot of the performance screen:



If you are further interested in the behavior of the queries (how many rows are analyzed, which keys are used), you can copy the query from the Views interface (once you've selected "Show SQL query"), and analyze the query using the MySQL explain command. There's plenty of articles like this one for explain usage on Google.