Does unset($form['field']); in hook_form_alter stop entity reference views select list queries from running?
Short Answer: Unsetting the form element won't stop the view from being called. You may want to use an autocomplete widget instead if that is an option (if only to test your theory is correct - that the views are slowing the page load).

Long Answer:
When hook_form_alter is called, what you get is a form that contains elements from the Field API. Modules such as Entity Reference typically do not create new Form API elements - instead they provide field widgets.

Such modules contain code that generate their widget using the base elements in the Form API. This is typically done when field_attach_form is called, and it is at this point that the view would be called.

This is done before hook_form_alter is called. So what you get in form alter is the low level Form API elements - not the high level widget. Unsetting the field won't make a difference, since the low-level elements that contain the list of options have already been created.

It is possible that some module would actually use hook_form_alter to populate their widget - I have seen this done, and must say it is very bad practice - but I do not think this is the case with Entity Reference.