Reducing number of DB queries when using views, search_api and relationships
Is there any way get Views to load all of the referenced entities in one go? No, there's not much you can easily change to fix that. 

There's a lot of behaviour built in so that Drupal can handle multilingual data or different data storage back ends like MongoDB.

If you're not already doing so, you can use entitycache module to lighten the load. For cached entities, you will eliminate a lot of queries.

FYI, Entitycache will not work on its own. You will need to integrate with a cache back end like Redis or Memcache. See the module documentation and READMEs for configuration details.