How to get vocabularies for an entity reference to a taxonomy?
So here is my process (it won't work when you're loading in the relevant entity types via a view, only by the "default" method):

$type is the content type, $field is the field of the ER.

$field_data = \Drupal::config('field.field.node.' . $type . "." . $field)->get();

$bundles = $field_data['settings']['handler_settings']['target_bundles'];

At which point you can iterate over the given bundles for an ER.

You'll probably want to do some flow control somewhere to ensure the ER has the correct handler you want - located here:

$handler = $field_data['settings']['handler'];


But this should work to get the data I asked for above.