How to add a base field for an entity reference to a specific vocabulary
Vocabularies are bundles of the entity type taxonomy term. If you want to reference a vocabulary then first set taxonomy_term as target type and second restrict the field to a specific vocabulary as target bundle:

$definition = BaseFieldDefinition::create('entity_reference')
  -&gt;setSetting('target_type', 'taxonomy_term')
  -&gt;setSetting('handler_settings', ['target_bundles' =&gt; ['custom_vocabulary' =&gt; 'custom_vocabulary']]);


See change record https://www.drupal.org/node/2576151