Custom entity datetime field in calendar
EntityViewsData provides the information for views for entity types, so that they are included in views. For datetime basefields this is not working at the moment, because of this issue https://www.drupal.org/node/2489476. So you need to provide the data in a hook_views_data or by extending EntityViewsData. From the documentation:


  
  argument: Specifies a handler to make it available to Views as an   argument, or contextual filter as it is known in the UI.
  


https://api.drupal.org/api/drupal/core%21modules%21views%21views.api.php/function/hook_views_data/8.3.x

You've mentioned that you have a working example if you use a field configured in UI, so you could try to copy the views data.