How to use Vertical Tabs on ConfigEntity Form
Use the vertical_tabs form element. There is an example shown in the API documentation for this form element: https://api.drupal.org/api/drupal/core%21lib%21Drupal%21Core%21Render%21Element%21VerticalTabs.php/class/VerticalTabs
In short, instead of using nested fieldsets, create a 'vertical_tabs' form element as the container then put multiple 'details' form elements inside it. Each of the 'details' elements will serve as the container for one vertical tab. See the example in the API documentation for the exact syntax.