Visibilities permissions of children links in the Toolbar
This is not a problem with the Admin Toolbar module, I think that you're not given the correct permissions.

In Drupal >= 8.5 you need to give the following permissions:


Use the administration pages and help
Use the administration toolbar
Tags: Create terms 
Tags: Delete terms 
Tags: Edit terms 
Access the taxonomy vocabulary overview page


In Drupal &lt; 8.5 as the Access the taxonomy vocabulary overview page not exists you need to assign the following permissions:


Use the administration pages and help
Use the administration toolbar
Tags: Create terms 
Tags: Delete terms 
Tags: Edit terms 
Administer vocabularies and terms


The problem here is that the user will have the permission to create new vocabularies.

You can read about this in New term overview and create permissions and in Add a dedicated permission to access the term overview page (without 'administer taxonomy' permission)