How do I only display list of categories at certain level of parent category that have entries
If I understand your issue correctly, try changing 

{% set topLevelCats = craft.categories.relatedTo({ sourceElement: entryIds }).group('filter').level(1) %}
{% set secondLevelCats = topLevelCats.slug(secondSeg) %}


to

{% set topLevelCats = craft.categories.relatedTo({ sourceElement: entryIds }).group('filter').level(1) %}
{% set secondLevelCats = craft.categories.slug(secondSeg).level(2) %}