How do I only display list of categories at certain level of parent category that have entries I try to explain you semantically what I want to achieve here. I couldn't make it shorter. Sorry.


Navigation will be constructed from .level(1) categories.
At "site.com/firstLevelCat" I want to display list of populated .level(2) categories like this


site.com/firstLevelCat/secondLevelCat
site.com/firstLevelCat/secondLevelCat2
site.com/firstLevelCat/secondLevelCat3

At "site.com/firstLevelCat/secondLevelCat" I'd like to display list of .level(2) categories along with a list of .level(3) categories like this


site.com/firstLevelCat/secondLevelCat/thirdLevelCat
site.com/firstLevelCat/secondLevelCat2/thirdLevelCat
site.com/firstLevelCat/secondLevelCat3/thirdLevelCat



I intend to build it up to 4th level.
If I understand your issue correctly, try changing 

{% set topLevelCats = craft.categories.relatedTo({ sourceElement: entryIds }).group('filter').level(1) %}
{% set secondLevelCats = topLevelCats.slug(secondSeg) %}


to

{% set topLevelCats = craft.categories.relatedTo({ sourceElement: entryIds }).group('filter').level(1) %}
{% set secondLevelCats = craft.categories.slug(secondSeg).level(2) %}