Entries from different category levels brought together
What you want is the categories that do not have descendants. I would loop over all the categories, and skip the ones with descendants. The test to skip is:

{% if not category.hasDescendants %} ... {% endif %}


Documentation for hasDescendants.