Build taxonomy categories with views, only children
Can you set up your Taxonomy view to accept the argument Parent term and pass it the current term? I don't know how complex your view is, but I got it to work in this basic example:

View type: Term
Add arguments: Taxonomy: Parent term
Action to take if argument is not present: Hide view / Page not found (404)
Validator: Taxonomy term
Argument type: Term ID
Add fields: Taxonomy: Term Term

That will display only the current term's children, i.e. those that have the current term as a parent.