Filter taxonomy term view by referenced taxonomy term
To answer you question: No, it shouldn't need to be configured differently; it should work the way you describe.
Here is a screenshot of my configuration (following your description) that works as you would expect (you can see the contextual filter in the background set to 'yellow'):



Perhaps you can spot some tiny deviation from your config here.
Good luck in any case!