Assigning User's In Taxonomy Terms
By default, the taxonomy term view lists content (nodes) and not users. If you want to use a new view on the default term page path, taxonomy/term/%, you have to disable the default view, create a new one, and give it that path. But it will also be in use for your others taxonomy terms pages !

So my advice is to disable and ignore this view, give a path to your full working new view, and set your term path pattern for this vocabulary the adress of your new view's page.

More simply : 


Make a new view of users, set a "page" display and set it like this : /mypath/%
Add a taxonomy term ID argument
Add your fields
Go to your pattern url alias setting page, and for your vocabulary "gender", set terms URL to /mypath/% .


Now, all gender terms on your site will be linked to that view.