Hierarchy in entity reference view display
I just came across this question, because I had the same issue. The solution I've found is to use a Relationship; no extra modules required.


Add a Relationship and choose "Parent Term". Check on "Require this Relationship" in the dialog box.
In the "Filter Criteria" section of the view, add the "Term" field (NOT the "Parent" field*). If you want, you can choose to show the hierarchy in a dropdown, in prep for choosing the term in step 4.
Choose the created Relationship in the "Relationship" field of the filter dialog box.
Choose the Parent term in that same filter dialog box (to use your example, say "Category A").


Now your view should output only the children of that parent term. (Again, using your example, if you chose Category A, your view would output:

Category A1
Category A2

*there is a "Parent" field in the Filter field options, and it seems intuitive to use that, but using that only outputs the top-level hierarchy of the Taxonomy, even when combined with an AND with the Taxonomy field. Maybe a bug?