Since the branching rules from U(N) to U(N-1) or SO(N) to SO(N-1) have multiplicity one, the irreducible summands corresponding to smaller and smaller N will eventually terminate in one-dimensional subspaces.
For the remaining classical group Sp(N), the branching is no longer multiplicity free, so that if V and W are irreducible representation of Sp(N-1) and Sp(N) the space of intertwiners HomSp(N-1)(V,W) can have dimension greater than one.