How to get Cloth to ignore collisions with specific vertex groups?
Instead of trying to selectively ignore vertices, just add some collision objects that follow the legs. Create some super simple ellipsoid colliders that are parented to his thighs, knees, and/or calves and make them more-or-less the same size as the legs.

Bob's your uncle.