Pinning cloth to animated model is making cloth go crazy
There are few things that seem to be causing this.

The main thing was that the legs had a largish outer collision margin which was causing the robe to collide constantly.

Try changing this to something like .004 on both legs:



Also try making the skirt bigger initially. There has to be room for the legs to move ;)

Result:



I also pinned the entire top edge loop instead of just the corner vertices, reduced the collision quality to about 3 and disabled self collision. These massively decreased the bake time without a lot of visual impact once the collision issue was fixed.