Clothes moving weirdly after rigging
One possible solution to your problem is to manually assign and remove vertices from the weight groups. having very little geometry as you do, it should be fairly simple to tweak the weights based on the geometry.
This can be accessed through the vertex group panel (upside-down triangle)
Based on your description I cannot tell if you have tried this yet or not.