Widening Hole in Cloth Sim
my guess is: bad topology of the cloth object...

here's your mesh topology:



And topology is all about good deformations...

I did a quick test of a "square" cloth object (not particularly good topology, but way better than your object...), just to check and, with default cloth / collision settings... is subdived and then also subssurfed...

animation starts like this:



and after 136 frames:



here is my test .blend


update:

here is a simpler test with higher "structural" setting...