Video background change by difference
First I have to say, that key background is not very good. It needs to be  a very separate colour to skin and foreground (blue or green is best) and it should have NO shadows. Either from folds or from the foreground objects. An even flat plane of color is best.
You could try the Keying screen Node derived by tracking marker samples or you could mix multiple keys to try and build a better key.
Here are a couple chroma keys added together with some erode node added to clean the mask up. Notice that I am using the colours of the foreground object, then I invert the matte to generate a key.

I only have your sample frame to work with so it may fail during animation.

You would still have to use a tracked mask to clean up the holes in the foreground element. Good luck, you're really going to need it.