What to do with a GAN that trained well but got worse over time?
This is from my own experience with (Vanilla) GANs, so it might not translate exactly to your application, but maybe it gives some orientation.

your learning rate seems quite high. I've quite frequently found that 1e-5 is a good value for me. The training might take longer but will probably be more stable.
have you tried using dropout? It's a good regularisation mechanism that can prevent from overfitting and also make training more stable. I've made good experience with a dropout rate of ~0.4 and keeping it in both training and testing stages.
what is your discriminator vs generator update ratio? In my experiments, ~5 discriminator updates per generator update has often shown to improve things.
how many training samples do you have? If they are limited, it might be helpful to add a bit of noise on them. This manipulates the data a bit, but might make training more stable.
I have often not seen a major change when varying the batch size.
what are your network architectures? I have the feeling that small networks are often sufficient...

This is entirely based on my own experience, but maybe there are some interesting directions for you to explore. Happy to hear back from you about the success of these tests.