How to politely turn down others' request for collaboration?
Thank you so much for keeping me in mind for this. Even though the idea sounds exciting, I'm currently swamped with work, and cannot join a new collaboration at this point. I'm very sorry!

Depending on whether you see the chance to collaborate in the future, or you just want to get rid of them: You can also add that they can contact you again at some later point, or even that you will contact them.