Conflict of interest as a referee?
Disclose your possible conflict to the editor and let them decide. "I had no direct role in the submitted work but am currently collaborating with Drs. Moe, Curly, and Larry on separate projects on plumbing, eye protection, and suit integrity."

It may be that your field is narrow enough that the editor will prefer your mostly impartial review, or they may prefer to find another referee.

This suggestion of course assumes that you think you can offer a fair review whether or not you find the work suitable for publishing. If you think for any reason you would be unwilling to reject the work if it is poor because of your collaborations, you should not agree to review at all.