Attending a conference where my ex-supervisor and his collaborator are present, should I attend?
First of all, I’m sorry to hear that you’re going through this.

I would ignore them in the conference. I don’t know if this motivates you, but going to a conference, doing an amazing job and still shining in my field despite their sabotage attempts would be really satisfying to me. 

On a professional level, if your job requires conference attendance and you categorically refuse for whatever reason, it will not help your career. Don’t let them sabotage you further!