The complication of not inviting my ex-supervisor to a podcast dedicated to our sub-field?
Using your position as an organizer to not invite the people you don't like to your thing is the only real reason anyone in academia ever organizes anything. They might retaliate but it doesn't really sound like your relationship is worth anything anyway. Just make sure not to actually say what you're doing, if anyone asks you about it just make up a generic excuse about scheduling.