SWD optoisolation problem
Yes, you're right X5 and X6 are making a positive feedback loop. You need a bidirectional optoisolator for SWDIO line. For speeds up to 1 Mhz you can find guidance in this NXP Application note. You also need to see if the 160 ns delay induced by the optocouplers (2x80ns,forward path + backward path) is acceptable for the programmer. You might want to change the point of view and insulate the USB side of the programmer with a device like USB ISO form Olimex