Trust issues with my PhD advisor, is it worth fixing or can I get out?
Faculty are human beings. We might be given only one slot a year for a doctoral student to work with us and we thus invest (or we feel we invest) an incredible amount of time into the students in our lab. We aren't paid extra for this, which perhaps exacerbates the feeling of ownership. 

While I don't know you or your advisor, I would find it easy to believe that his feelings were hurt by your desire to shift PhD programs. You are basically taking his investment in you (which you may feel is minimal, but he may feel is considerable) and throwing it in the trash. 

Managing people's feelings are part of being a professional academic. You might as well practice being good at it.  Some day you'll be on the other end of the equation -- being told by a student that you had high hopes for that they no longer value working with you and that they want to move to a different program altogether. It's all part of the karmic cycle of being faculty. 

Now, with that preamble out of the way. You need to let your advisor know that your shifting programs isn't seen as any rejection at all of his lab or his style of mentoring you, etc. etc. (even if part of this might be true); but rather that you are seeking something different at the other program. It's the stereotypical "It's not you, it's me" form of a break up.