SQL Service Broker and AlwaysOn Issue
You need to setup a AG aware SSB route between x and y. See Service Broker with AlwaysOn Availability Groups. This route will deliver the messages over the network even when x and y are collocated on the same host. When failover occurs and x and y are on different hosts, the delivery continues uninterrupted.


  My personal guess is that SSB Always uses the local services and I have no way out of this.


You should read Service Broker Routing (although is a bit dated, as it misses the AG aware endpoints). SSB uses the local services (the "auto" routes) as a last resort, if no explicit route is declared. You always have a way "out" by explicitly creating a route.,