Is it right for a supervisor to try to prevent his students from pursuing other research directions by simultaneously working with other professors?
First and foremost it is the advisor's responsibility to aid and guide your research to get you out the door and into a career. But there are many ways to do that. 

One interpretation of the advisor not wanting you to work with others  is that it is a distraction and leading to your loss of focus. That would be a good reason for "preventing" your work with others. 

But, after two years, if you don't yet have a problem to focus on, then the advisor is falling down on the job, perhaps. They should be making suggestions about questions and directions and suggesting starting points, etc. But if they have been good at directing your research then you should have no issue with them. 

You have hearsay evidence about their letters of recommendation, but no direct evidence (I think). It is probably best to ignore it. 

I doubt that it is in any advisor's best interest to "keep people around." Everyone is better served by getting people out the door and new people in. There are a few exceptions, but you don't give indicators of that here. 

The advisor is correct to try to get you to focus. You will be best served by leaving other things aside and focusing on the one big idea that will lead to success. Perhaps the issues with "exchange" opportunities is also about getting you to focus, but it is impossible to say from a distance. 

If you have a lot of ideas, I suggest that you simply record them in a "future research" notebook and then let them go for now. It is very useful to start a new job with such a notebook. But wandering around a vast landscape won't get you to that spot on the research frontier that will lead to success. 

Nose to the grindstone. 

However, there needs to be mutual trust between student and advisor. If you have truly lost that, then it might be best for you to give up your current advisor and move to another. It might mean starting over, and it might or might not have an effect on your ability to "focus".