How can I prevent Messages from opening up a new chat window?
Unfortunately this is something that isn't possible. As you are aware, the only control we have over any double-clicking in the "Double-Click Speed" in Mouse preferences. Any changes here would be undesirable as it would affect the entire system and your goal is just the Messages app. 

There is a convenient keyboard shortcut, Command + W, to close the new window that comes up, so at least you don't have to go over and manually close the window.

I suppose this is a feature request that doesn't come up very often but you could at least do what you can and fill out a feature request.