Setting to stop opening a new instance of an already open application?
If you are saying that you want each double-click to open the new file in a new tab of the current Sublime window, rather than opening a new window in Sublime, then that is going to be an application-specific preference.  Meaning, you'll need to look at the preferences for Sublime and see if there isn't some equivalent of "Open new file in tab" or "Open new window in new tab" setting.