How to prevent SQL files opening new instances of SSMS 17.9.1?
I have encountered this issue after installing SSMS 18 instead of 17.9. Then I have tried solutions from the list above but none worked. Uninstalling SSMS 18 and installing 17.9 again did not work either. I was finally able to resolve this issue with SSMS 18, but I hope it will fix incorrect behavior of SSMS 17 as well. First of all I've uninstalled SSMS, then I've searched through registry to find and remove all meaningful nodes containing info about SSMS or sql_auto_file. I removed the following nodes (don't forget to backup each of them prior to deleting!):


HKEY_CLASSES_ROOT.sql
HKEY_CLASSES_ROOT\Applications\Ssms.exe
HKEY_CLASSES_ROOT\sql_auto_file
HKEY_CURRENT_USER\Software\Microsoft\SQL Server Management Studio
Removed all records for ssms in HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\ApplicationAssociationToasts 
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts.sql
HKEY_USERS.DEFAULT\Software\Microsoft\SQL Server Management Studio


Then I've restarted my PC, then installed SSMS 18, then restarted again. After that SSMS opens sql files in existing SSMS instance and explorer has correct file icon assigned (document with small database icon)