Auto Swoosh to space on a per app basis
I finally figured out a way to do it. I've kept the Auto Swoosh setting on but I've changed the way I open safari. I created a new application using automator with a single applescript block. It has the following code:
tell application &quot;Safari&quot;
    make new document
    activate
end tell

The order is important because its what prevents the spaces switching on activate as there is already a new window in the current space.
To open a new safari window in my current space I simply run this app.
(In case it doesn't make any sense how this is fast, I use spotlight as an app switcher - simply type the first two letters of any app)
EDIT:
I've made a new version that also supports opening links. This is so that when you click a link in e.g. Mail it opens a new safari windows instead of randomly moving to a different space where safari is already open.
Specifically:

Opening this app creates a new safari window without moving spaces
Clicking a link in any non-safari app will create a new window or a new tab
depending on if there is a window already in the space it will open a new tab in that window

Note: the code is modified from here
on open location theURL
    tell application &quot;System Events&quot;
        tell process &quot;Safari&quot;
            try
                set frontWinName to name of window 1
            on error
                my openNewWindow(theURL)
                return
            end try
        end tell
    end tell
    
    tell application &quot;Safari&quot;
        try
            tell (first window whose name is frontWinName and miniaturized is false)
                set current tab to make new tab with properties {URL:theURL}
                #activate
            end tell
        on error
            my openNewWindow(theURL)
        end try
    end tell
end open location


on openNewWindow(theURL)
    tell application &quot;Safari&quot;
        make new document with properties {URL:theURL}
        #activate
    end tell
end openNewWindow

tell application &quot;Safari&quot;
    make new document
    activate
end tell


Save this in AppleScript as application and set it as default browser using this or this.
Copy the following into the app's info.plist to register it as allowing for link opening
 &lt;key&gt;CFBundleURLTypes&lt;/key&gt;
 &lt;array&gt;
     &lt;dict&gt;
         &lt;key&gt;CFBundleURLName&lt;/key&gt;
         &lt;string&gt;local file&lt;/string&gt;
         &lt;key&gt;CFBundleURLSchemes&lt;/key&gt;
         &lt;array&gt;
             &lt;string&gt;file&lt;/string&gt;
         &lt;/array&gt;
     &lt;/dict&gt;
     &lt;dict&gt;
         &lt;key&gt;CFBundleURLName&lt;/key&gt;
         &lt;string&gt;http URL&lt;/string&gt;
         &lt;key&gt;CFBundleURLSchemes&lt;/key&gt;
         &lt;array&gt;
             &lt;string&gt;http&lt;/string&gt;
         &lt;/array&gt;
     &lt;/dict&gt;
     &lt;dict&gt;
         &lt;key&gt;CFBundleURLName&lt;/key&gt;
         &lt;string&gt;Secure http URL&lt;/string&gt;
         &lt;key&gt;CFBundleURLSchemes&lt;/key&gt;
         &lt;array&gt;
             &lt;string&gt;https&lt;/string&gt;
         &lt;/array&gt;
     &lt;/dict&gt;
 &lt;/array&gt;


Accept app permissions (for some reason the app needs to be in the /Applications folder first). And voila! Now just change the app icon to the safari icon and retain your muscle memory to type the name of this app instead of &quot;safari&quot; when opening new browser windows. (I have put way too much effort into this but after months of using it, I'm so happy with the change)