Only Transmission traffic through VPN
Are you prepared to change VPN provider?
Private Internet Access (PIA) has a 'split tunnel' feature.  The settings have a preference pane to enable and configure split tunnel.  Here is how it should look to send just Transmission traffic via the VPN.

My own experience with PIA's split tunnel is that it has been difficult to implement for macOS 11 but is now working reliably.  I had system hangs with the first beta versions of PIA for macOS 11.  I have used PIA's split tunnel the other way round - sending most traffic via the VPN and just specific apps bypassing the VPN.