Change physical keyboard layout
The solution involved deactivating all keyboards from the on-screen keyboard options.

After doing so, the only one that was left (as on-screen) was also the only one left in physical. Even SwiftKey Swedish (not External keyboard helper) now works.

With several possible keyboards in on-screen, I still cannot select which one is default (the options show up with radio buttons, but are not selectable), but I think I can live with it.