How to choose the correct resolution for a 5K2K monitor on macOS?
Try the 3360 x 1417 scaling mode for a while. That's exactly why it was introduced to macOS as an option -- so that HiDPI displays can actually be made usable instead of having everything tiny. No it won't be perfect, but there's no perfect answer here. After doing your regular workflow for a few days, you'll know if this mode is acceptable to you.



For desktop use, most people are comfortable when the apparent display resolution is around 100 to 110 ppi. People with lower visual acuity tend to prefer a lower resolution around 80-90 ppi.

You purchased a display with a native ppi of 164 ppi, which is much too high to be comfortable. The most performant scaling mode is 2:1 which would give you a ppi of 82, which is too small to be comfortable for most people.

Next time you find yourself purchasing a display, try to find something around 200-220 ppi (looks like 100-110 ppi at 2:1 scaling). Because your laptop has a limit on how many pixels it can push, that means you're looking at a maximum diagonal size of around 29" (at 5K) to get the picture quality you're looking for.