Can mac os support greater than 2x retina (for a small 4k screen)?
While macOS currently does not support scaling to more than 2x, you can always run a display at less than native resolution. For example, run the display at 2560x1440 while using 2x, so the size of the UI would be equivalent to 1280x720. This should still look perfectly fine on a small screen with such high resolution.
The display tells the Mac which resolutions it supports. macOS may not show all of them in System Preferences, but the important ones are usually there.