New install of Blender; extremely slow when panning a video
Unfortunately, I believe this is 'normal'. It's 4k video, so it takes a lot to edit and needs a lot of optimization that is just not there yet.
Make sure you update your video drivers because if they are not up to date that might cause issues, but I think it will lag some in any case. I would not consider using lower resolution preview a hack since that is what is done in many video editing aplications in situations like this, it is a pretty standard thing to do. Video editing in Blender is being very actively improved at the moment so you can also try the experimental nightly builds, however be aware that they are in development and might be unstable so maybe save your work more often and save a backup file from the stable Blender version.