upscaling videos on iMac 5k with VLC (or other player) - just enlarge the window?
The fundamental problem is that your sources are low quality and going to a larger, higher resolution screen is just going to make them look more crummy than on a lower res, smaller screen unless you actually process the videos.
You can try interpolating the sources but it's a long process and probably not worth it. The best option is XMedia Recode. You can thank me later ;)