How do I allow users to just click a marker on an OpenLayers map to go directly to the node?
Usually jumping to new page just by clicking object is not what users want, as they need to be sure first that the clicked object is correct one, this is why there is no built-in function and you do not see this much in the wild.

OpenLayers module has behaviours for this kind of actions. It seems that the core module does not have what you expect, and nobody has done it yet, so no easy configuration option :( Here is the guide how to create new behaviours from scratch. I would use Popup behavior as sample.