Is there a way to change the OSX system font size per monitor?
No. There is no way to change the OSX system font size per monitor per window.

The key is that each separate OSX window is rendered as one. With that, each window can span over, or moved between, multiple monitors. When a window is part of multiple screens at the same time, different font sizes at different parts of same the window would be inoperable.