If water level is rising on its own on oil heating unit instead of going down, could I have a leak? Or something worse?
The two most common causes are:


Defective auto-fill valve (most common in steam systems).

The fill valve is similar to a toilet fill valve where a float ball
raises or lowers with the level of the water maintaining correct
levels.
Leaking heating coil.

Also, a leaking coil within a tank-less or indirect domestic water
heater. The coil sits in the furnace and if it leaks, municipal
water collects within the furnace, causing an overfilled condition.