Zeta topology DC-DC converter unstable
Problem fixed by significantly increasing the C1 to 47uF. Probably that's too much, but there is no way you can "overkill" it within reasonable limits. Adding an input 220uF filter capacitor reduced the voltage unstability greatly to +-0.05V.

Texas instrumets has some great materials regarging Zeta converters design:


Designing DC/DC converters based on ZETA topology
Power Topologies Handbook


While C1 value might seem to be unimportant (as I assumed firstly), it actually has a certain minimal capacity since the voltage drop on it might become too large. I have no idea, why it happens at duty cycles near 50%, however.

In the first link the C1 capacitor minimal value is shown on page 4 of the document. It is largely dependant on the frequency of your controller, maximum/minimum duty cycle and typical input and output voltages/currents, so this capacitor value varies from case to case.