Why does this SEPIC buck-boost oscillate so much?
Here is a CircuitLab circuit which models what happens when the two inductors and coupling capacitor from your circuit are driven with a 5V supply and 1.2MHz, 50% constant duty cycle switch.



simulate this circuit &ndash; Schematic created using CircuitLab
Although this circuit shows a relatively stable Vout, (after it stabilizes when C2 is charged at around 2.2mSec) it can be seen that the voltage across the coupling capacitor has over 2V peak-to-peak oscillations, and the two inductors have nearly 2A peak-to-peak oscillations.


Although by the output voltage everything seems OK, by the way these components are oscillating, it could be a problem. That is why I wrote:

By my analysis, the two inductors and sepic cap should be oscillating quite strongly around 5kHz. Could you post a scope shot of Vout for a 4 mSec total duration after the circuit has settled a bit (say after 100 mSec or even 1 sec)? – Math Keeps Me Busy


I will wait on the results, but I suspect that your sepic cap is too big. Do you need it that big for current? If not, I suggest that you remove your 20uF and replace it with the original 5.6 uF and test that. – Math Keeps Me Busy

You then changed the coupling capacitor to 4.7 uF. According to the circuit lab model, this reduced the oscillations to about a quarter of the value when the cap was 20 uF:


Since on your actual board:

@MathKeepsMeBusy I changed the cap back to 4.7uF, single cap. it got a lot more better both on 4K ohm load and 100 ohm load (set to 14V). I used 2 parallel caps for lower ESR. it seems that wasn't a good idea.

I think it is safe to conclude that much of your problem was related to excessive oscillations of the inductors and coupling capacitor.