I assume that AOZ1280 buck converter misses datasheet specs
I use this part in a lot of projects, mostly for applications with higher input voltage, e.g. 9-24V.  But when I started trying to use from LiPo I saw the same behavior as you described.  I followed the reference design also so I don't think there is anything wrong with your implementation.

I don't have a good explanation for how it behaves when starting from a low voltage compared to starting with a higher voltage.  But I think if you increase your load you will see it behaves more closely both ways.

The main reason I believe the low input voltage performance is poor is because the max duty cycle is 87% (something I missed initially).  I ended up changing to a different controller with 100% max duty cycle to reduce the dropout voltage.  

I agree the specs can be misleading, especially recommended voltage range up to Vin.  If you think about it there must be some dropout though since it's simply buck topology, but I also expected to get much closer to Vin.