Is there any risk in leaving a Li-ion connected to a solar panel with only step-down converter?
Instead of what you are using just use a propper board. It will charge better and the risk is going to be smaller.

Search on eBay or Aliexpress for CN3791. The board is less than 4$ and does eveything properly.

If you are curious to learn about it, read it's datasheet.