Solar panel not charging LiPo Batteries
The charger (plus battery) is pushing the solar panel away from its Maximum Power Point, especially when the battery is discharged and sitting at a low voltage. This limits the power the solar panel can deliver to about 60 to 80% of what it could be.
A better solution would be getting one of the pseudo-MPPT charging ICs or modules; these will try to keep the solar panel at or around its Maximum Power Point, and maximise the current flowing into the battery. If that isn't enough to charge your battery fast enough, you will also need a larger solar panel that can deliver more current at the same voltage.
CN3065 and LT3652 are examples of ICs that can do this, but there are others.
Googling &quot;solar li-ion charger&quot; should show a few modules you can use. Check if they can work with a 5V solar panel; some state they require a 6V panel.
As said in another answer, you don't need (or want) the separate DC/DC converter, just the solar charging module.