My daughter's science project is to charge AA batteries with a solar panel
You have the diode backwards.  Connect the black end of the diode to the red wire from the solar panel, and the end of the diode with the silver band to the red wire of the battery holder.
A 4.5V solar panel won't generate enough voltage to charge the batteries.  Lithium ion cells have a voltage of 3.6 volts and higher.  Four such cells in series means a charging voltage of at least 14.4 volts.  Your solar cell can't produce that voltage.
This is probably a bad idea, any way.  Lithium ion batteries tend to catch fire if improperly charged.  A proper charger is not a solar cell and a diode.

You could use your solar cell and diode to charge nickel cadmium cells or nickel metal hydride cells.  The crude charger will damage them over time, but they won't explode or catch fire.  The voltage of the cells also fits better with the available voltage from your panel - the nominal cell voltage for NiCad and NiMh is around 1.2 volts, so four of them in series could get a decent part of a full charge from your solar panel.
Your cheapest bet is to switch to NiMh cells.  They are readily available, and you can charge them with your solar panel.
If you want to stick with lithium ion cells, then you'll need to look into getting a proper battery management IC or module.  That can get expensive and complicated for a science fair project.

The Tenavolts batteries you mention in the update contain safety circuits and regulator circuits.
They probably wouldn't go up in smoke if you tried to charge them with your solar cell - but there's no guarantee they would charge at all.
Tenavolts is using lithium ion cells with a buck circuit to provide a constant 1.5V output until the cell reaches its discharge limit - at which point it just shuts off.
It probably also has a charge circuit built in, but Tenavolts doesn't provide any information about that.