Why doesn't my solar charger work?
You did a lot of handwaving.  A picture is good, but relying on it for all pertinent information is not good.  I'm going to assume that is a USB A connector (like a host), and that the black rectangle on the right is a solar battery with the cells on the other side.

This is a very simple circuit.  The solar battery is sized to charge the rechargable batteries when in sunlight.  The batteries then supply roughly 5 V to the USB jack.

It makes sense this works with a dumb device like a fan that also only connects to the USB power leads.  Apparently your phone is smarter than that and tries to follow the rules of USB.  It may be trying to communicate with the host before it draws more than 100 mA, as required by the USB standard.

It may also be expecting its own proprietary charger that probably puts specific voltages and/or resistances on the data lines.  When the phone sees this particular voltage/resistance signature, it knows it's hooked up to its proprietary charger and that it can draw some large amount of current.  Most likely the real charger is capable of more than the 500 mA maximum a normal USB socket can provide.  The special signature of the real charger is for two reasons: To let the phone know it's OK to draw more than 500 mA, and to make it difficult for third parties to make chargers.  The latter is necessary since a charger could be made quite cheaply and the company that makes the phone wants to make lots of money selling simple and cheap electronics at obscene markup.