Problems controlling an LM317 with an op-amp
You should consider looking closely at the LM358 op amp data sheet. The output of the part is capable of saturating to the negative rail if the load resistor is over 10K ohms. 



In your application the 470 ohm feedback resistor on the LM317 has 1.25 volts across it. That results in about 2.6 mA needing to be sinked by the LM358. If you look closely at the LM358's equivalent circuit you will note that the output connects to a PNP transistor emitter. If the output tries to sink current in any moderate amount exceeding the avail base bias on that PNP transistor the emitter pulls up by a VBE drop of about 0.7 volts to get that transistor turned on so that it can sink load current.  



One way out of this problem is to either change to a different type of op-amp that includes GND in its output range or has a rail-to-rail output. A second way you may be able handle this with the LM358 is to add a negative supply to the LM358 of a volt or so that its output can sink current all the way down to a GND level.