Opamp stability issue
As Transistor says, (maybe) you're missing a connection from R1/R2 to the input. The input should be no more negative than one diode drop provided the model is realistic.
Your resistors are a bit low at 1K. Changing them all to 10K would be better.
There is no guarantee the LM324 will start up properly if you drive an input below the negative rail as it is starting up. There is a phenomenon called &quot;phase reversal&quot;. See page 6 in this data sheet.
I strongly suggest a Schottky diode from the inverting input to ground (eg. BAT54 or 1N5819).
You should also have capacitors on the LM317 input and output but they're not essential for stability and won't cause an issue with your simulation.

Here is a simulation in LTspice using the National Semiconductor LM324 model, with and without the diode:


With the resistors reduced to 1K the voltage at the inverting input is about -2V in my simulation. More than I would expect in reality.
For what it's worth, a quick reality check with a similar Thevenin source (-8V with 1.5K in series) gives  -970mV at the non-inverting input of a generic LM324.