LM2596 inverting mode to generate -5v
You may be confused because 'ground' is now connected to the top end of the voltage divider rather than the bottom. However the LM2596 always regulates voltage relative to its own GND pin. The regulator still 'thinks' that it is regulating a positive voltage, and it considers the -12V output to be ground.

the LM2596-ADJ maintains 1.23V between its feedback and GND pins, producing a fixed current through R3. That same current also flows through R4, so its value is adjusted to drop the remaining output voltage. The resistor values are calculated using the same formula as in positive output configuration, but substituting R3 for R1 and R4 for R2 like so:-

R4 = R3 ((Vout/Vref)-1.0)

R3 should be between 1k and 5k. If you want to use standard resistor values then you may have to try various values for R3 until you get a standard value (or close enough to it) for R4.