DC to DC converter, different V-
Which DC-DC converter for stepping down the voltage are you using? The non-isolated such as buck converter or the isolated one such as fly-buck converter?

If it's an isolated converter then it is normal that there will be a voltage difference whether it is an AC or DC voltage within the negative terminal. It is the result of the parasitic capacitive coupling within the transformer. 

By using a high input impedance multimeter, you can classify if it's an isolated or not by touching your probe while measuring the voltage between the negative terminal. If the voltage is dropped then it's an isolated converter. Your body will acts as a load and the stray capacitive coupling voltage will be dropped.

Or by using a low input impedance multimeter, the stray capacitance won't affect your measurement, sometimes you need high one, sometimes you need a low one, it depends on your needs, really.


  Certain models of Keysight’s handheld multimeters, for example, the U1242C, have a unique feature: a ZLow function (Figure 2) that allows you to switch from high input impedance mode to low input impedance mode to check for the presence of stray voltages. This solution eliminates the need to carry both a low impedance meter and a high impedance meter.
  https://community.keysight.com/community/keysight-blogs/general-electronics-measurement/blog/2018/04/25/how-to-detect-stray-voltages-with-a-handheld-multimeter