Why are the output voltages way too high on this stacked LM317 circuit?
A linear regulator like the LM317 can only source current, not sink it. Since you have the upper regulators outputs are stacked on top of the lower ones, current is flowing down through the output resistor chain that they cannot sink, and therefore the voltage rises and they lose regulation.  

I think to test the BMS properly you would need some properly isolated voltages, e.g. transformer(s) with independent output windings rectified to drive each regulator, then each section will look like a battery.