What exactly is happening when a graphing calculator LCD malfunctions when batteries are weak or overloaded?
All ICs (Integrated circuits or "chips") have a working voltage range outside which their behaviour is undefined. This means the manufacturers of the chips don't specify what will happen, just that it (probably) won't work as it should.  

So the microcontroller in your Nspire may have a operating range of say 4V - 5.5V and outside this it might do just about anything. Typical stuff would be resetting, running code it's not meant to, etc.  

This goes for all the other chips in there, including the LCD controller chip (the chip that drives the display) so odd behaviour is to be expected.
As long as the voltage is lower than it should be, damage will be very unlikely, it just won't work properly.

EDIT - some speculation:

My guess at what is actually happening is RAM that holds the pixel information for the LCD is not being read correctly at the lower supply voltages. Either that or the pixel matrix drivers are not transferring the data correctly.
It's hard to say without more info on the display type (e.g. TN, STN, TFT, OLED) as the method of driving them is a bit different. Also seeing the "fast moving lines" would help - for example if they are horizontal then that might fit with the scanning row by row from top to bottom that is usually the way the pixels are driven in LCDs.

Of course it's also possible the LCD read port is not reading correctly, the microcontroller is acting up, etc, etc. Normally it should have a low voltage sense and brownout active, but if it's in diagnostic mode these features are probably turned off.

All the effects described will be mainly due to the transistors no longer acting enough like switches as vicatcu discusses in his answer. 

EDIT 2 - more info provided  

Thanks for updating with more info. However, there's not much more that can be added which isn't vague speculation. To find the exact cause of things you would need to check things like: The display and microcontroller datasheets, microcontroller source code, scope the uC -> LCD connections, possibly even scope the LCD controller chip. Even then you may not be guaranteed to find the exact (original) cause as it may be on an unreachable part of the silicon (so all you see is secondary effects) for example an internal RC oscillator changing frequency so timing is not met on the chip, etc.