Does unsupported RAM effect display cards?
Yes. Apple uses some of your RAM as VRAM as explained in this answer.

So using unsupported RAM can affect your integrated video card.  Try switching back to the stock modules that came with the device and see if you can still replicate your problem.