Mid-2011 27" iMac random reboots, GPU defective?
You may find this question of interest to you. 

Unfortunately, this iMac still has intermittent problems and sometimes boots and sometimes doesn't. When it doesn't it just keeps restarting, and even when it does boot it will randomly reboot at some point.

However, to answer your question, yes, a graphics card can cause random reboots. Whether or not it's worth $250 to replace only you can answer. Obviously, if it fixes the problem it's money well spent. If it doesn't, then not. 

But it's what I wanted to do next on the iMac I was troubleshooting, but the owner decided to call it quits. If it was me, I'd personally take the plunge and if it didn't resolve the problem I'd try and resell the graphics card. But that's a personal choice on my behalf and you need to do what you're comfortable doing.