GPU Compute (CUDA) Frequently Crashes Computer
It's common for this problem to be caused by running out of VRAM on your card.

One way to alleviate the problem is to reduce the tile size. Some of the things in a Blender scene can be swapped to and from the GPU on a per-tile basis. In those cases, a smaller tile size means less data on the GPU.

Other things will be the same size on the GPU regardless of which tile is being rendered, or how big the tile is. Hopefully, that overhead is small enough that reducing the tile size will help.

Cetagon's suggestions are also worth checking out.