Multiple Blender Threads (targetting CPU and GPU )
Blender uses OpenGL for both the 3D viewport and rendering. If you use GPU rendering the render engine occupies the whole GPU and AFAIK there are no settings to influence that in the current builds (even not in development builds). 

The only way to get a little more responsiveness while rendering in another instance is to switch from Cycles to Blender internal and then switch the 3D viewport from GLSL to Multitexture. This might at least help with the modeling part. 

Other software might continue to work without (or maybe a little) lag, if they don't make use of your GPU in any way. Keep in mind that even more recent versions of MS Office (since 2013) puts load on the GPU to speed up rendering as well as some audio players.