Can I program a GPU when it is the system's only graphics card?
Can I run PyTorch or Tensorflow on Windows on a GPU that is also acting as the system's graphics card (e.g. there is no graphics built-in to a Ryzen 3600 CPU)? If so, is there any downside, or would I be better off getting a CPU with built-in graphics?

Yes, it is possible to run i.e. Tensorflow on GPU while also using the GPU for your system. You do not need a 2nd graphics-card or integrated GPU.
Keep in mind, your graphicscard will share memory and processing-power between all your programs. GPU-intensive work might slow down the fps on your system and the other way around. Also keep an eye on the memory usage.

I had tensorflow_gpu with a multi-layer CNN running, while playing AAA-Games (i.e. GTA V) using a Ryzen 3600. It worked on my super old NVIDIA GTX 260 (2GB memory) but it crashed quite often cause of the limited memory. Upgraded to a GTX 1080 with 8GB and it worked quite well. Needless to say, you can always fill up your GPU-memory and crash, no matter the size.
