installing CuPy with Pytorch
I want to pass tensors from Pytorch to CuPy and back to do some ops not available in Pytorch. What's the recommended way to install them together?

I already installed Pytorch using conda, and it has installed cudatoolkit package. I'd prefer to use conda to install cupy as well. However CuPy installation instructions recommend uninstalling cudatoolkit (at the very bottom). 

So, how do I make sure both Pytorch and CuPy are using the same cuda version, as well as CuDNN, NVCC, etc? Or should I not use conda to install cupy?

Just do conda install -c conda-forge cupy and it should work. That part of documentation is outdated and to be updated.
