title: CUDA: do I need different streams on multiple GPUs to execute in parallel?
body: <p>I want to run kernels on multiple GPUs in parallel. For this purpose I switch between the devices using <code>cudaSetDevice()</code> and then start my kernel in the corresponding device. Now, usually all calls in one stream are executed sequentially and one has to use different streams if they shall be executed in parallel. Is this also the case when using different devices or can I in this case run my kernel calls on the default stream on both devices and they will still run in parallel?</p>