<h2 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Feature</h2>

<h2 dir="auto">Motivation</h2>

<p dir="auto">Changing CPU code to use GPU is just much harder than it intuitively would need to be, as having different operands of an operation in different backends would result in runtime error.<br>
Many logically simple operations like <code class="notranslate">latent = latent + noise</code> or <code class="notranslate">loss = cross_entropy(inferred, labels)</code> would need extra lines of <code class="notranslate">if</code> conditions to work properly for both CPU and GPU.<br>
Pytorch autograd makes programming much easier by automatically making the result of any operation have <code class="notranslate">require_grad=True</code> if any of the operands has so.<br>
So my question is: Why the same logic isn't applied to the cuda-ness of the result, especially considering that the current behavior is getting a runtime error?</p>
<h2 dir="auto">Pitch</h2>

<p dir="auto">The proposed feature is having an API call like <code class="notranslate">torch.cuda.auto(True)</code> and letting the programmer activate or deactivate it manually, and when activated, having CPU and GPU operands in one operation wouldn't cause an error as a GPU copy of the CPU tensors would be created and the whole operation would be done in GPU and return a GPU result, just like the logic for <code class="notranslate">require_grad</code>.<br>
In that case, just adding a <code class="notranslate">model.cuda()</code> line to the code would be enough for many programs to run on GPU.</p>
<h2 dir="auto">Alternatives</h2>

<h2 dir="auto">Additional context</h2>
<p dir="auto">The <a href="https://pytorch.org/2018/04/22/0_4_0-migration-guide.html" rel="nofollow">PyTorch 0.4 Migration Guide</a>, simplifies writing device-agnostic code as follows:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="# at beginning of the script
device = torch.device(&quot;cuda:0&quot; if torch.cuda.is_available() else &quot;cpu&quot;)

...

# then whenever you get a new Tensor or Module
# this won't copy if they are already on the desired device
input = data.to(device)
model = MyModule(...).to(device)"><pre class="notranslate"><span class="pl-c"># at beginning of the script</span>
<span class="pl-s1">device</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">device</span>(<span class="pl-s">"cuda:0"</span> <span class="pl-k">if</span> <span class="pl-s1">torch</span>.<span class="pl-s1">cuda</span>.<span class="pl-en">is_available</span>() <span class="pl-k">else</span> <span class="pl-s">"cpu"</span>)

...

<span class="pl-c"># then whenever you get a new Tensor or Module</span>
<span class="pl-c"># this won't copy if they are already on the desired device</span>
<span class="pl-s1">input</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>.<span class="pl-en">to</span>(<span class="pl-s1">device</span>)
<span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-v">MyModule</span>(...).<span class="pl-en">to</span>(<span class="pl-s1">device</span>)</pre></div>
<p dir="auto">However, this is still not clean.</p>
<p dir="auto">Ideally, we would like PyTorch to move everything over to the GPU, if it's available...<br>
much like TensorFlow.</p>
<p dir="auto">I tried setting the global tensor type to a cuda tensor using the <code class="notranslate">torch.set_default_tensor_type()</code> method.</p>
<p dir="auto">However, there are some fundamental problems with setting the default tensor type.</p>
<ul dir="auto">
<li>
<p dir="auto">Dataloaders give normal (non-cuda) tensors by default. They have to be manually cast using the <code class="notranslate">Tensor.to()</code> method.</p>
</li>
<li>
<p dir="auto">Many methods are simply not implemented for <code class="notranslate">torch.cuda.*Tensor</code>. Thus, setting the global tensor type to cuda fails.</p>
</li>
<li>
<p dir="auto">Conversions to numpy using the <code class="notranslate">numpy()</code> method aren’t available for cuda tensors. One has to go <code class="notranslate">x.cpu().numpy()</code>.<br>
Although this chain is agnostic, it defeats the purpose.</p>
</li>
</ul>
<hr>
<p dir="auto">I find that I use methods like <code class="notranslate">.to(device)</code> and <code class="notranslate">.cpu()</code> far too often in my projects.</p>
<p dir="auto">In my view, <em>it makes the code more verbose than it needs to be</em> and makes it just a little harder to read.</p>
<p dir="auto"><strong>I think that there is room for a global <code class="notranslate">use_gpu</code> flag that can enable developers to run the entire subsequent code in the GPU, where required.</strong></p>
<p dir="auto">Specifically, my request is the following:</p>
<h4 dir="auto">1. Abolish need for the <code class="notranslate">.to(device)</code> suffix:</h4>
<p dir="auto">Circumvent it by letting the developer set the device using a global method like <code class="notranslate">torch.set_default_device()</code> or a convinience method/flag like <code class="notranslate">use_gpu</code></p>
<p dir="auto">Then, whenever, an error is encountered because a CUDA tensor is expected in place of a regular tensor or vice-versa, automatically cast the tensor to the expected device.</p>
<p dir="auto">Additionally,<br>
a. Move <code class="notranslate">nn.Module</code> automatically to the default device.<br>
b. Move the yield of <code class="notranslate">DataLoader</code>s to the default device.</p>
<p dir="auto">Prevent the need to manually cast to the default device.</p>
<h4 dir="auto">2. Add the numpy() method to cuda tensors:</h4>
<p dir="auto">The existing way is to move the tensor to cpu first.<br>
Thus, we have <code class="notranslate">x.cpu().numpy()</code>, which is agnostic but redundant.</p>
<h4 dir="auto">3. Use GPU by default if available:</h4>
<p dir="auto">PyTorch is built from the ground up with the Deep Learning community in mind.</p>
<p dir="auto">With most Deep Learning done on GPUs, they be considered as the default device automatically.</p>
<p dir="auto"><em>Let PyTorch give first preference to the GPU.</em></p>