<h2 dir="auto">Issue description</h2>
<p dir="auto">When I use some functions in Distribution with CUDA tensors, it will fail.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="m = torch.distributions.Normal(0, 1)
&gt;&gt;&gt; m.cdf(torch.tensor(.5, device='cuda'))
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
  File &quot;/home/qbx2/anaconda3/lib/python3.6/site-packages/torch/distributions/normal.py&quot;, line 72, in cdf
    return 0.5 * (1 + torch.erf((value - self.loc) * self.scale.reciprocal() / math.sqrt(2)))
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.FloatTensor for argument #3 'other'"><pre class="notranslate"><code class="notranslate">m = torch.distributions.Normal(0, 1)
&gt;&gt;&gt; m.cdf(torch.tensor(.5, device='cuda'))
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/qbx2/anaconda3/lib/python3.6/site-packages/torch/distributions/normal.py", line 72, in cdf
    return 0.5 * (1 + torch.erf((value - self.loc) * self.scale.reciprocal() / math.sqrt(2)))
RuntimeError: Expected object of type torch.cuda.FloatTensor but found type torch.FloatTensor for argument #3 'other'
</code></pre></div>
<p dir="auto">I think it is helpful if there is to() or cuda() to move internal variables to wanted device.</p>
<p dir="auto">I temporarily solved the problem by</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; m.loc = m.loc.to('cuda')
&gt;&gt;&gt; m.scale = m.scale.to('cuda')
&gt;&gt;&gt; m.cdf(torch.tensor(.5, device='cuda'))
tensor(0.6915, device='cuda:0')"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; m.loc = m.loc.to('cuda')
&gt;&gt;&gt; m.scale = m.scale.to('cuda')
&gt;&gt;&gt; m.cdf(torch.tensor(.5, device='cuda'))
tensor(0.6915, device='cuda:0')
</code></pre></div>
<p dir="auto">It would be easy to add the function, how do you think?</p>
<h2 dir="auto">System Info</h2>
<ul dir="auto">
<li>PyTorch or Caffe2: pytorch</li>
<li>How you installed PyTorch (conda, pip, source): source</li>
<li>Build command you used (if compiling from source):</li>
<li>OS: Ubuntu 18.04 LTS</li>
<li>PyTorch version: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/pytorch/pytorch/commit/a24163a95edb193ff7b06e98cd69bf7cfd4c0d2f/hovercard" href="https://github.com/pytorch/pytorch/commit/a24163a95edb193ff7b06e98cd69bf7cfd4c0d2f"><tt>a24163a</tt></a> (I've checked the source in master though)</li>
<li>Python version: 3.6.5+</li>
<li>CUDA/cuDNN version:</li>
<li>GPU models and configuration:</li>
<li>GCC version (if compiling from source):</li>
<li>CMake version:</li>
<li>Versions of any other relevant libraries:</li>
</ul>
<p dir="auto">When I use a custom collate_fn, multiprocessing and move the batch to cuda in the collate_fn, I get a <code class="notranslate">RuntimeError: CUDA error: initialization error</code>. Below a MWE. Tested on google colab with pytorch 1.0.1.post2.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
import torch.utils.data
from collections import defaultdict

class Dataset(torch.utils.data.Dataset):
  
  def __init__(self):
    self.data = defaultdict(dict)
    for i in range(32):
      self.data[i]['sequence'] = list(range(10))
    
  def __getitem__(self, idx):
    return self.data[idx]
 
  def __len__(self):
    return len(self.data)
  

def collate_fn(data):
  batch = defaultdict(list)
  for key in data[0].keys():
    batch[key] = [s[key] for s in data]
    batch[key] = torch.Tensor(batch[key]).long()
    batch[key] = batch[key].to(torch.device('cuda'))
      
  return batch

# default collate works
try:
  dataloader = torch.utils.data.DataLoader(Dataset(), 4, num_workers=2)
  for epoch in range(2):
    for iteration, batch in enumerate(dataloader):
      pass
  print(&quot;Default collate_fn works!&quot;)
except Exception as e:
  print(&quot;Default collate_fn failed!&quot;)
  print(e)

# custom collate fails
try:
  dataloader = torch.utils.data.DataLoader(Dataset(), 4, num_workers=2, collate_fn=collate_fn)
  for epoch in range(2):
    for iteration, batch in enumerate(dataloader):
      print(epoch, iteration)
  print(&quot;Custom collate_fn works!&quot;)
except Exception as e:
  print(&quot;Custom collate_fn failed!&quot;)
  print(e)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-k">import</span> <span class="pl-s1">torch</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">data</span>
<span class="pl-k">from</span> <span class="pl-s1">collections</span> <span class="pl-k">import</span> <span class="pl-s1">defaultdict</span>

<span class="pl-k">class</span> <span class="pl-v">Dataset</span>(<span class="pl-s1">torch</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">data</span>.<span class="pl-v">Dataset</span>):
  
  <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>):
    <span class="pl-s1">self</span>.<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-en">defaultdict</span>(<span class="pl-s1">dict</span>)
    <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">32</span>):
      <span class="pl-s1">self</span>.<span class="pl-s1">data</span>[<span class="pl-s1">i</span>][<span class="pl-s">'sequence'</span>] <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-en">range</span>(<span class="pl-c1">10</span>))
    
  <span class="pl-k">def</span> <span class="pl-en">__getitem__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">idx</span>):
    <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-s1">data</span>[<span class="pl-s1">idx</span>]
 
  <span class="pl-k">def</span> <span class="pl-en">__len__</span>(<span class="pl-s1">self</span>):
    <span class="pl-k">return</span> <span class="pl-en">len</span>(<span class="pl-s1">self</span>.<span class="pl-s1">data</span>)
  

<span class="pl-k">def</span> <span class="pl-en">collate_fn</span>(<span class="pl-s1">data</span>):
  <span class="pl-s1">batch</span> <span class="pl-c1">=</span> <span class="pl-en">defaultdict</span>(<span class="pl-s1">list</span>)
  <span class="pl-k">for</span> <span class="pl-s1">key</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>[<span class="pl-c1">0</span>].<span class="pl-en">keys</span>():
    <span class="pl-s1">batch</span>[<span class="pl-s1">key</span>] <span class="pl-c1">=</span> [<span class="pl-s1">s</span>[<span class="pl-s1">key</span>] <span class="pl-k">for</span> <span class="pl-s1">s</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>]
    <span class="pl-s1">batch</span>[<span class="pl-s1">key</span>] <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>(<span class="pl-s1">batch</span>[<span class="pl-s1">key</span>]).<span class="pl-en">long</span>()
    <span class="pl-s1">batch</span>[<span class="pl-s1">key</span>] <span class="pl-c1">=</span> <span class="pl-s1">batch</span>[<span class="pl-s1">key</span>].<span class="pl-en">to</span>(<span class="pl-s1">torch</span>.<span class="pl-en">device</span>(<span class="pl-s">'cuda'</span>))
      
  <span class="pl-k">return</span> <span class="pl-s1">batch</span>

<span class="pl-c"># default collate works</span>
<span class="pl-k">try</span>:
  <span class="pl-s1">dataloader</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">data</span>.<span class="pl-v">DataLoader</span>(<span class="pl-v">Dataset</span>(), <span class="pl-c1">4</span>, <span class="pl-s1">num_workers</span><span class="pl-c1">=</span><span class="pl-c1">2</span>)
  <span class="pl-k">for</span> <span class="pl-s1">epoch</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">2</span>):
    <span class="pl-k">for</span> <span class="pl-s1">iteration</span>, <span class="pl-s1">batch</span> <span class="pl-c1">in</span> <span class="pl-en">enumerate</span>(<span class="pl-s1">dataloader</span>):
      <span class="pl-k">pass</span>
  <span class="pl-en">print</span>(<span class="pl-s">"Default collate_fn works!"</span>)
<span class="pl-k">except</span> <span class="pl-v">Exception</span> <span class="pl-k">as</span> <span class="pl-s1">e</span>:
  <span class="pl-en">print</span>(<span class="pl-s">"Default collate_fn failed!"</span>)
  <span class="pl-en">print</span>(<span class="pl-s1">e</span>)

<span class="pl-c"># custom collate fails</span>
<span class="pl-k">try</span>:
  <span class="pl-s1">dataloader</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">data</span>.<span class="pl-v">DataLoader</span>(<span class="pl-v">Dataset</span>(), <span class="pl-c1">4</span>, <span class="pl-s1">num_workers</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">collate_fn</span><span class="pl-c1">=</span><span class="pl-s1">collate_fn</span>)
  <span class="pl-k">for</span> <span class="pl-s1">epoch</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-c1">2</span>):
    <span class="pl-k">for</span> <span class="pl-s1">iteration</span>, <span class="pl-s1">batch</span> <span class="pl-c1">in</span> <span class="pl-en">enumerate</span>(<span class="pl-s1">dataloader</span>):
      <span class="pl-en">print</span>(<span class="pl-s1">epoch</span>, <span class="pl-s1">iteration</span>)
  <span class="pl-en">print</span>(<span class="pl-s">"Custom collate_fn works!"</span>)
<span class="pl-k">except</span> <span class="pl-v">Exception</span> <span class="pl-k">as</span> <span class="pl-s1">e</span>:
  <span class="pl-en">print</span>(<span class="pl-s">"Custom collate_fn failed!"</span>)
  <span class="pl-en">print</span>(<span class="pl-s1">e</span>)</pre></div>
<blockquote>
<p dir="auto">Default collate_fn works!<br>
0 0<br>
0 1<br>
0 2<br>
0 3<br>
0 4<br>
0 5<br>
0 6<br>
0 7<br>
Custom collate_fn failed!<br>
Traceback (most recent call last):<br>
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop<br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File "", line 29, in collate_fn<br>
batch[key] = batch[key].to(torch.device('cuda'))<br>
RuntimeError: CUDA error: initialization error</p>
</blockquote>
<p dir="auto">Edit: Obviously for this MWE a custom collate is not necessary. However, in my use case, I need it because I have variable length data, so I do the padding in the collate_fn. But as you can see from the trace, it fails when moving the data to cuda.</p>