<p dir="auto">Reported at <a href="https://discuss.pytorch.org/t/torch-multiprocessing-rebuild-cuda-tensor-having-trouble-with-bn-num-batches-tracked/27453" rel="nofollow">https://discuss.pytorch.org/t/torch-multiprocessing-rebuild-cuda-tensor-having-trouble-with-bn-num-batches-tracked/27453</a></p>
<p dir="auto">Post copied below:</p>
<p dir="auto">I am trying to pass a set of gpu models to a pool of processes, so something like the example code below:<br>
(torch version 1.0.0.dev20181015)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch.multiprocessing as mp
mp.set_start_method('spawn', force=True)
import torch
import torch.nn as nn
import time


def run(inpt):
    workers, worker_queue = inpt
    while worker_queue.empty():
        time.sleep(0.1)
    worker_id = worker_queue.get()
    model = workers[worker_id]
    data = torch.rand(5, 10)
    data = data.cuda()
    with torch.no_grad():
        result = model(data)
    worker_queue.put(worker_id)
    return result


def main():
    workers = []
    worker_queue = mp.Manager().Queue()  # mp.Queue()
    for worker_id in range(10):
        worker_model = nn.BatchNorm1d(10)
        worker_model.cuda()
        worker_model.eval()
        workers.append(worker_model)
        worker_queue.put(worker_id)

    pools = mp.Pool(10)

    jobs = [(workers, worker_queue) for _ in range(20)]  # jobs
    result = []
    for res_id, res in enumerate(pools.imap_unordered(run, jobs)):
        result.append(res)
    print(result)


if __name__==&quot;__main__&quot;:
    main()"><pre class="notranslate"><code class="notranslate">import torch.multiprocessing as mp
mp.set_start_method('spawn', force=True)
import torch
import torch.nn as nn
import time


def run(inpt):
    workers, worker_queue = inpt
    while worker_queue.empty():
        time.sleep(0.1)
    worker_id = worker_queue.get()
    model = workers[worker_id]
    data = torch.rand(5, 10)
    data = data.cuda()
    with torch.no_grad():
        result = model(data)
    worker_queue.put(worker_id)
    return result


def main():
    workers = []
    worker_queue = mp.Manager().Queue()  # mp.Queue()
    for worker_id in range(10):
        worker_model = nn.BatchNorm1d(10)
        worker_model.cuda()
        worker_model.eval()
        workers.append(worker_model)
        worker_queue.put(worker_id)

    pools = mp.Pool(10)

    jobs = [(workers, worker_queue) for _ in range(20)]  # jobs
    result = []
    for res_id, res in enumerate(pools.imap_unordered(run, jobs)):
        result.append(res)
    print(result)


if __name__=="__main__":
    main()
</code></pre></div>
<p dir="auto">However, when there is BatchNorm, the following error is raised in torch._utils.rebuild_cuda_tensor:</p>
<p dir="auto">RuntimeError: Expected object of data type 4 but got data type 6 for argument <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="171402941" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/2" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/2/hovercard" href="https://github.com/pytorch/pytorch/issues/2">#2</a> 'source'<br>
It is probably caused by the attribute bn.num_batches_tracked, which is supposed to be type 4 (int64_t), but looks like the error message is reporting data type 6 (float). What is likely the cause of this issue? Is there any way to fix it?</p>
<p dir="auto">I was trying to send multiple tensors in tuple via <code class="notranslate">multiprocessing.Queue</code> to anther process. However when I access to a cuda LongTensor, I get this error</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="RuntimeError: Attempt to access Storage having data type Float as data type Long"><pre class="notranslate"><code class="notranslate">RuntimeError: Attempt to access Storage having data type Float as data type Long
</code></pre></div>
<h2 dir="auto">Code example</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import torch
import torch.multiprocessing as mp
import time

def producer(queue):
    while True:
        a = torch.ones(2,2).float().cuda()
        idx = torch.LongTensor([[0, 0], [0, 1]]).cuda()
        queue.put((a, idx))

def consumer(queue):
    while True:
        a, idx = queue.get()
        print(idx.type())
        print(idx)

if __name__ == '__main__':
    mp.set_start_method('spawn')

    queue = mp.Queue()

    p = mp.Process(target=producer, args=(queue,))
    c = mp.Process(target=consumer, args=(queue,))
    p.start()
    c.start()
    
    time.sleep(10)

    p.join()
    c.join()"><pre class="notranslate"><code class="notranslate">import torch
import torch.multiprocessing as mp
import time

def producer(queue):
    while True:
        a = torch.ones(2,2).float().cuda()
        idx = torch.LongTensor([[0, 0], [0, 1]]).cuda()
        queue.put((a, idx))

def consumer(queue):
    while True:
        a, idx = queue.get()
        print(idx.type())
        print(idx)

if __name__ == '__main__':
    mp.set_start_method('spawn')

    queue = mp.Queue()

    p = mp.Process(target=producer, args=(queue,))
    c = mp.Process(target=consumer, args=(queue,))
    p.start()
    c.start()
    
    time.sleep(10)

    p.join()
    c.join()
</code></pre></div>
<p dir="auto">Output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="torch.cuda.LongTensor
Process Process-2:
Traceback (most recent call last):
  File &quot;***/lib/python3.5/multiprocessing/process.py&quot;, line 252, in _bootstrap
    self.run()
  File &quot;***/lib/python3.5/multiprocessing/process.py&quot;, line 93, in run
    self._target(*self._args, **self._kwargs)
  File &quot;queue_test.py&quot;, line 15, in consumer
    print(idx)
  File &quot;***/lib/python3.5/site-packages/torch/tensor.py&quot;, line 57, in __repr__
    return torch._tensor_str._str(self)
  File &quot;***/lib/python3.5/site-packages/torch/_tensor_str.py&quot;, line 256, in _str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File &quot;***/lib/python3.5/site-packages/torch/_tensor_str.py&quot;, line 76, in __init__
    copy = torch.empty(tensor.size(), dtype=torch.long).copy_(tensor).view(tensor.nelement())
RuntimeError: Attempt to access Storage having data type Float as data type Long"><pre class="notranslate"><code class="notranslate">torch.cuda.LongTensor
Process Process-2:
Traceback (most recent call last):
  File "***/lib/python3.5/multiprocessing/process.py", line 252, in _bootstrap
    self.run()
  File "***/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "queue_test.py", line 15, in consumer
    print(idx)
  File "***/lib/python3.5/site-packages/torch/tensor.py", line 57, in __repr__
    return torch._tensor_str._str(self)
  File "***/lib/python3.5/site-packages/torch/_tensor_str.py", line 256, in _str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "***/lib/python3.5/site-packages/torch/_tensor_str.py", line 76, in __init__
    copy = torch.empty(tensor.size(), dtype=torch.long).copy_(tensor).view(tensor.nelement())
RuntimeError: Attempt to access Storage having data type Float as data type Long
</code></pre></div>
<h2 dir="auto">System Info</h2>
<p dir="auto">PyTorch version: 0.4.1<br>
Is debug build: No<br>
CUDA used to build PyTorch: 8.0.61</p>
<p dir="auto">OS: Ubuntu 14.04.5 LTS<br>
GCC version: (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4<br>
CMake version: version 3.11.0</p>
<p dir="auto">Python version: 3.5<br>
Is CUDA available: Yes<br>
CUDA runtime version: 8.0.61<br>
GPU models and configuration:<br>
GPU 0: Tesla P100-PCIE-16GB<br>
GPU 1: Tesla P100-PCIE-16GB</p>
<p dir="auto">Nvidia driver version: 375.66<br>
cuDNN version: Probably one of the following:<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.1.10<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6.0.21<br>
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn_static.a</p>
<p dir="auto">Versions of relevant libraries:<br>
[pip] numpy (1.13.3)<br>
[pip] torch (0.4.1)<br>
[pip] torchvision (0.2.1)<br>
[conda] torch                     0.4.1                     <br>
[conda] torchvision               0.2.1                     </p>